pax_global_header00006660000000000000000000000064141720725700014517gustar00rootroot0000000000000052 comment=51e46458efbadd0f6ccaf748585d9bf7ef72a98a h11-0.13.0/000077500000000000000000000000001417207257000121715ustar00rootroot00000000000000h11-0.13.0/.coveragerc000066400000000000000000000001661417207257000143150ustar00rootroot00000000000000[run] branch=True source=h11 omit= setup.py [report] exclude_lines = pragma: no cover ^def test_ precision = 1 h11-0.13.0/.github/000077500000000000000000000000001417207257000135315ustar00rootroot00000000000000h11-0.13.0/.github/workflows/000077500000000000000000000000001417207257000155665ustar00rootroot00000000000000h11-0.13.0/.github/workflows/ci.yml000066400000000000000000000014221417207257000167030ustar00rootroot00000000000000name: CI on: push: branches: ["master"] pull_request: branches: ["master"] jobs: tox: runs-on: ubuntu-latest strategy: max-parallel: 5 matrix: python-version: - 3.6 - 3.7 - 3.8 - 3.9 - pypy3 steps: - uses: actions/checkout@v2 - uses: actions/setup-python@v2 with: python-version: ${{ matrix.python-version }} - name: Install tox run: | python -m pip install --upgrade pip setuptools pip install --upgrade tox tox-gh-actions - name: Initialize tox envs run: | tox --parallel auto --notest - name: Test with tox run: | tox --parallel 0 - uses: codecov/codecov-action@v1 with: file: ./coverage.xml h11-0.13.0/.gitignore000066400000000000000000000015061417207257000141630ustar00rootroot00000000000000# Project-specific generated files docs/source/_static/CLIENT.dot docs/source/_static/CLIENT.svg docs/source/_static/SERVER.dot docs/source/_static/SERVER.svg docs/source/_static/special-states.dot docs/source/_static/special-states.svg docs/build/ bench/results/ bench/env/ bench/h11/ fuzz/results/ # Byte-compiled / optimized / DLL files __pycache__/ *.py[cod] *~ \#* .#* # C extensions *.so # Distribution / packaging .Python /build/ /develop-eggs/ /dist/ /eggs/ /lib/ /lib64/ /parts/ /sdist/ /var/ *.egg-info/ .installed.cfg *.egg # Installer logs pip-log.txt # Unit test / coverage reports htmlcov/ .tox/ .coverage .cache nosetests.xml coverage.xml .pytest_cache # Translations *.mo # Mr Developer .mr.developer.cfg .project .pydevproject # Rope .ropeproject # Django stuff: *.log *.pot # Sphinx documentation doc/_build/ h11-0.13.0/CODE_OF_CONDUCT.md000066400000000000000000000045171417207257000147770ustar00rootroot00000000000000# Contributor Code of Conduct As contributors and maintainers of this project, and in the interest of fostering an open and welcoming community, we pledge to respect all people who contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, and other activities. We are committed to making participation in this project a harassment-free experience for everyone, regardless of level of experience, gender, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, religion, or nationality. Examples of unacceptable behavior by participants include: * The use of sexualized language or imagery * Personal attacks * Trolling or insulting/derogatory comments * Public or private harassment * Publishing other's private information, such as physical or electronic addresses, without explicit permission * Other unethical or unprofessional conduct Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful. By adopting this Code of Conduct, project maintainers commit themselves to fairly and consistently applying these principles to every aspect of managing this project. Project maintainers who do not follow or enforce the Code of Conduct may be permanently removed from the project team. This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting a project maintainer at njs@pobox.com. All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. Maintainers are obligated to maintain confidentiality with regard to the reporter of an incident. This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.3.0, available at [http://contributor-covenant.org/version/1/3/0/][version] [homepage]: http://contributor-covenant.org [version]: http://contributor-covenant.org/version/1/3/0/ h11-0.13.0/CONTRIBUTING.md000066400000000000000000000122431417207257000144240ustar00rootroot00000000000000 # Contributing to h11 Thanks for your interest in contributing to h11! Please take a moment to review this document in order to make the contribution process easy and effective for everyone involved. Following these guidelines helps to communicate that you respect the time of the developers managing and developing this free and open source project. In return, they should reciprocate that respect in addressing your issue, assessing changes, and helping you finalize your pull requests. ## What we're looking for h11 is largely feature-complete, in the sense that it has a fairly well-defined scope and (as far as we know) implements pretty much everything that fits within that scope. If we're wrong, please let us know :-). But mostly we're not looking for major new features. On the other hand, the following are all very welcome: * Bug reports and bug fixes * API feedback and suggestions, especially based on experience using h11 * Help making the docs more clear, complete, and generally useful * Good examples of using h11 in different settings (e.g. with twisted, with asyncio, ...) to accomplish different tasks * Improvements in test coverage * Patches that make the code simpler * Patches that make the code faster ## Contributor responsibilities * Code should work across all currently supported Python releases. * Code must be formatted using [black](https://github.com/python/black) and [isort](https://github.com/timothycrosley/isort) as configured in the project. With those projects installed the commands, black h11/ bench/ examples/ fuzz/ isort --profile black --dt h11 bench examples fuzz will format your code for you. * If you change the code, then you have to also add or fix at least one test. (See below for how to run the test suite.) This helps us make sure that we won't later accidentally break whatever you just fixed, and undo your hard work. * [Statement and branch coverage](https://codecov.io/gh/python-hyper/h11) needs to remain at 100.0%. But don't stress too much about making this work up front -- if you post a pull request, then the codecov bot will automatically post a reply letting you know whether you've managed this, and you can iterate to improve it. * The test suite needs to pass. The easy way to check is: ``` pip install tox tox ``` But note that: (1) this might print slightly misleading coverage statistics, because it only shows coverage for individual python versions, and there might be some lines that are only executed on some python versions or implementations, and (2) the full test suite will automatically get run when you submit a pull request, so you don't need to worry too much about tracking down a version of cpython 3.3 or whatever just to run the tests. * Proposed speedups require some profiling and benchmarks to justify the change. * Generally each pull request should be self-contained and fix one bug or implement one new feature. If you can split it up, then you probably should. This makes changes easier to review, and helps us merge things as quickly as possible. * Be welcoming to newcomers and encourage diverse new contributors from all backgrounds. * Respect our [code of conduct](https://github.com/python-hyper/h11/blob/master/CODE_OF_CONDUCT.md>) in all project spaces. ## How to submit a contribution You don't have to sign a license agreement or anything to contribute to h11 -- just make your changes and submit a pull request! (Though you should probably review the [MIT license we use](https://github.com/python-hyper/h11/blob/master/LICENSE.txt) and make sure you're happy licensing your contribution under those terms.) If you're new to Github and pull requests, then are some tutorials on how to get started: * [Make a pull request](http://makeapullrequest.com/) * [How to contribute to an Open Source Project on GitHub](https://egghead.io/series/how-to-contribute-to-an-open-source-project-on-github) ### Release notes We use towncrier to manage our release notes. Basically, every pull request that has a user visible effect should add a short file to the newsfragments/ directory describing the change, with a name like ..rst. See newsfragments/README.rst for details. This way we can keep a good list of changes as we go, which makes the release manager happy, which means we get more frequent releases, which means your change gets into users’ hands faster. ## After you submit a PR We'll try to review it promptly and give feedback -- but if you haven't heard from us after a week, please do send a ping! It's totally fine and normal to post a comment that just says "ping". If your PR needs further changes before it can be merged, just make more changes in your branch and push them to Github -- Github will automatically add your new commits to the existing PR. But Github *won't* automatically *tell* anyone that new commits have been added, so after you've fixed things and are ready for people to take another look, then please post a comment saying so! That will send us a notification so we know to take another look. ## And again, thanks! h11-0.13.0/LICENSE.txt000066400000000000000000000021441417207257000140150ustar00rootroot00000000000000The MIT License (MIT) Copyright (c) 2016 Nathaniel J. Smith and other contributors Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. h11-0.13.0/MANIFEST.in000066400000000000000000000002601417207257000137250ustar00rootroot00000000000000include LICENSE.txt README.rst notes.org tiny-client-demo.py h11/py.typed recursive-include docs * recursive-include h11/tests/data * recursive-include fuzz * prune docs/build h11-0.13.0/README.rst000066400000000000000000000157741417207257000136760ustar00rootroot00000000000000h11 === .. image:: https://travis-ci.org/python-hyper/h11.svg?branch=master :target: https://travis-ci.org/python-hyper/h11 :alt: Automated test status .. image:: https://codecov.io/gh/python-hyper/h11/branch/master/graph/badge.svg :target: https://codecov.io/gh/python-hyper/h11 :alt: Test coverage .. image:: https://readthedocs.org/projects/h11/badge/?version=latest :target: http://h11.readthedocs.io/en/latest/?badge=latest :alt: Documentation Status This is a little HTTP/1.1 library written from scratch in Python, heavily inspired by `hyper-h2 `_. It's a "bring-your-own-I/O" library; h11 contains no IO code whatsoever. This means you can hook h11 up to your favorite network API, and that could be anything you want: synchronous, threaded, asynchronous, or your own implementation of `RFC 6214 `_ -- h11 won't judge you. (Compare this to the current state of the art, where every time a `new network API `_ comes along then someone gets to start over reimplementing the entire HTTP protocol from scratch.) Cory Benfield made an `excellent blog post describing the benefits of this approach `_, or if you like video then here's his `PyCon 2016 talk on the same theme `_. This also means that h11 is not immediately useful out of the box: it's a toolkit for building programs that speak HTTP, not something that could directly replace ``requests`` or ``twisted.web`` or whatever. But h11 makes it much easier to implement something like ``requests`` or ``twisted.web``. At a high level, working with h11 goes like this: 1) First, create an ``h11.Connection`` object to track the state of a single HTTP/1.1 connection. 2) When you read data off the network, pass it to ``conn.receive_data(...)``; you'll get back a list of objects representing high-level HTTP "events". 3) When you want to send a high-level HTTP event, create the corresponding "event" object and pass it to ``conn.send(...)``; this will give you back some bytes that you can then push out through the network. For example, a client might instantiate and then send a ``h11.Request`` object, then zero or more ``h11.Data`` objects for the request body (e.g., if this is a POST), and then a ``h11.EndOfMessage`` to indicate the end of the message. Then the server would then send back a ``h11.Response``, some ``h11.Data``, and its own ``h11.EndOfMessage``. If either side violates the protocol, you'll get a ``h11.ProtocolError`` exception. h11 is suitable for implementing both servers and clients, and has a pleasantly symmetric API: the events you send as a client are exactly the ones that you receive as a server and vice-versa. `Here's an example of a tiny HTTP client `_ It also has `a fine manual `_. FAQ --- *Whyyyyy?* I wanted to play with HTTP in `Curio `__ and `Trio `__, which at the time didn't have any HTTP libraries. So I thought, no big deal, Python has, like, a dozen different implementations of HTTP, surely I can find one that's reusable. I didn't find one, but I did find Cory's call-to-arms blog-post. So I figured, well, fine, if I have to implement HTTP from scratch, at least I can make sure no-one *else* has to ever again. *Should I use it?* Maybe. You should be aware that it's a very young project. But, it's feature complete and has an exhaustive test-suite and complete docs, so the next step is for people to try using it and see how it goes :-). If you do then please let us know -- if nothing else we'll want to talk to you before making any incompatible changes! *What are the features/limitations?* Roughly speaking, it's trying to be a robust, complete, and non-hacky implementation of the first "chapter" of the HTTP/1.1 spec: `RFC 7230: HTTP/1.1 Message Syntax and Routing `_. That is, it mostly focuses on implementing HTTP at the level of taking bytes on and off the wire, and the headers related to that, and tries to be anal about spec conformance. It doesn't know about higher-level concerns like URL routing, conditional GETs, cross-origin cookie policies, or content negotiation. But it does know how to take care of framing, cross-version differences in keep-alive handling, and the "obsolete line folding" rule, so you can focus your energies on the hard / interesting parts for your application, and it tries to support the full specification in the sense that any useful HTTP/1.1 conformant application should be able to use h11. It's pure Python, and has no dependencies outside of the standard library. It has a test suite with 100.0% coverage for both statements and branches. Currently it supports Python 3 (testing on 3.6-3.9) and PyPy 3. The last Python 2-compatible version was h11 0.11.x. (Originally it had a Cython wrapper for `http-parser `_ and a beautiful nested state machine implemented with ``yield from`` to postprocess the output. But I had to take these out -- the new *parser* needs fewer lines-of-code than the old *parser wrapper*, is written in pure Python, uses no exotic language syntax, and has more features. It's sad, really; that old state machine was really slick. I just need a few sentences here to mourn that.) I don't know how fast it is. I haven't benchmarked or profiled it yet, so it's probably got a few pointless hot spots, and I've been trying to err on the side of simplicity and robustness instead of micro-optimization. But at the architectural level I tried hard to avoid fundamentally bad decisions, e.g., I believe that all the parsing algorithms remain linear-time even in the face of pathological input like slowloris, and there are no byte-by-byte loops. (I also believe that it maintains bounded memory usage in the face of arbitrary/pathological input.) The whole library is ~800 lines-of-code. You can read and understand the whole thing in less than an hour. Most of the energy invested in this so far has been spent on trying to keep things simple by minimizing special-cases and ad hoc state manipulation; even though it is now quite small and simple, I'm still annoyed that I haven't figured out how to make it even smaller and simpler. (Unfortunately, HTTP does not lend itself to simplicity.) The API is ~feature complete and I don't expect the general outlines to change much, but you can't judge an API's ergonomics until you actually document and use it, so I'd expect some changes in the details. *How do I try it?* .. code-block:: sh $ pip install h11 $ git clone git@github.com:python-hyper/h11 $ cd h11/examples $ python basic-client.py and go from there. *License?* MIT *Code of conduct?* Contributors are requested to follow our `code of conduct `_ in all project spaces. h11-0.13.0/bench/000077500000000000000000000000001417207257000132505ustar00rootroot00000000000000h11-0.13.0/bench/README.rst000066400000000000000000000010031417207257000147310ustar00rootroot00000000000000Benchmarking h11 ================ See the `asv docs `_ for how to run our (currently very simple) benchmark suite and track speed changes over time. E.g.: * ``PYTHONPATH=.. asv bench`` Or for cases that asv doesn't handle too well (hit control-C when bored of watching numbers scroll): * ``PYTHONPATH=.. pypy benchmarks/benchmarks.py`` * ``PYTHONPATH=.. python -m vmprof --web benchmarks/benchmarks.py`` * ``PYTHONPATH=.. pypy -m vmprof --web benchmarks/benchmarks.py`` h11-0.13.0/bench/asv.conf.json000066400000000000000000000052761417207257000156720ustar00rootroot00000000000000{ // The version of the config file format. Do not change, unless // you know what you are doing. "version": 1, // The name of the project being benchmarked "project": "h11", // The project's homepage "project_url": "https://h11.readthedocs.io/", // The URL or local path of the source code repository for the // project being benchmarked "repo": "..", // List of branches to benchmark. If not provided, defaults to "master" // (for git) or "tip" (for mercurial). // "branches": ["master"], // for git // "branches": ["tip"], // for mercurial // The DVCS being used. If not set, it will be automatically // determined from "repo" by looking at the protocol in the URL // (if remote), or by looking for special directories, such as // ".git" (if local). // "dvcs": "git", // The tool to use to create environments. May be "conda", // "virtualenv" or other value depending on the plugins in use. // If missing or the empty string, the tool will be automatically // determined by looking for tools on the PATH environment // variable. "environment_type": "virtualenv", // the base URL to show a commit for the project. // "show_commit_url": "http://github.com/owner/project/commit/", // The Pythons you'd like to test against. If not provided, defaults // to the current version of Python used to run `asv`. "pythons": ["3.8", "pypy3"], // The matrix of dependencies to test. Each key is the name of a // package (in PyPI) and the values are version numbers. An empty // list indicates to just test against the default (latest) // version. // "matrix": { // "numpy": ["1.6", "1.7"] // }, // The directory (relative to the current directory) that benchmarks are // stored in. If not provided, defaults to "benchmarks" // "benchmark_dir": "benchmarks", // The directory (relative to the current directory) to cache the Python // environments in. If not provided, defaults to "env" // "env_dir": "env", // The directory (relative to the current directory) that raw benchmark // results are stored in. If not provided, defaults to "results". // "results_dir": "results", // The directory (relative to the current directory) that the html tree // should be written to. If not provided, defaults to "html". // "html_dir": "html", // The number of characters to retain in the commit hashes. // "hash_length": 8, // `asv` will cache wheels of the recent builds in each // environment, making them faster to install next time. This is // number of builds to keep, per environment. // "wheel_cache_size": 0 } h11-0.13.0/bench/benchmarks/000077500000000000000000000000001417207257000153655ustar00rootroot00000000000000h11-0.13.0/bench/benchmarks/__init__.py000066400000000000000000000000001417207257000174640ustar00rootroot00000000000000h11-0.13.0/bench/benchmarks/benchmarks.py000066400000000000000000000042521417207257000200570ustar00rootroot00000000000000# Write the benchmarking functions here. # See "Writing benchmarks" in the asv docs for more information. import h11 # Basic ASV benchmark of core functionality def time_server_basic_get_with_realistic_headers(): c = h11.Connection(h11.SERVER) c.receive_data( b"GET / HTTP/1.1\r\n" b"Host: example.com\r\n" b"User-Agent: Mozilla/5.0 (X11; Linux x86_64; " b"rv:45.0) Gecko/20100101 Firefox/45.0\r\n" b"Accept: text/html,application/xhtml+xml," b"application/xml;q=0.9,*/*;q=0.8\r\n" b"Accept-Language: en-US,en;q=0.5\r\n" b"Accept-Encoding: gzip, deflate, br\r\n" b"DNT: 1\r\n" b"Cookie: ID=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" b"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" b"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" b"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" b"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\r\n" b"Connection: keep-alive\r\n\r\n" ) while True: event = c.next_event() if event is h11.NEED_DATA: break c.send( h11.Response( status_code=200, headers=[ (b"Cache-Control", b"private, max-age=0"), (b"Content-Encoding", b"gzip"), (b"Content-Type", b"text/html; charset=UTF-8"), (b"Date", b"Fri, 20 May 2016 09:23:41 GMT"), (b"Expires", b"-1"), (b"Server", b"gws"), (b"X-Frame-Options", b"SAMEORIGIN"), (b"X-XSS-Protection", b"1; mode=block"), (b"Content-Length", b"1000"), ], ) ) c.send(h11.Data(data=b"x" * 1000)) c.send(h11.EndOfMessage()) # Useful for manual benchmarking, e.g. with vmprof or on PyPy def _run_basic_get_repeatedly(): from timeit import default_timer REPEAT = 10000 # while True: for _ in range(7): start = default_timer() for _ in range(REPEAT): time_server_basic_get_with_realistic_headers() finish = default_timer() print("{:.1f} requests/sec".format(REPEAT / (finish - start))) if __name__ == "__main__": _run_basic_get_repeatedly() h11-0.13.0/docs/000077500000000000000000000000001417207257000131215ustar00rootroot00000000000000h11-0.13.0/docs/Makefile000066400000000000000000000177241417207257000145740ustar00rootroot00000000000000# Makefile for Sphinx documentation # # So the build will be able to find the h11 sources export PYTHONPATH := $(CURDIR)/.. # You can set these variables from the command line. SPHINXOPTS = SPHINXBUILD = sphinx-build PAPER = BUILDDIR = build # User-friendly check for sphinx-build ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) $(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don\'t have Sphinx installed, grab it from http://sphinx-doc.org/) endif # Internal variables. PAPEROPT_a4 = -D latex_paper_size=a4 PAPEROPT_letter = -D latex_paper_size=letter ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source # the i18n builder cannot share the environment and doctrees with the others I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) source .PHONY: help help: @echo "Please use \`make ' where is one of" @echo " html to make standalone HTML files" @echo " dirhtml to make HTML files named index.html in directories" @echo " singlehtml to make a single large HTML file" @echo " pickle to make pickle files" @echo " json to make JSON files" @echo " htmlhelp to make HTML files and a HTML help project" @echo " qthelp to make HTML files and a qthelp project" @echo " applehelp to make an Apple Help Book" @echo " devhelp to make HTML files and a Devhelp project" @echo " epub to make an epub" @echo " epub3 to make an epub3" @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" @echo " latexpdf to make LaTeX files and run them through pdflatex" @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" @echo " text to make text files" @echo " man to make manual pages" @echo " texinfo to make Texinfo files" @echo " info to make Texinfo files and run them through makeinfo" @echo " gettext to make PO message catalogs" @echo " changes to make an overview of all changed/added/deprecated items" @echo " xml to make Docutils-native XML files" @echo " pseudoxml to make pseudoxml-XML files for display purposes" @echo " linkcheck to check all external links for integrity" @echo " doctest to run all doctests embedded in the documentation (if enabled)" @echo " coverage to run coverage check of the documentation (if enabled)" @echo " dummy to check syntax errors of document sources" .PHONY: clean clean: rm -rf $(BUILDDIR)/* .PHONY: html html: $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." .PHONY: dirhtml dirhtml: $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." .PHONY: singlehtml singlehtml: $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml @echo @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." .PHONY: pickle pickle: $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle @echo @echo "Build finished; now you can process the pickle files." .PHONY: json json: $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json @echo @echo "Build finished; now you can process the JSON files." .PHONY: htmlhelp htmlhelp: $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp @echo @echo "Build finished; now you can run HTML Help Workshop with the" \ ".hhp project file in $(BUILDDIR)/htmlhelp." .PHONY: qthelp qthelp: $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp @echo @echo "Build finished; now you can run "qcollectiongenerator" with the" \ ".qhcp project file in $(BUILDDIR)/qthelp, like this:" @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/h11.qhcp" @echo "To view the help file:" @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/h11.qhc" .PHONY: applehelp applehelp: $(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp @echo @echo "Build finished. The help book is in $(BUILDDIR)/applehelp." @echo "N.B. You won't be able to view it unless you put it in" \ "~/Library/Documentation/Help or install it in your application" \ "bundle." .PHONY: devhelp devhelp: $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp @echo @echo "Build finished." @echo "To view the help file:" @echo "# mkdir -p $$HOME/.local/share/devhelp/h11" @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/h11" @echo "# devhelp" .PHONY: epub epub: $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub @echo @echo "Build finished. The epub file is in $(BUILDDIR)/epub." .PHONY: epub3 epub3: $(SPHINXBUILD) -b epub3 $(ALLSPHINXOPTS) $(BUILDDIR)/epub3 @echo @echo "Build finished. The epub3 file is in $(BUILDDIR)/epub3." .PHONY: latex latex: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." @echo "Run \`make' in that directory to run these through (pdf)latex" \ "(use \`make latexpdf' here to do that automatically)." .PHONY: latexpdf latexpdf: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo "Running LaTeX files through pdflatex..." $(MAKE) -C $(BUILDDIR)/latex all-pdf @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." .PHONY: latexpdfja latexpdfja: $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex @echo "Running LaTeX files through platex and dvipdfmx..." $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." .PHONY: text text: $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text @echo @echo "Build finished. The text files are in $(BUILDDIR)/text." .PHONY: man man: $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man @echo @echo "Build finished. The manual pages are in $(BUILDDIR)/man." .PHONY: texinfo texinfo: $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo @echo @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." @echo "Run \`make' in that directory to run these through makeinfo" \ "(use \`make info' here to do that automatically)." .PHONY: info info: $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo @echo "Running Texinfo files through makeinfo..." make -C $(BUILDDIR)/texinfo info @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." .PHONY: gettext gettext: $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale @echo @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." .PHONY: changes changes: $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes @echo @echo "The overview file is in $(BUILDDIR)/changes." .PHONY: linkcheck linkcheck: $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck @echo @echo "Link check complete; look for any errors in the above output " \ "or in $(BUILDDIR)/linkcheck/output.txt." .PHONY: doctest doctest: $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest @echo "Testing of doctests in the sources finished, look at the " \ "results in $(BUILDDIR)/doctest/output.txt." .PHONY: coverage coverage: $(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage @echo "Testing of coverage in the sources finished, look at the " \ "results in $(BUILDDIR)/coverage/python.txt." .PHONY: xml xml: $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml @echo @echo "Build finished. The XML files are in $(BUILDDIR)/xml." .PHONY: pseudoxml pseudoxml: $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml @echo @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." .PHONY: dummy dummy: $(SPHINXBUILD) -b dummy $(ALLSPHINXOPTS) $(BUILDDIR)/dummy @echo @echo "Build finished. Dummy builder generates no files." h11-0.13.0/docs/requirements.txt000066400000000000000000000000331417207257000164010ustar00rootroot00000000000000mistune jsonschema ipython h11-0.13.0/docs/source/000077500000000000000000000000001417207257000144215ustar00rootroot00000000000000h11-0.13.0/docs/source/_examples/000077500000000000000000000000001417207257000163765ustar00rootroot00000000000000h11-0.13.0/docs/source/_examples/myclient.py000066400000000000000000000023351417207257000205770ustar00rootroot00000000000000import socket, ssl import h11 class MyHttpClient: def __init__(self, host, port): self.sock = socket.create_connection((host, port)) if port == 443: ctx = ssl.create_default_context() self.sock = ctx.wrap_socket(self.sock, server_hostname=host) self.conn = h11.Connection(our_role=h11.CLIENT) def send(self, *events): for event in events: data = self.conn.send(event) if data is None: # event was a ConnectionClosed(), meaning that we won't be # sending any more data: self.sock.shutdown(socket.SHUT_WR) else: self.sock.sendall(data) # max_bytes_per_recv intentionally set low for pedagogical purposes def next_event(self, max_bytes_per_recv=200): while True: # If we already have a complete event buffered internally, just # return that. Otherwise, read some data, add it to the internal # buffer, and then try again. event = self.conn.next_event() if event is h11.NEED_DATA: self.conn.receive_data(self.sock.recv(max_bytes_per_recv)) continue return event h11-0.13.0/docs/source/_static/000077500000000000000000000000001417207257000160475ustar00rootroot00000000000000h11-0.13.0/docs/source/_static/closelabel.png000066400000000000000000000002501417207257000206570ustar00rootroot00000000000000PNG  IHDRtEXtSoftwareAdobe ImageReadyqe<JIDATxb```@ŀ@ *įP+4.0 I&/6X H7uѼ 0 na,IENDB`h11-0.13.0/docs/source/_static/facebox.css000066400000000000000000000022071417207257000201710ustar00rootroot00000000000000#facebox { position: absolute; top: 0; left: 0; z-index: 100; text-align: left; } #facebox .popup{ position:relative; border:3px solid rgba(0,0,0,0); -webkit-border-radius:5px; -moz-border-radius:5px; border-radius:5px; -webkit-box-shadow:0 0 18px rgba(0,0,0,0.4); -moz-box-shadow:0 0 18px rgba(0,0,0,0.4); box-shadow:0 0 18px rgba(0,0,0,0.4); } #facebox .content { display:table; width: 370px; padding: 10px; background: #fff; -webkit-border-radius:4px; -moz-border-radius:4px; border-radius:4px; } #facebox .content > p:first-child{ margin-top:0; } #facebox .content > p:last-child{ margin-bottom:0; } #facebox .close{ position:absolute; top:5px; right:5px; padding:2px; background:#fff; } #facebox .close img{ opacity:0.3; } #facebox .close:hover img{ opacity:1.0; } #facebox .loading { text-align: center; } #facebox .image { text-align: center; } #facebox img { border: 0; margin: 0; } #facebox_overlay { position: fixed; top: 0px; left: 0px; height:100%; width:100%; } .facebox_hide { z-index:-100; } .facebox_overlayBG { background-color: #000; z-index: 99; }h11-0.13.0/docs/source/_static/facebox.js000066400000000000000000000221531417207257000200170ustar00rootroot00000000000000/* * Facebox (for jQuery) * version: 1.2 (05/05/2008) * @requires jQuery v1.2 or later * * Examples at http://famspam.com/facebox/ * * Licensed under the MIT: * http://www.opensource.org/licenses/mit-license.php * * Copyright 2007, 2008 Chris Wanstrath [ chris@ozmm.org ] * * Usage: * * jQuery(document).ready(function() { * jQuery('a[rel*=facebox]').facebox() * }) * * Terms * Loads the #terms div in the box * * Terms * Loads the terms.html page in the box * * Terms * Loads the terms.png image in the box * * * You can also use it programmatically: * * jQuery.facebox('some html') * jQuery.facebox('some html', 'my-groovy-style') * * The above will open a facebox with "some html" as the content. * * jQuery.facebox(function($) { * $.get('blah.html', function(data) { $.facebox(data) }) * }) * * The above will show a loading screen before the passed function is called, * allowing for a better ajaxy experience. * * The facebox function can also display an ajax page, an image, or the contents of a div: * * jQuery.facebox({ ajax: 'remote.html' }) * jQuery.facebox({ ajax: 'remote.html' }, 'my-groovy-style') * jQuery.facebox({ image: 'stairs.jpg' }) * jQuery.facebox({ image: 'stairs.jpg' }, 'my-groovy-style') * jQuery.facebox({ div: '#box' }) * jQuery.facebox({ div: '#box' }, 'my-groovy-style') * * Want to close the facebox? Trigger the 'close.facebox' document event: * * jQuery(document).trigger('close.facebox') * * Facebox also has a bunch of other hooks: * * loading.facebox * beforeReveal.facebox * reveal.facebox (aliased as 'afterReveal.facebox') * init.facebox * afterClose.facebox * * Simply bind a function to any of these hooks: * * $(document).bind('reveal.facebox', function() { ...stuff to do after the facebox and contents are revealed... }) * */ (function($) { $.facebox = function(data, klass) { $.facebox.loading() if (data.ajax) fillFaceboxFromAjax(data.ajax, klass) else if (data.image) fillFaceboxFromImage(data.image, klass) else if (data.div) fillFaceboxFromHref(data.div, klass) else if ($.isFunction(data)) data.call($) else $.facebox.reveal(data, klass) } /* * Public, $.facebox methods */ $.extend($.facebox, { settings: { opacity : 0.2, overlay : true, /* I don't know why absolute paths don't work. If you try to use facebox * outside of the examples folder these images won't show up. */ loadingImage : '_static/loading.gif', closeImage : '_static/closelabel.png', imageTypes : [ 'png', 'jpg', 'jpeg', 'gif' ], faceboxHtml : '\ ' }, loading: function() { init() if ($('#facebox .loading').length == 1) return true showOverlay() $('#facebox .content').empty() $('#facebox .body').children().hide().end(). append('
') $('#facebox').css({ top: getPageScroll()[1] + (getPageHeight() / 10), left: $(window).width() / 2 - 205 }).show() $(document).bind('keydown.facebox', function(e) { if (e.keyCode == 27) $.facebox.close() return true }) $(document).trigger('loading.facebox') }, reveal: function(data, klass) { $(document).trigger('beforeReveal.facebox') if (klass) $('#facebox .content').addClass(klass) $('#facebox .content').append(data) $('#facebox .loading').remove() $('#facebox .body').children().fadeIn('normal') $('#facebox').css('left', $(window).width() / 2 - ($('#facebox .popup').width() / 2)) $(document).trigger('reveal.facebox').trigger('afterReveal.facebox') }, close: function() { $(document).trigger('close.facebox') return false } }) /* * Public, $.fn methods */ $.fn.facebox = function(settings) { if ($(this).length == 0) return init(settings) function clickHandler() { $.facebox.loading(true) // support for rel="facebox.inline_popup" syntax, to add a class // also supports deprecated "facebox[.inline_popup]" syntax var klass = this.rel.match(/facebox\[?\.(\w+)\]?/) if (klass) klass = klass[1] fillFaceboxFromHref(this.href, klass) return false } return this.bind('click.facebox', clickHandler) } /* * Private methods */ // called one time to setup facebox on this page function init(settings) { if ($.facebox.settings.inited) return true else $.facebox.settings.inited = true $(document).trigger('init.facebox') makeCompatible() var imageTypes = $.facebox.settings.imageTypes.join('|') $.facebox.settings.imageTypesRegexp = new RegExp('\.(' + imageTypes + ')$', 'i') if (settings) $.extend($.facebox.settings, settings) $('body').append($.facebox.settings.faceboxHtml) var preload = [ new Image(), new Image() ] preload[0].src = $.facebox.settings.closeImage preload[1].src = $.facebox.settings.loadingImage $('#facebox').find('.b:first, .bl').each(function() { preload.push(new Image()) preload.slice(-1).src = $(this).css('background-image').replace(/url\((.+)\)/, '$1') }) $('#facebox .close').click($.facebox.close) $('#facebox .close_image').attr('src', $.facebox.settings.closeImage) } // getPageScroll() by quirksmode.com function getPageScroll() { var xScroll, yScroll; if (self.pageYOffset) { yScroll = self.pageYOffset; xScroll = self.pageXOffset; } else if (document.documentElement && document.documentElement.scrollTop) { // Explorer 6 Strict yScroll = document.documentElement.scrollTop; xScroll = document.documentElement.scrollLeft; } else if (document.body) {// all other Explorers yScroll = document.body.scrollTop; xScroll = document.body.scrollLeft; } return new Array(xScroll,yScroll) } // Adapted from getPageSize() by quirksmode.com function getPageHeight() { var windowHeight if (self.innerHeight) { // all except Explorer windowHeight = self.innerHeight; } else if (document.documentElement && document.documentElement.clientHeight) { // Explorer 6 Strict Mode windowHeight = document.documentElement.clientHeight; } else if (document.body) { // other Explorers windowHeight = document.body.clientHeight; } return windowHeight } // Backwards compatibility function makeCompatible() { var $s = $.facebox.settings $s.loadingImage = $s.loading_image || $s.loadingImage $s.closeImage = $s.close_image || $s.closeImage $s.imageTypes = $s.image_types || $s.imageTypes $s.faceboxHtml = $s.facebox_html || $s.faceboxHtml } // Figures out what you want to display and displays it // formats are: // div: #id // image: blah.extension // ajax: anything else function fillFaceboxFromHref(href, klass) { // div if (href.match(/#/)) { var url = window.location.href.split('#')[0] var target = href.replace(url,'') if (target == '#') return $.facebox.reveal($(target).html(), klass) // image } else if (href.match($.facebox.settings.imageTypesRegexp)) { fillFaceboxFromImage(href, klass) // ajax } else { fillFaceboxFromAjax(href, klass) } } function fillFaceboxFromImage(href, klass) { var image = new Image() image.onload = function() { $.facebox.reveal('
', klass) } image.src = href } function fillFaceboxFromAjax(href, klass) { $.get(href, function(data) { $.facebox.reveal(data, klass) }) } function skipOverlay() { return $.facebox.settings.overlay == false || $.facebox.settings.opacity === null } function showOverlay() { if (skipOverlay()) return if ($('#facebox_overlay').length == 0) $("body").append('
') $('#facebox_overlay').hide().addClass("facebox_overlayBG") .css('opacity', $.facebox.settings.opacity) .click(function() { $(document).trigger('close.facebox') }) .fadeIn(200) return false } function hideOverlay() { if (skipOverlay()) return $('#facebox_overlay').fadeOut(200, function(){ $("#facebox_overlay").removeClass("facebox_overlayBG") $("#facebox_overlay").addClass("facebox_hide") $("#facebox_overlay").remove() }) return false } /* * Bindings */ $(document).bind('close.facebox', function() { $(document).unbind('keydown.facebox') $('#facebox').fadeOut(function() { $('#facebox .content').removeClass().addClass('content') $('#facebox .loading').remove() $(document).trigger('afterClose.facebox') }) hideOverlay() }) })(jQuery); h11-0.13.0/docs/source/_static/loading.gif000077500000000000000000000053171417207257000201640ustar00rootroot00000000000000GIF89a 򺺺444ėTTT! NETSCAPE2.0! , H *\p hp"8G>D)R4CIË\9p:ȹs1_2`p` u< uSYڐkǞ`Fhvƴ6S>u+ryJ/QM.0@p_ ++/KY&]9ى Mr `ixr\˪ vfjMO&*Z؇o>;ܦŝ",,@CPؼrSE.ٴjTWYR Y+ѫKb ڌ! ,H*/g, ">") .replace(/"/g, """) .replace(/'/g, "'")) } function scrapeText(codebox){ /// Returns input lines cleaned of prompt1 and prompt2 var lines = codebox.split('\n'); var newlines = new Array(); $.each(lines, function() { if (this.match(/^In \[\d+]: /)){ newlines.push(this.replace(/^(\s)*In \[\d+]: /,"")); } else if (this.match(/^(\s)*\.+:/)){ newlines.push(this.replace(/^(\s)*\.+: /,"")); } } ); return newlines.join('\\n'); } $(document).ready( function() { // grab all code boxes var ipythoncode = $(".highlight-ipython"); $.each(ipythoncode, function() { var code = scrapeText($(this).text()); // give them a facebox pop-up with plain text code $(this).append('View Code'); $(this,"textarea").select(); }); }); h11-0.13.0/docs/source/api.rst000066400000000000000000001274231417207257000157350ustar00rootroot00000000000000.. _API-documentation: API documentation ================= .. module:: h11 .. contents:: h11 has a fairly small public API, with all public symbols available directly at the top level: .. ipython:: In [2]: import h11 @verbatim In [3]: h11. h11.CLIENT h11.MUST_CLOSE h11.CLOSED h11.NEED_DATA h11.Connection h11.PAUSED h11.ConnectionClosed h11.PRODUCT_ID h11.Data h11.ProtocolError h11.DONE h11.RemoteProtocolError h11.EndOfMessage h11.Request h11.ERROR h11.Response h11.IDLE h11.SEND_BODY h11.InformationalResponse h11.SEND_RESPONSE h11.LocalProtocolError h11.SERVER h11.MIGHT_SWITCH_PROTOCOL h11.SWITCHED_PROTOCOL These symbols fall into three main categories: event classes, special constants used to track different connection states, and the :class:`Connection` class itself. We'll describe them in that order. .. _events: Events ------ *Events* are the core of h11: the whole point of h11 is to let you think about HTTP transactions as being a series of events sent back and forth between a client and a server, instead of thinking in terms of bytes. All events behave in essentially similar ways. Let's take :class:`Request` as an example. Like all events, this is a "final" class -- you cannot subclass it. And like all events, it has several fields. For :class:`Request`, there are four of them: :attr:`~Request.method`, :attr:`~Request.target`, :attr:`~Request.headers`, and :attr:`~Request.http_version`. :attr:`~Request.http_version` defaults to ``b"1.1"``; the rest have no default, so to create a :class:`Request` you have to specify their values: .. ipython:: python req = h11.Request(method="GET", target="/", headers=[("Host", "example.com")]) Event constructors accept only keyword arguments, not positional arguments. Events have a useful repr: .. ipython:: python req And their fields are available as regular attributes: .. ipython:: python req.method req.target req.headers req.http_version Notice that these attributes have been normalized to byte-strings. In general, events normalize and validate their fields when they're constructed. Some of these normalizations and checks are specific to a particular event -- for example, :class:`Request` enforces RFC 7230's requirement that HTTP/1.1 requests must always contain a ``"Host"`` header: .. ipython:: python # HTTP/1.0 requests don't require a Host: header h11.Request(method="GET", target="/", headers=[], http_version="1.0") .. ipython:: python :okexcept: # But HTTP/1.1 requests do h11.Request(method="GET", target="/", headers=[]) This helps protect you from accidentally violating the protocol, and also helps protect you from remote peers who attempt to violate the protocol. A few of these normalization rules are standard across multiple events, so we document them here: .. _headers-format: :attr:`headers`: In h11, headers are represented internally as a list of (*name*, *value*) pairs, where *name* and *value* are both byte-strings, *name* is always lowercase, and *name* and *value* are both guaranteed not to have any leading or trailing whitespace. When constructing an event, we accept any iterable of pairs like this, and will automatically convert native strings containing ascii or :term:`bytes-like object`\s to byte-strings and convert names to lowercase: .. ipython:: python original_headers = [("HOST", bytearray(b"Example.Com"))] req = h11.Request(method="GET", target="/", headers=original_headers) original_headers req.headers If any names are detected with leading or trailing whitespace, then this is an error ("in the past, differences in the handling of such whitespace have led to security vulnerabilities" -- `RFC 7230 `_). We also check for certain other protocol violations, e.g. it's always illegal to have a newline inside a header value, and ``Content-Length: hello`` is an error because `Content-Length` should always be an integer. We may add additional checks in the future. While we make sure to expose header names as lowercased bytes, we also preserve the original header casing that is used. Compliant HTTP agents should always treat headers in a case insensitive manner, but this may not always be the case. When sending bytes over the wire we send headers preserving whatever original header casing was used. It is possible to access the headers in their raw original casing, which may be useful for some user output or debugging purposes. .. ipython:: python original_headers = [("Host", "example.com")] req = h11.Request(method="GET", target="/", headers=original_headers) req.headers.raw_items() .. _http_version-format: It's not just headers we normalize to being byte-strings: the same type-conversion logic is also applied to the :attr:`Request.method` and :attr:`Request.target` field, and -- for consistency -- all :attr:`http_version` fields. In particular, we always represent HTTP version numbers as byte-strings like ``b"1.1"``. :term:`Bytes-like object`\s and native strings will be automatically converted to byte strings. Note that the HTTP standard `specifically guarantees `_ that all HTTP version numbers will consist of exactly two digits separated by a dot, so comparisons like ``req.http_version < b"1.1"`` are safe and valid. When manually constructing an event, you generally shouldn't specify :attr:`http_version`, because it defaults to ``b"1.1"``, and if you attempt to override this to some other value then :meth:`Connection.send` will reject your event -- h11 only speaks HTTP/1.1. But it does understand other versions of HTTP, so you might receive events with other ``http_version`` values from remote peers. Here's the complete set of events supported by h11: .. autoclass:: Request .. autoclass:: InformationalResponse .. autoclass:: Response .. autoclass:: Data .. autoclass:: EndOfMessage .. autoclass:: ConnectionClosed .. _state-machine: The state machine ----------------- Now that you know what the different events are, the next question is: what can you do with them? A basic HTTP request/response cycle looks like this: * The client sends: * one :class:`Request` event with request metadata and headers, * zero or more :class:`Data` events with the request body (if any), * and an :class:`EndOfMessage` event. * And then the server replies with: * zero or more :class:`InformationalResponse` events, * one :class:`Response` event, * zero or more :class:`Data` events with the response body (if any), * and a :class:`EndOfMessage` event. And once that's finished, both sides either close the connection, or they go back to the top and re-use it for another request/response cycle. To coordinate this interaction, the h11 :class:`Connection` object maintains several state machines: one that tracks what the client is doing, one that tracks what the server is doing, and a few more tiny ones to track whether :ref:`keep-alive ` is enabled and whether the client has proposed to :ref:`switch protocols `. h11 always keeps track of all of these state machines, regardless of whether it's currently playing the client or server role. The state machines look like this (click on each to expand): .. ipython:: python :suppress: import sys import subprocess subprocess.check_call([sys.executable, sys._h11_hack_docs_source_path + "/make-state-diagrams.py"]) .. |client-image| image:: _static/CLIENT.svg :target: _static/CLIENT.svg :width: 100% :align: top .. |server-image| image:: _static/SERVER.svg :target: _static/SERVER.svg :width: 100% :align: top .. |special-image| image:: _static/special-states.svg :target: _static/special-states.svg :width: 100% +----------------+----------------+ | |client-image| | |server-image| | +----------------+----------------+ | |special-image| | +---------------------------------+ If you squint at the first two diagrams, you can see the client's IDLE -> SEND_BODY -> DONE path and the server's IDLE -> SEND_RESPONSE -> SEND_BODY -> DONE path, which encode the basic sequence of events we described above. But there's a fair amount of other stuff going on here as well. The first thing you should notice is the different colors. These correspond to the different ways that our state machines can change state. * Dark blue arcs are *event-triggered transitions*: if we're in state A, and this event happens, when we switch to state B. For the client machine, these transitions always happen when the client *sends* an event. For the server machine, most of them involve the server sending an event, except that the server also goes from IDLE -> SEND_RESPONSE when the client sends a :class:`Request`. * Green arcs are *state-triggered transitions*: these are somewhat unusual, and are used to couple together the different state machines -- if, at any moment, one machine is in state A and another machine is in state B, then the first machine immediately transitions to state C. For example, if the CLIENT machine is in state DONE, and the SERVER machine is in the CLOSED state, then the CLIENT machine transitions to MUST_CLOSE. And the same thing happens if the CLIENT machine is in the state DONE and the keep-alive machine is in the state disabled. * There are also two purple arcs labeled :meth:`~Connection.start_next_cycle`: these correspond to an explicit method call documented below. Here's why we have all the stuff in those diagrams above, beyond what's needed to handle the basic request/response cycle: * Server sending a :class:`Response` directly from :data:`IDLE`: This is used for error responses, when the client's request never arrived (e.g. 408 Request Timed Out) or was unparseable gibberish (400 Bad Request) and thus didn't register with our state machine as a real :class:`Request`. * The transitions involving :data:`MUST_CLOSE` and :data:`CLOSE`: keep-alive and shutdown handling; see :ref:`keepalive-and-pipelining` and :ref:`closing`. * The transitions involving :data:`MIGHT_SWITCH_PROTOCOL` and :data:`SWITCHED_PROTOCOL`: See :ref:`switching-protocols`. * That weird :data:`ERROR` state hanging out all lonely on the bottom: to avoid cluttering the diagram, we don't draw any arcs coming into this node, but that doesn't mean it can't be entered. In fact, it can be entered from any state: if any exception occurs while trying to send/receive data, then the corresponding machine will transition directly to this state. Once there, though, it can never leave -- that part of the diagram is accurate. See :ref:`error-handling`. And finally, note that in these diagrams, all the labels that are in *italics* are informal English descriptions of things that happen in the code, while the labels in upright text correspond to actual objects in the public API. You've already seen the event objects like :class:`Request` and :class:`Response`; there are also a set of opaque sentinel values that you can use to track and query the client and server's states. Special constants ----------------- h11 exposes some special constants corresponding to the different states in the client and server state machines described above. The complete list is: .. data:: IDLE SEND_RESPONSE SEND_BODY DONE MUST_CLOSE CLOSED MIGHT_SWITCH_PROTOCOL SWITCHED_PROTOCOL ERROR For example, we can see that initially the client and server start in state :data:`IDLE` / :data:`IDLE`: .. ipython:: python conn = h11.Connection(our_role=h11.CLIENT) conn.states And then if the client sends a :class:`Request`, then the client switches to state :data:`SEND_BODY`, while the server switches to state :data:`SEND_RESPONSE`: .. ipython:: python conn.send(h11.Request(method="GET", target="/", headers=[("Host", "example.com")])); conn.states And we can test these values directly using constants like :data:`SEND_BODY`: .. ipython:: python conn.states[h11.CLIENT] is h11.SEND_BODY This shows how the :class:`Connection` type tracks these state machines and lets you query their current state. The above also showed the special constants that can be used to indicate the two different roles that a peer can play in an HTTP connection: .. data:: CLIENT SERVER And finally, there are also two special constants that can be returned from :meth:`Connection.next_event`: .. data:: NEED_DATA PAUSED All of these behave the same, and their behavior is modeled after :data:`None`: they're opaque singletons, their :meth:`__repr__` is their name, and you compare them with ``is``. .. _sentinel-type-trickiness: Finally, h11's constants have a quirky feature that can sometimes be useful: they are instances of themselves. .. ipython:: python type(h11.NEED_DATA) is h11.NEED_DATA type(h11.PAUSED) is h11.PAUSED The main application of this is that when handling the return value from :meth:`Connection.next_event`, which is sometimes an instance of an event class and sometimes :data:`NEED_DATA` or :data:`PAUSED`, you can always call ``type(event)`` to get something useful to dispatch one, using e.g. a handler table, :func:`functools.singledispatch`, or calling ``getattr(some_object, "handle_" + type(event).__name__)``. Not that this kind of dispatch-based strategy is always the best approach -- but the option is there if you want it. The Connection object --------------------- .. autoclass:: Connection .. automethod:: receive_data .. automethod:: next_event .. automethod:: send .. automethod:: send_with_data_passthrough .. automethod:: send_failed .. automethod:: start_next_cycle .. attribute:: our_role :data:`CLIENT` if this is a client; :data:`SERVER` if this is a server. .. attribute:: their_role :data:`SERVER` if this is a client; :data:`CLIENT` if this is a server. .. autoattribute:: states .. autoattribute:: our_state .. autoattribute:: their_state .. attribute:: their_http_version The version of HTTP that our peer claims to support. ``None`` if we haven't yet received a request/response. This is preserved by :meth:`start_next_cycle`, so it can be handy for a client making multiple requests on the same connection: normally you don't know what version of HTTP the server supports until after you do a request and get a response -- so on an initial request you might have to assume the worst. But on later requests on the same connection, the information will be available here. .. attribute:: client_is_waiting_for_100_continue True if the client sent a request with the ``Expect: 100-continue`` header, and is still waiting for a response (i.e., the server has not sent a 100 Continue or any other kind of response, and the client has not gone ahead and started sending the body anyway). See `RFC 7231 section 5.1.1 `_ for details. .. attribute:: they_are_waiting_for_100_continue True if :attr:`their_role` is :data:`CLIENT` and :attr:`client_is_waiting_for_100_continue`. .. autoattribute:: trailing_data .. _error-handling: Error handling -------------- Given the vagaries of networks and the folks on the other side of them, it's extremely important to be prepared for errors. Most errors in h11 are signaled by raising one of :exc:`ProtocolError`'s two concrete base classes, :exc:`LocalProtocolError` and :exc:`RemoteProtocolError`: .. autoexception:: ProtocolError .. autoexception:: LocalProtocolError .. autoexception:: RemoteProtocolError There are four cases where these exceptions might be raised: * When trying to instantiate an event object (:exc:`LocalProtocolError`): This indicates that something about your event is invalid. Your event wasn't constructed, but there are no other consequences -- feel free to try again. * When calling :meth:`Connection.start_next_cycle` (:exc:`LocalProtocolError`): This indicates that the connection is not ready to be re-used, because one or both of the peers are not in the :data:`DONE` state. The :class:`Connection` object remains usable, and you can try again later. * When calling :meth:`Connection.next_event` (:exc:`RemoteProtocolError`): This indicates that the remote peer has violated our protocol assumptions. This is unrecoverable -- we don't know what they're doing and we cannot safely proceed. :attr:`Connection.their_state` immediately becomes :data:`ERROR`, and all further calls to :meth:`~Connection.next_event` will also raise :exc:`RemoteProtocolError`. :meth:`Connection.send` still works as normal, so if you're implementing a server and this happens then you have an opportunity to send back a 400 Bad Request response. But aside from that, your only real option is to close your socket and make a new connection. * When calling :meth:`Connection.send` or :meth:`Connection.send_with_data_passthrough` (:exc:`LocalProtocolError`): This indicates that *you* violated our protocol assumptions. This is also unrecoverable -- h11 doesn't know what you're doing, its internal state may be inconsistent, and we cannot safely proceed. :attr:`Connection.our_state` immediately becomes :data:`ERROR`, and all further calls to :meth:`~Connection.send` will also raise :exc:`LocalProtocolError`. The only thing you can reasonably due at this point is to close your socket and make a new connection. So that's how h11 tells you about errors that it detects. In some cases, it's also useful to be able to tell h11 about an error that you detected. In particular, the :class:`Connection` object assumes that after you call :meth:`Connection.send`, you actually send that data to the remote peer. But sometimes, for one reason or another, this doesn't actually happen. Here's a concrete example. Suppose you're using h11 to implement an HTTP client that keeps a pool of connections so it can re-use them when possible (see :ref:`keepalive-and-pipelining`). You take a connection from the pool, and start to do a large upload... but then for some reason this gets cancelled (maybe you have a GUI and a user clicked "cancel"). This can cause h11's model of this connection to diverge from reality: for example, h11 might think that you successfully sent the full request, because you passed an :class:`EndOfMessage` object to :meth:`Connection.send`, but in fact you didn't, because you never sent the resulting bytes. And then – here's the really tricky part! – if you're not careful, you might think that it's OK to put this connection back into the connection pool and re-use it, because h11 is telling you that a full request/response cycle was completed. But this is wrong; in fact you have to close this connection and open a new one. The solution is simple: call :meth:`Connection.send_failed`, and now h11 knows that your send failed. In this case, :attr:`Connection.our_state` immediately becomes :data:`ERROR`, just like if you had tried to do something that violated the protocol. .. _framing: Message body framing: ``Content-Length`` and all that ----------------------------------------------------- There are two different headers that HTTP/1.1 uses to indicate a framing mechanism for request/response bodies: ``Content-Length`` and ``Transfer-Encoding``. Our general philosophy is that the way you tell h11 what configuration you want to use is by setting the appropriate headers in your request / response, and then h11 will both pass those headers on to the peer and encode the body appropriately. Currently, the only supported ``Transfer-Encoding`` is ``chunked``. On requests, this means: * No ``Content-Length`` or ``Transfer-Encoding``: no body, equivalent to ``Content-Length: 0``. * ``Content-Length: ...``: You're going to send exactly the specified number of bytes. h11 will keep track and signal an error if your :class:`EndOfMessage` doesn't happen at the right place. * ``Transfer-Encoding: chunked``: You're going to send a variable / not yet known number of bytes. Note 1: only HTTP/1.1 servers are required to support ``Transfer-Encoding: chunked``, and as a client you have to decide whether to send this header before you get to see what protocol version the server is using. Note 2: even though HTTP/1.1 servers are required to support ``Transfer-Encoding: chunked``, this doesn't necessarily mean that they actually do -- e.g., applications using Python's standard WSGI API cannot accept chunked requests. Nonetheless, this is the only way to send request where you don't know the size of the body ahead of time, so if that's the situation you find yourself in then you might as well try it and hope. On responses, things are a bit more subtle. There are effectively two cases: * ``Content-Length: ...``: You're going to send exactly the specified number of bytes. h11 will keep track and signal an error if your :class:`EndOfMessage` doesn't happen at the right place. * ``Transfer-Encoding: chunked``, *or*, neither framing header is provided: These two cases are handled differently at the wire level, but as far as the application is concerned they provide (almost) exactly the same semantics: in either case, you'll send a variable / not yet known number of bytes. The difference between them is that ``Transfer-Encoding: chunked`` works better (compatible with keep-alive, allows trailing headers, clearly distinguishes between successful completion and network errors), but requires an HTTP/1.1 client; for HTTP/1.0 clients the only option is the no-headers approach where you have to close the socket to indicate completion. Since this is (almost) entirely a wire-level-encoding concern, h11 abstracts it: when sending a response you can set either ``Transfer-Encoding: chunked`` or leave off both framing headers, and h11 will treat both cases identically: it will automatically pick the best option given the client's advertised HTTP protocol level. You need to watch out for this if you're using trailing headers (i.e., a non-empty ``headers`` attribute on :class:`EndOfMessage`), since trailing headers are only legal if we actually ended up using ``Transfer-Encoding: chunked``. Trying to send a non-empty set of trailing headers to a HTTP/1.0 client will raise a :exc:`LocalProtocolError`. If this use case is important to you, check :attr:`Connection.their_http_version` to confirm that the client speaks HTTP/1.1 before you attempt to send any trailing headers. .. _keepalive-and-pipelining: Re-using a connection: keep-alive and pipelining ------------------------------------------------ HTTP/1.1 allows a connection to be re-used for multiple request/response cycles (also known as "keep-alive"). This can make things faster by letting us skip the costly connection setup, but it does create some complexities: we have to keep track of whether a connection is reusable, and when there are multiple requests and responses flowing through the same connection we need to be careful not to get confused about which request goes with which response. h11 considers a connection to be reusable if, and only if, both sides (a) speak HTTP/1.1 (HTTP/1.0 did have some complex and fragile support for keep-alive bolted on, but h11 currently doesn't support that -- possibly this will be added in the future), and (b) neither side has explicitly disabled keep-alive by sending a ``Connection: close`` header. If you plan to make only a single request or response and then close the connection, you should manually set the ``Connection: close`` header in your request/response. h11 will notice and update its state appropriately. There are also some situations where you are required to send a ``Connection: close`` header, e.g. if you are a server talking to a client that doesn't support keep-alive. You don't need to worry about these cases -- h11 will automatically add this header when necessary. Just worry about setting it when it's actually something that you're actively choosing. If you want to re-use a connection, you have to wait until both the request and the response have been completed, bringing both the client and server to the :data:`DONE` state. Once this has happened, you can explicitly call :meth:`Connection.start_next_cycle` to reset both sides back to the :data:`IDLE` state. This makes sure that the client and server remain synched up. If keep-alive is disabled for whatever reason -- someone set ``Connection: close``, lack of protocol support, one of the sides just unilaterally closed the connection -- then the state machines will skip past the :data:`DONE` state directly to the :data:`MUST_CLOSE` or :data:`CLOSED` states. In this case, trying to call :meth:`~Connection.start_next_cycle` will raise an error, and the only thing you can legally do is to close this connection and make a new one. HTTP/1.1 also allows for a more aggressive form of connection re-use, in which a client sends multiple requests in quick succession, and then waits for the responses to stream back in order ("pipelining"). This is generally considered to have been a bad idea, because it makes things like error recovery very complicated. As a client, h11 does not support pipelining. This is enforced by the structure of the state machine: after sending one :class:`Request`, you can't send another until after calling :meth:`~Connection.start_next_cycle`, and you can't call :meth:`~Connection.start_next_cycle` until the server has entered the :data:`DONE` state, which requires reading the server's full response. As a server, h11 provides the minimal support for pipelining required to comply with the HTTP/1.1 standard: if the client sends multiple pipelined requests, then we handle the first request until we reach the :data:`DONE` state, and then :meth:`~Connection.next_event` will pause and refuse to parse any more events until the response is completed and :meth:`~Connection.start_next_cycle` is called. See the next section for more details. .. _flow-control: Flow control ------------ Presumably you know when you want to send things, and the :meth:`~Connection.send` interface is very simple: it just immediately returns all the data you need to send for the given event, so you can apply whatever send buffer strategy you want. But reading from the remote peer is a bit trickier: you don't want to read data from the remote peer if it can't be processed (i.e., you want to apply backpressure and avoid building arbitrarily large in-memory buffers), and you definitely don't want to block waiting on data from the remote peer at the same time that it's blocked waiting for you, because that will cause a deadlock. One complication here is that if you're implementing a server, you have to be prepared to handle :class:`Request`\s that have an ``Expect: 100-continue`` header. You can `read the spec `_ for the full details, but basically what this header means is that after sending the :class:`Request`, the client plans to pause and wait until they see some response from the server before they send that request's :class:`Data`. The server's response would normally be an :class:`InformationalResponse` with status ``100 Continue``, but it could be anything really (e.g. a full :class:`Response` with a 4xx status code). The crucial thing as a server, though, is that you should never block trying to read a request body if the client is blocked waiting for you to tell them to send the request body. Fortunately, h11 makes this easy, because it tracks whether the client is in the waiting-for-100-continue state, and exposes this as :attr:`Connection.they_are_waiting_for_100_continue`. So you don't have to pay attention to the ``Expect`` header yourself; you just have to make sure that before you block waiting to read a request body, you execute some code like: .. code-block:: python if conn.they_are_waiting_for_100_continue: do_send(conn, h11.InformationalResponse(100, headers=[...])) do_read(...) In fact, if you're lazy (and what programmer isn't?) then you can just do this check before all reads -- it's mandatory before blocking to read a request body, but it's safe at any time. And the other thing you want to pay attention to is the special values that :meth:`~Connection.next_event` might return: :data:`NEED_DATA` and :data:`PAUSED`. :data:`NEED_DATA` is what it sounds like: it means that :meth:`~Connection.next_event` is guaranteed not to return any more real events until you've called :meth:`~Connection.receive_data` at least once. :data:`PAUSED` is a little more subtle: it means that :meth:`~Connection.next_event` is guaranteed not to return any more real events until something else has happened to clear up the paused state. There are three cases where this can happen: 1) We received a full request/response from the remote peer, and then we received some more data after that. (The main situation where this might happen is a server responding to a pipelining client.) The :data:`PAUSED` state will go away after you call :meth:`~Connection.start_next_cycle`. 2) A successful ``CONNECT`` or ``Upgrade:`` request has caused the connection to switch to some other protocol (see :ref:`switching-protocols`). This :data:`PAUSED` state is permanent; you should abandon this :class:`Connection` and go do whatever it is you're going to do with your new protocol. 3) We're a server, and the client we're talking to proposed to switch protocols (see :ref:`switching-protocols`), and now is waiting to find out whether their request was successful or not. Once we either accept or deny their request then this will turn into one of the above two states, so you probably don't need to worry about handling it specially. Putting all this together -- If your I/O is organized around a "pull" strategy, where your code requests events as its ready to handle them (e.g. classic synchronous code, or asyncio's ``await loop.sock_recv(...)``, or `Trio's streams `__), then you'll probably want logic that looks something like: .. code-block:: python # Replace do_sendall and do_recv with your I/O code def get_next_event(): while True: event = conn.next_event() if event is h11.NEED_DATA: if conn.they_are_waiting_for_100_continue: do_sendall(conn, h11.InformationalResponse(100, ...)) conn.receive_data(do_recv()) continue return event And then your code that calls this will need to make sure to call it only at appropriate times (e.g., not immediately after receiving :class:`EndOfMessage` or :data:`PAUSED`). If your I/O is organized around a "push" strategy, where the network drives processing (e.g. you're using `Twisted `_, or implementing an :class:`asyncio.Protocol`), then you'll want to internally apply back-pressure whenever you see :data:`PAUSED`, remove back-pressure when you call :meth:`~Connection.start_next_cycle`, and otherwise just deliver events as they arrive. Something like: .. code-block:: python class HTTPProtocol(asyncio.Protocol): # Save the transport for later -- needed to access the # backpressure API. def connection_made(self, transport): self._transport = transport # Internal helper function -- deliver all pending events def _deliver_events(self): while True: event = self.conn.next_event() if event is h11.NEED_DATA: break elif event is h11.PAUSED: # Apply back-pressure self._transport.pause_reading() break else: self.event_received(event) # Called by "someone" whenever new data appears on our socket def data_received(self, data): self.conn.receive_data(data) self._deliver_events() # Called by "someone" whenever the peer closes their socket def eof_received(self): self.conn.receive_data(b"") self._deliver_events() # asyncio will close our socket unless we return True here. return True # Called by your code when its ready to start a new # request/response cycle def start_next_cycle(self): self.conn.start_next_cycle() # New events might have been buffered internally, and only # become deliverable after calling start_next_cycle self._deliver_events() # Remove back-pressure self._transport.resume_reading() # Fill in your code here def event_received(self, event): ... And your code that uses this will have to remember to check for :attr:`~Connection.they_are_waiting_for_100_continue` at the appropriate time. .. _closing: Closing connections ------------------- h11 represents a connection shutdown with the special event type :class:`ConnectionClosed`. You can send this event, in which case :meth:`~Connection.send` will simply update the state machine and then return ``None``. You can receive this event, if you call ``conn.receive_data(b"")``. (The actual receipt might be delayed if the connection is :ref:`paused `.) It's safe and legal to call ``conn.receive_data(b"")`` multiple times, and once you've done this once, then all future calls to :meth:`~Connection.receive_data` will also return ``ConnectionClosed()``: .. ipython:: python conn = h11.Connection(our_role=h11.CLIENT) conn.receive_data(b"") conn.receive_data(b"") conn.receive_data(None) (Or if you try to actually pass new data in after calling ``conn.receive_data(b"")``, that will raise an exception.) h11 is careful about interpreting connection closure in a *half-duplex fashion*. TCP sockets pretend to be a two-way connection, but really they're two one-way connections. In particular, it's possible for one party to shut down their sending connection -- which causes the other side to be notified that the connection has closed via the usual ``socket.recv(...) -> b""`` mechanism -- while still being able to read from their receiving connection. (On Unix, this is generally accomplished via the ``shutdown(2)`` system call.) So, for example, a client could send a request, and then close their socket for writing to indicate that they won't be sending any more requests, and then read the response. It's this kind of closure that is indicated by h11's :class:`ConnectionClosed`: it means that this party will not be sending any more data -- nothing more, nothing less. You can see this reflected in the :ref:`state machine `, in which one party transitioning to :data:`CLOSED` doesn't immediately halt the connection, but merely prevents it from continuing for another request/response cycle. The state machine also indicates that :class:`ConnectionClosed` events can only happen in certain states. This isn't true, of course -- any party can close their connection at any time, and h11 can't stop them. But what h11 can do is distinguish between clean and unclean closes. For example, if both sides complete a request/response cycle and then close the connection, that's a clean closure and everyone will transition to the :data:`CLOSED` state in an orderly fashion. On the other hand, if one party suddenly closes the connection while they're in the middle of sending a chunked response body, or when they promised a ``Content-Length:`` of 1000 bytes but have only sent 500, then h11 knows that this is a violation of the HTTP protocol, and will raise a :exc:`ProtocolError`. Basically h11 treats an unexpected close the same way it would treat unexpected, uninterpretable data arriving -- it lets you know that something has gone wrong. As a client, the proper way to perform a single request and then close the connection is: 1) Send a :class:`Request` with ``Connection: close`` 2) Send the rest of the request body 3) Read the server's :class:`Response` and body 4) ``conn.our_state is h11.MUST_CLOSE`` will now be true. Call ``conn.send(ConnectionClosed())`` and then close the socket. Or really you could just close the socket -- the thing calling ``send`` will do is raise an error if you're not in :data:`MUST_CLOSE` as expected. So it's between you and your conscience and your code reviewers. (Technically it would also be legal to shutdown your socket for writing as step 2.5, but this doesn't serve any purpose and some buggy servers might get annoyed, so it's not recommended.) As a server, the proper way to perform a response is: 1) Send your :class:`Response` and body 2) Check if ``conn.our_state is h11.MUST_CLOSE``. This might happen for a variety of reasons; for example, if the response had unknown length and the client speaks only HTTP/1.0, then the client will not consider the connection complete until we issue a close. You should be particularly careful to take into consideration the following note fromx `RFC 7230 section 6.6 `_: If a server performs an immediate close of a TCP connection, there is a significant risk that the client will not be able to read the last HTTP response. If the server receives additional data from the client on a fully closed connection, such as another request that was sent by the client before receiving the server's response, the server's TCP stack will send a reset packet to the client; unfortunately, the reset packet might erase the client's unacknowledged input buffers before they can be read and interpreted by the client's HTTP parser. To avoid the TCP reset problem, servers typically close a connection in stages. First, the server performs a half-close by closing only the write side of the read/write connection. The server then continues to read from the connection until it receives a corresponding close by the client, or until the server is reasonably certain that its own TCP stack has received the client's acknowledgement of the packet(s) containing the server's last response. Finally, the server fully closes the connection. .. _switching-protocols: Switching protocols ------------------- h11 supports two kinds of "protocol switches": requests with method ``CONNECT``, and the newer ``Upgrade:`` header, most commonly used for negotiating WebSocket connections. Both follow the same pattern: the client proposes that they switch from regular HTTP to some other kind of interaction, and then the server either rejects the suggestion -- in which case we return to regular HTTP rules -- or else accepts it. (For ``CONNECT``, acceptance means a response with 2xx status code; for ``Upgrade:``, acceptance means an :class:`InformationalResponse` with status ``101 Switching Protocols``) If the proposal is accepted, then both sides switch to doing something else with their socket, and h11's job is done. As a developer using h11, it's your responsibility to send and interpret the actual ``CONNECT`` or ``Upgrade:`` request and response, and to figure out what to do after the handover; it's h11's job to understand what's going on, and help you make the handover smoothly. Specifically, what h11 does is :ref:`pause ` parsing incoming data at the boundary between the two protocols, and then you can retrieve any unprocessed data from the :attr:`Connection.trailing_data` attribute. .. _sendfile: Support for ``sendfile()`` -------------------------- Many networking APIs provide some efficient way to send particular data, e.g. asking the operating system to stream files directly off of the disk and into a socket without passing through userspace. It's possible to use these APIs together with h11. The basic strategy is: * Create some placeholder object representing the special data, that your networking code knows how to "send" by invoking whatever the appropriate underlying APIs are. * Make sure your placeholder object implements a ``__len__`` method returning its size in bytes. * Call ``conn.send_with_data_passthrough(Data(data=))`` * This returns a list whose contents are a mixture of (a) bytes-like objects, and (b) your placeholder object. You should send them to the network in order. Here's a sketch of what this might look like: .. code-block:: python class FilePlaceholder: def __init__(self, file, offset, count): self.file = file self.offset = offset self.count = count def __len__(self): return self.count def send_data(sock, data): if isinstance(data, FilePlaceholder): # socket.sendfile added in Python 3.5 sock.sendfile(data.file, data.offset, data.count) else: # data is a bytes-like object to be sent directly sock.sendall(data) placeholder = FilePlaceholder(open("...", "rb"), 0, 200) for data in conn.send_with_data_passthrough(Data(data=placeholder)): send_data(sock, data) This works with all the different framing modes (``Content-Length``, ``Transfer-Encoding: chunked``, etc.) -- h11 will add any necessary framing data, update its internal state, and away you go. Identifying h11 in requests and responses ----------------------------------------- According to RFC 7231, client requests are supposed to include a ``User-Agent:`` header identifying what software they're using, and servers are supposed to respond with a ``Server:`` header doing the same. h11 doesn't construct these headers for you, but to make it easier for you to construct this header, it provides: .. data:: PRODUCT_ID A string suitable for identifying the current version of h11 in a ``User-Agent:`` or ``Server:`` header. The version of h11 that was used to build these docs identified itself as: .. ipython:: python h11.PRODUCT_ID .. _chunk-delimiters-are-bad: Chunked Transfer Encoding Delimiters ------------------------------------ .. versionadded:: 0.7.0 HTTP/1.1 allows for the use of Chunked Transfer Encoding to frame request and response bodies. This form of transfer encoding allows the implementation to provide its body data in the form of length-prefixed "chunks" of data. RFC 7230 is extremely clear that the breaking points between chunks of data are non-semantic: that is, users should not rely on them or assign any meaning to them. This is particularly important given that RFC 7230 also allows intermediaries such as proxies and caches to change the chunk boundaries as they see fit, or even to remove the chunked transfer encoding entirely. However, for some applications it is valuable or essential to see the chunk boundaries because the peer implementation has assigned meaning to them. While this is against the specification, if you do really need access to this information h11 makes it available to you in the form of the :data:`Data.chunk_start` and :data:`Data.chunk_end` properties of the :class:`Data` event. :data:`Data.chunk_start` is set to ``True`` for the first :class:`Data` event for a given chunk of data. :data:`Data.chunk_end` is set to ``True`` for the last :class:`Data` event that is emitted for a given chunk of data. h11 guarantees that it will always emit at least one :class:`Data` event for each chunk of data received from the remote peer, but due to its internal buffering logic it may return more than one. It is possible for a single :class:`Data` event to have both :data:`Data.chunk_start` and :data:`Data.chunk_end` set to ``True``, in which case it will be the only :class:`Data` event for that chunk of data. Again, it is *strongly encouraged* that you avoid relying on this information if at all possible. This functionality should be considered an escape hatch for when there is no alternative but to rely on the information, rather than a general source of data that is worth relying on. h11-0.13.0/docs/source/basic-usage.rst000066400000000000000000000305271417207257000173450ustar00rootroot00000000000000Getting started: Writing your own HTTP/1.1 client ================================================= .. currentmodule:: h11 h11 can be used to implement both HTTP/1.1 clients and servers. To give a flavor for how the API works, we'll demonstrate a small client. HTTP basics ----------- An HTTP interaction always starts with a client sending a *request*, optionally some *data* (e.g., a POST body); and then the server responds with a *response* and optionally some *data* (e.g. the requested document). Requests and responses have some data associated with them: for requests, this is a method (e.g. ``GET``), a target (e.g. ``/index.html``), and a collection of headers (e.g. ``User-agent: demo-clent``). For responses, it's a status code (e.g. 404 Not Found) and a collection of headers. Of course, as far as the network is concerned, there's no such thing as "requests" and "responses" -- there's just bytes being sent from one computer to another. Let's see what this looks like, by fetching https://httpbin.org/xml: .. ipython:: python import ssl, socket ctx = ssl.create_default_context() sock = ctx.wrap_socket(socket.create_connection(("httpbin.org", 443)), server_hostname="httpbin.org") # Send request sock.sendall(b"GET /xml HTTP/1.1\r\nhost: httpbin.org\r\n\r\n") # Read response response_data = sock.recv(1024) # Let's see what we got! print(response_data) .. warning:: If you try to reproduce these examples interactively, then you'll have the most luck if you paste them in all at once. Remember we're talking to a remote server here – if you type them in one at a time, and you're too slow, then the server might give up on waiting for you and close the connection. One way to recognize that this has happened is if ``response_data`` comes back as an empty string, or later on when we're working with h11 this might cause errors that mention ``ConnectionClosed``. So that's, uh, very convenient and readable. It's a little more understandable if we print the bytes as text: .. ipython:: python print(response_data.decode("ascii")) Here we can see the status code at the top (200, which is the code for "OK"), followed by the headers, followed by the data (a silly little XML document). But we can already see that working with bytes by hand like this is really cumbersome. What we need to do is to move up to a higher level of abstraction. This is what h11 does. Instead of talking in bytes, it lets you talk in high-level HTTP "events". To see what this means, let's repeat the above exercise, but using h11. We start by making a TLS connection like before, but now we'll also import :mod:`h11`, and create a :class:`h11.Connection` object: .. ipython:: python import ssl, socket import h11 ctx = ssl.create_default_context() sock = ctx.wrap_socket(socket.create_connection(("httpbin.org", 443)), server_hostname="httpbin.org") conn = h11.Connection(our_role=h11.CLIENT) Next, to send an event to the server, there are three steps we have to take. First, we create an object representing the event we want to send -- in this case, a :class:`h11.Request`: .. ipython:: python request = h11.Request(method="GET", target="/xml", headers=[("Host", "httpbin.org")]) Next, we pass this to our connection's :meth:`~Connection.send` method, which gives us back the bytes corresponding to this message: .. ipython:: python bytes_to_send = conn.send(request) And then we send these bytes across the network: .. ipython:: python sock.sendall(bytes_to_send) There's nothing magical here -- these are the same bytes that we sent up above: .. ipython:: python bytes_to_send Why doesn't h11 go ahead and send the bytes for you? Because it's designed to be usable no matter what socket API you're using -- doesn't matter if it's synchronous like this, asynchronous, callback-based, whatever; if you can read and write bytes from the network, then you can use h11. In this case, we're not quite done yet -- we have to send another event to tell the other side that we're finished, which we do by sending an :class:`EndOfMessage` event: .. ipython:: python end_of_message_bytes_to_send = conn.send(h11.EndOfMessage()) sock.sendall(end_of_message_bytes_to_send) Of course, it turns out that in this case, the HTTP/1.1 specification tells us that any request that doesn't contain either a ``Content-Length`` or ``Transfer-Encoding`` header automatically has a 0 length body, and h11 knows that, and h11 knows that the server knows that, so it actually encoded the :class:`EndOfMessage` event as the empty string: .. ipython:: python end_of_message_bytes_to_send But there are other cases where it might not, depending on what headers are set, what message is being responded to, the HTTP version of the remote peer, etc. etc. So for consistency, h11 requires that you *always* finish your messages by sending an explicit :class:`EndOfMessage` event; then it keeps track of the details of what that actually means in any given situation, so that you don't have to. Finally, we have to read the server's reply. By now you can probably guess how this is done, at least in the general outline: we read some bytes from the network, then we hand them to the connection (using :meth:`Connection.receive_data`) and it converts them into events (using :meth:`Connection.next_event`). .. ipython:: python bytes_received = sock.recv(1024) conn.receive_data(bytes_received) conn.next_event() conn.next_event() conn.next_event() (Remember, if you're following along and get an error here mentioning ``ConnectionClosed``, then try again, but going through the steps faster!) Here the server sent us three events: a :class:`Response` object, which is similar to the :class:`Request` object that we created earlier and has the response's status code (200 OK) and headers; a :class:`Data` object containing the response data; and another :class:`EndOfMessage` object. This similarity between what we send and what we receive isn't accidental: if we were using h11 to write an HTTP server, then these are the objects we would have created and passed to :meth:`~Connection.send` -- h11 in client and server mode has an API that's almost exactly symmetric. One thing we have to deal with, though, is that an entire response doesn't always arrive in a single call to :meth:`socket.recv` -- sometimes the network will decide to trickle it in at its own pace, in multiple pieces. Let's try that again: .. ipython:: python import ssl, socket import h11 ctx = ssl.create_default_context() sock = ctx.wrap_socket(socket.create_connection(("httpbin.org", 443)), server_hostname="httpbin.org") conn = h11.Connection(our_role=h11.CLIENT) request = h11.Request(method="GET", target="/xml", headers=[("Host", "httpbin.org")]) sock.sendall(conn.send(request)) and this time, we'll read in chunks of 200 bytes, to see how h11 handles it: .. ipython:: python bytes_received = sock.recv(200) conn.receive_data(bytes_received) conn.next_event() :data:`NEED_DATA` is a special value that indicates that we, well, need more data. h11 has buffered the first chunk of data; let's read some more: .. ipython:: python bytes_received = sock.recv(200) conn.receive_data(bytes_received) conn.next_event() Now it's managed to read a complete :class:`Request`. A basic client object --------------------- Now let's use what we've learned to wrap up our socket and :class:`Connection` into a single object with some convenience methods: .. literalinclude:: _examples/myclient.py .. ipython:: python :suppress: import sys with open(sys._h11_hack_docs_source_path + "/_examples/myclient.py") as f: exec(f.read()) And then we can send requests: .. ipython:: python client = MyHttpClient("httpbin.org", 443) client.send(h11.Request(method="GET", target="/xml", headers=[("Host", "httpbin.org")])) client.send(h11.EndOfMessage()) And read back the events: .. ipython:: python client.next_event() client.next_event() Note here that we received a :class:`Data` event that only has *part* of the response body -- this is another consequence of our reading in small chunks. h11 tries to buffer as little as it can, so it streams out data as it arrives, which might mean that a message body might be split up into multiple :class:`Data` events. (Of course, if you're the one sending data, you can do the same thing: instead of buffering all your data in one giant :class:`Data` event, you can send multiple :class:`Data` events yourself to stream the data out incrementally; just make sure that you set the appropriate ``Content-Length`` / ``Transfer-Encoding`` headers.) If we keep reading, we'll see more :class:`Data` events, and then eventually the :class:`EndOfMessage`: .. ipython:: python client.next_event() client.next_event() client.next_event() Now we can see why :class:`EndOfMessage` is so important -- otherwise, we can't tell when we've received the end of the data. And since that's the end of this response, the server won't send us anything more until we make another request -- if we try, then the socket read will just hang forever, unless we set a timeout or interrupt it: .. ipython:: python :okexcept: client.sock.settimeout(2) client.next_event() Keep-alive ---------- For some servers, we'd have to stop here, because they require a new connection for every request/response. But, this server is smarter than that -- it supports `keep-alive `_, so we can re-use this connection to send another request. There's a few ways we can tell. First, if it didn't, then it would have closed the connection already, and we would have gotten a :class:`ConnectionClosed` event on our last call to :meth:`~Connection.next_event`. We can also tell by checking h11's internal idea of what state the two sides of the conversation are in: .. ipython:: python client.conn.our_state, client.conn.their_state If the server didn't support keep-alive, then these would be :data:`MUST_CLOSE` and either :data:`MUST_CLOSE` or :data:`CLOSED`, respectively (depending on whether we'd seen the socket actually close yet). :data:`DONE` / :data:`DONE`, on the other hand, means that this request/response cycle has totally finished, but the connection itself is still viable, and we can start over and send a new request on this same connection. To do this, we tell h11 to get ready (this is needed as a safety measure to make sure different requests/responses on the same connection don't get accidentally mixed up): .. ipython:: python client.conn.start_next_cycle() This resets both sides back to their initial :data:`IDLE` state, allowing us to send another :class:`Request`: .. ipython:: python client.conn.our_state, client.conn.their_state client.send(h11.Request(method="GET", target="/get", headers=[("Host", "httpbin.org")])) client.send(h11.EndOfMessage()) client.next_event() What's next? ------------ Here's some ideas of things you might try: * Adapt the above examples to make a POST request. (Don't forget to set the ``Content-Length`` header -- but don't worry, if you do forget, then h11 will give you an error when you try to send data): .. code-block:: python client.send(h11.Request(method="POST", target="/post", headers=[("Host", "httpbin.org"), ("Content-Length", "10")])) client.send(h11.Data(data=b"1234567890")) client.send(h11.EndOfMessage()) * Experiment with what happens if you try to violate the HTTP protocol by sending a :class:`Response` as a client, or sending two :class:`Request`\s in a row. * Write your own basic ``http_get`` function that takes a URL, parses out the host/port/path, then connects to the server, does a ``GET`` request, and then collects up all the resulting :class:`Data` objects, concatenates their payloads, and returns it. * Adapt the above code to use your favorite non-blocking API * Use h11 to write a simple HTTP server. (If you get stuck, `here's an example `_.) And of course, you'll want to read the :ref:`API-documentation` for all the details. h11-0.13.0/docs/source/changes.rst000066400000000000000000000210701417207257000165630ustar00rootroot00000000000000History of changes ================== .. currentmodule:: h11 .. towncrier release notes start v0.13.0 (2022-01-19) -------------------- Features ~~~~~~~~ - Clarify that the Headers class is a Sequence and inherit from the collections Sequence abstract base class to also indicate this (and gain the mixin methods). See also #104. (`#112 `__) - Switch event classes to dataclasses for easier typing and slightly improved performance. (`#124 `__) - Shorten traceback of protocol errors for easier readability (`#132 `__). - Add typing including a PEP 561 marker for usage by type checkers (`#135 `__). - Expand the allowed status codes to [0, 999] from [0, 600] (`#134 https://github.com/python-hyper/h11/issues/134`__). Backwards **in**\compatible changes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - Ensure request method is a valid token (`#141 https://github.com/python-hyper/h11/pull/141>`__). v0.12.0 (2021-01-01) -------------------- Features ~~~~~~~~ - Added support for servers with broken line endings. After this change h11 accepts both ``\r\n`` and ``\n`` as a headers delimiter. (`#7 `__) - Add early detection of invalid http data when request line starts with binary (`#122 `__) Deprecations and Removals ~~~~~~~~~~~~~~~~~~~~~~~~~ - Python 2.7 and PyPy 2 support is removed. h11 now requires Python>=3.6 including PyPy 3. Users running `pip install h11` on Python 2 will automatically get the last Python 2-compatible version. (`#114 `__) v0.11.0 (2020-10-05) -------------------- New features: * h11 now stores and makes available the raw header name as received. In addition h11 will write out header names with the same casing as passed to it. This allows compatibility with systems that expect titlecased header names. See `#31 `__. * Multiple content length headers are now merged into a single header if all the values are equal, if any are unequal a LocalProtocol error is raised (as before). See `#92 `__. Backwards **in**\compatible changes: * Headers added by h11, rather than passed to it, now have titlecased names. Whilst this should help compatibility it replaces the previous lowercased header names. v0.10.0 (2020-08-14) -------------------- Other changes: * Drop support for Python 3.4. * Support Python 3.8. * Make error messages returned by match failures less ambiguous (`#98 `__). v0.9.0 (2019-05-15) ------------------- Bug fixes: * Allow a broader range of characters in header values. This violates the RFC, but is apparently required for compatibility with real-world code, like Google Analytics cookies (`#57 `__, `#58 `__). * Validate incoming and outgoing request paths for invalid characters. This prevents a variety of potential security issues that have affected other HTTP clients. (`#69 `__). * Force status codes to be integers, thereby allowing stdlib HTTPStatus IntEnums to be used when constructing responses (`#72 `__). Other changes: * Make all sentinel values inspectable by IDEs, and split ``SEND_BODY_DONE`` into ``SEND_BODY``, and ``DONE`` (`#75 `__). * Drop support for Python 3.3. * LocalProtocolError raised in start_next_cycle now shows states for more informative errors (`#80 `__). v0.8.1 (2018-04-14) ------------------- Bug fixes: * Always return headers as ``bytes`` objects (`#60 `__) Other changes: * Added proper license notices to the Javascript used in our documentation (`#61 `__) v0.8.0 (2018-03-20) ------------------- Backwards **in**\compatible changes: * h11 now performs stricter validation on outgoing header names and header values: illegal characters are now rejected (example: you can't put a newline into an HTTP header), and header values with leading/trailing whitespace are also rejected (previously h11 would silently discard the whitespace). All these checks were already performed on incoming headers; this just extends that to outgoing headers. New features: * New method :meth:`Connection.send_failed`, to notify a :class:`Connection` object when data returned from :meth:`Connection.send` was *not* sent. Bug fixes: * Make sure that when computing the framing headers for HEAD responses, we produce the same results as we would for the corresponding GET. * Error out if a request has multiple Host: headers. * Send the Host: header first, as recommended by RFC 7230. * The Expect: header `is case-insensitive `__, so use case-insensitive matching when looking for 100-continue. Other changes: * Better error messages in several cases. * Provide correct ``error_status_hint`` in exception raised when encountering an invalid ``Transfer-Encoding`` header. * For better compatibility with broken servers, h11 now tolerates responses where the reason phrase is missing (not just empty). * Various optimizations and documentation improvements. v0.7.0 (2016-11-25) ------------------- New features (backwards compatible): * Made it so that sentinels are :ref:`instances of themselves `, to enable certain dispatch tricks on the return value of :func:`Connection.next_event` (see `issue #8 `__ for discussion). * Added :data:`Data.chunk_start` and :data:`Data.chunk_end` properties to the :class:`Data` event. These provide the user information about where chunk delimiters are in the data stream from the remote peer when chunked transfer encoding is in use. You :ref:`probably shouldn't use these `, but sometimes there's no alternative (see `issue #19 `__ for discussion). * Expose :data:`Response.reason` attribute, making it possible to read or set the textual "reason phrase" on responses (`issue #13 `__). Bug fixes: * Fix the error message given when a call to an event constructor is missing a required keyword argument (`issue #14 `__). * Fixed encoding of empty :class:`Data` events (``Data(data=b"")``) when using chunked encoding (`issue #21 `__). v0.6.0 (2016-10-24) ------------------- This is the first release since we started using h11 to write non-trivial server code, and this experience triggered a number of substantial API changes. Backwards **in**\compatible changes: * Split the old :meth:`receive_data` into the new :meth:`~Connection.receive_data` and :meth:`~Connection.next_event`, and replaced the old :class:`Paused` pseudo-event with the new :data:`NEED_DATA` and :data:`PAUSED` sentinels. * Simplified the API by replacing the old :meth:`Connection.state_of`, :attr:`Connection.client_state`, :attr:`Connection.server_state` with the new :attr:`Connection.states`. * Renamed the old :meth:`prepare_to_reuse` to the new :meth:`~Connection.start_next_cycle`. * Removed the ``Paused`` pseudo-event. Backwards compatible changes: * State machine: added a :data:`DONE` -> :data:`MUST_CLOSE` transition triggered by our peer being in the :data:`ERROR` state. * Split :exc:`ProtocolError` into :exc:`LocalProtocolError` and :exc:`RemoteProtocolError` (see :ref:`error-handling`). Use case: HTTP servers want to be able to distinguish between an error that originates locally (which produce a 500 status code) versus errors caused by remote misbehavior (which produce a 4xx status code). * Changed the :data:`PRODUCT_ID` from ``h11/`` to ``python-h11/``. (This is similar to what requests uses, and much more searchable than plain h11.) Other changes: * Added a minimal benchmark suite, and used it to make a few small optimizations (maybe ~20% speedup?). v0.5.0 (2016-05-14) ------------------- * Initial release. h11-0.13.0/docs/source/conf.py000066400000000000000000000247251417207257000157320ustar00rootroot00000000000000#!/usr/bin/env python3 # -*- coding: utf-8 -*- # # h11 documentation build configuration file, created by # sphinx-quickstart on Tue May 3 00:20:14 2016. # # This file is execfile()d with the current directory set to its # containing dir. # # Note that not all possible configuration values are present in this # autogenerated file. # # All configuration values have a default; values that are commented out # serve to show the default. import sys import os ################################################################ # hack hack # # The live ipython examples want to know where the docs source/ directory is, # so that they can find files that live there. # # There's no guarantee that our CWD == the source directory, but conf.py # *does* know what directory it lives in, so it can stash that in a public # place where the later code can find it. # # (In particular, the sphinx Makefile runs sphinx-build from a different # directory -- but RTD runs sphinx-build directly from inside the source/ # directory, so there's no single value of this that works for both.) # import os.path sys._h11_hack_docs_source_path = os.path.dirname(__file__) ################################################################ # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. #sys.path.insert(0, os.path.abspath('.')) # -- General configuration ------------------------------------------------ # If your documentation needs a minimal Sphinx version, state it here. #needs_sphinx = '1.0' # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = [ 'sphinx.ext.autodoc', 'sphinx.ext.intersphinx', 'sphinx.ext.viewcode', 'sphinx.ext.napoleon', 'IPython.sphinxext.ipython_directive', 'IPython.sphinxext.ipython_console_highlighting', ] # Undocumented trick: if we def setup here in conf.py, it gets called just # like an extension's setup function. def setup(app): app.add_javascript("show-code.js") app.add_javascript("facebox.js") app.add_stylesheet("facebox.css") # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] # The suffix(es) of source filenames. # You can specify multiple suffix as a list of string: # source_suffix = ['.rst', '.md'] source_suffix = '.rst' # The encoding of source files. #source_encoding = 'utf-8-sig' # The master toctree document. master_doc = 'index' # General information about the project. project = 'h11' copyright = '2016, Nathaniel J. Smith' author = 'Nathaniel J. Smith' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # The short X.Y version. import h11 version = h11.__version__ # The full version, including alpha/beta/rc tags. release = h11.__version__ # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. # # This is also used if you do content translation via gettext catalogs. # Usually you set "language" from the command line for these cases. language = None # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: #today = '' # Else, today_fmt is used as the format for a strftime call. #today_fmt = '%B %d, %Y' # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This patterns also effect to html_static_path and html_extra_path exclude_patterns = [] # The reST default role (used for this markup: `text`) to use for all # documents. #default_role = None # If true, '()' will be appended to :func: etc. cross-reference text. #add_function_parentheses = True # If true, the current module name will be prepended to all description # unit titles (such as .. function::). #add_module_names = True # If true, sectionauthor and moduleauthor directives will be shown in the # output. They are ignored by default. #show_authors = False # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'sphinx' # A list of ignored prefixes for module index sorting. #modindex_common_prefix = [] # If true, keep warnings as "system message" paragraphs in the built documents. #keep_warnings = False # If true, `todo` and `todoList` produce output, else they produce nothing. todo_include_todos = False # -- Options for HTML output ---------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. # html_theme = 'alabaster' # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the # documentation. #html_theme_options = {} # Add any paths that contain custom themes here, relative to this directory. #html_theme_path = [] # The name for this set of Sphinx documents. # " v documentation" by default. #html_title = 'h11 v0.0.1' # A shorter title for the navigation bar. Default is the same as html_title. #html_short_title = None # The name of an image file (relative to this directory) to place at the top # of the sidebar. #html_logo = None # The name of an image file (relative to this directory) to use as a favicon of # the docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 # pixels large. #html_favicon = None # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ['_static'] # Add any extra paths that contain custom files (such as robots.txt or # .htaccess) here, relative to this directory. These files are copied # directly to the root of the documentation. #html_extra_path = [] # If not None, a 'Last updated on:' timestamp is inserted at every page # bottom, using the given strftime format. # The empty string is equivalent to '%b %d, %Y'. #html_last_updated_fmt = None # If true, SmartyPants will be used to convert quotes and dashes to # typographically correct entities. #html_use_smartypants = True # Custom sidebar templates, maps document names to template names. #html_sidebars = {} # Additional templates that should be rendered to pages, maps page names to # template names. #html_additional_pages = {} # If false, no module index is generated. #html_domain_indices = True # If false, no index is generated. #html_use_index = True # If true, the index is split into individual pages for each letter. #html_split_index = False # If true, links to the reST sources are added to the pages. #html_show_sourcelink = True # If true, "Created using Sphinx" is shown in the HTML footer. Default is True. #html_show_sphinx = True # If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. #html_show_copyright = True # If true, an OpenSearch description file will be output, and all pages will # contain a tag referring to it. The value of this option must be the # base URL from which the finished HTML is served. #html_use_opensearch = '' # This is the file name suffix for HTML files (e.g. ".xhtml"). #html_file_suffix = None # Language to be used for generating the HTML full-text search index. # Sphinx supports the following languages: # 'da', 'de', 'en', 'es', 'fi', 'fr', 'h', 'it', 'ja' # 'nl', 'no', 'pt', 'ro', 'r', 'sv', 'tr', 'zh' #html_search_language = 'en' # A dictionary with options for the search language support, empty by default. # 'ja' uses this config value. # 'zh' user can custom change `jieba` dictionary path. #html_search_options = {'type': 'default'} # The name of a javascript file (relative to the configuration directory) that # implements a search results scorer. If empty, the default will be used. #html_search_scorer = 'scorer.js' # Output file base name for HTML help builder. htmlhelp_basename = 'h11doc' # -- Options for LaTeX output --------------------------------------------- latex_elements = { # The paper size ('letterpaper' or 'a4paper'). #'papersize': 'letterpaper', # The font size ('10pt', '11pt' or '12pt'). #'pointsize': '10pt', # Additional stuff for the LaTeX preamble. #'preamble': '', # Latex figure (float) alignment #'figure_align': 'htbp', } # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, # author, documentclass [howto, manual, or own class]). latex_documents = [ (master_doc, 'h11.tex', 'h11 Documentation', 'Nathaniel J. Smith', 'manual'), ] # The name of an image file (relative to this directory) to place at the top of # the title page. #latex_logo = None # For "manual" documents, if this is true, then toplevel headings are parts, # not chapters. #latex_use_parts = False # If true, show page references after internal links. #latex_show_pagerefs = False # If true, show URL addresses after external links. #latex_show_urls = False # Documents to append as an appendix to all manuals. #latex_appendices = [] # If false, no module index is generated. #latex_domain_indices = True # -- Options for manual page output --------------------------------------- # One entry per manual page. List of tuples # (source start file, name, description, authors, manual section). man_pages = [ (master_doc, 'h11', 'h11 Documentation', [author], 1) ] # If true, show URL addresses after external links. #man_show_urls = False # -- Options for Texinfo output ------------------------------------------- # Grouping the document tree into Texinfo files. List of tuples # (source start file, target name, title, author, # dir menu entry, description, category) texinfo_documents = [ (master_doc, 'h11', 'h11 Documentation', author, 'h11', 'One line description of project.', 'Miscellaneous'), ] # Documents to append as an appendix to all manuals. #texinfo_appendices = [] # If false, no module index is generated. #texinfo_domain_indices = True # How to display URL addresses: 'footnote', 'no', or 'inline'. #texinfo_show_urls = 'footnote' # If true, do not generate a @detailmenu in the "Top" node's menu. #texinfo_no_detailmenu = False # Example configuration for intersphinx: refer to the Python standard library. intersphinx_mapping = { 'python': ('https://docs.python.org/3.5', None), } h11-0.13.0/docs/source/examples.rst000066400000000000000000000011661417207257000167750ustar00rootroot00000000000000Examples ======== .. If we add any more examples then we should probably split this out into separate pages for each example You can also find these in the `examples/ directory of a source checkout `_. Minimal client, using synchronous I/O ------------------------------------- .. literalinclude:: ../../examples/basic-client.py :language: python Fairly complete server with error handling, using Trio for async I/O -------------------------------------------------------------------- .. literalinclude:: ../../examples/trio-server.py :language: python h11-0.13.0/docs/source/index.rst000066400000000000000000000054751417207257000162750ustar00rootroot00000000000000h11: A pure-Python HTTP/1.1 protocol library ============================================ h11 is an HTTP/1.1 protocol library written in Python, heavily inspired by `hyper-h2 `_. h11's goal is to be a simple, robust, complete, and non-hacky implementation of the first "chapter" of the HTTP/1.1 spec: `RFC 7230: HTTP/1.1 Message Syntax and Routing `_. That is, it mostly focuses on implementing HTTP at the level of taking bytes on and off the wire, and the headers related to that, and tries to be picky about spec conformance when possible. It doesn't know about higher-level concerns like URL routing, conditional GETs, cross-origin cookie policies, or content negotiation. But it does know how to take care of framing, cross-version differences in keep-alive handling, and the "obsolete line folding" rule, and to use bounded time and space to process even pathological / malicious input, so that you can focus your energies on the hard / interesting parts for your application. And it tries to support the full specification in the sense that any useful HTTP/1.1 conformant application should be able to use h11. This is a "bring-your-own-I/O" protocol library; like h2, it contains no I/O code whatsoever. This means you can hook h11 up to your favorite network API, and that could be anything you want: synchronous, threaded, asynchronous, or your own implementation of `RFC 6214 `_ -- h11 won't judge you. This is h11's main feature compared to the current state of the art, where every HTTP library is tightly bound to a particular network framework, and every time a `new network API `_ comes along then someone has to start over reimplementing the entire HTTP stack from scratch. We highly recommend `Cory Benfield's excellent blog post about the advantages of this approach `_. This also means that h11 is not immediately useful out of the box: it's a toolkit for building programs that speak HTTP, not something that could directly replace ``requests`` or ``twisted.web`` or whatever. But h11 makes it much easier to implement something like ``requests`` or ``twisted.web``. Vital statistics ---------------- * Requirements: Python 3.6+ (PyPy works great) The last Python 2-compatible version was h11 0.11.x. * Install: ``pip install h11`` * Sources and bug tracker: https://github.com/python-hyper/h11 * Docs: https://h11.readthedocs.io * License: MIT * Code of conduct: Contributors are requested to follow our `code of conduct `_ in all project spaces. Contents -------- .. toctree:: :maxdepth: 2 basic-usage.rst api.rst examples.rst supported-http.rst changes.rst h11-0.13.0/docs/source/make-state-diagrams.py000066400000000000000000000144371417207257000206240ustar00rootroot00000000000000#!python import sys sys.path.append("../..") import os.path import subprocess from h11._events import * from h11._state import * from h11._state import ( _SWITCH_UPGRADE, _SWITCH_CONNECT, EVENT_TRIGGERED_TRANSITIONS, STATE_TRIGGERED_TRANSITIONS, ) _EVENT_COLOR = "#002092" _STATE_COLOR = "#017517" _SPECIAL_COLOR = "#7600a1" HEADER = """ digraph { graph [fontname = "Lato" bgcolor="transparent"] node [fontname = "Lato"] edge [fontname = "Lato"] """ def finish(machine_name): return (""" labelloc="t" labeljust="l" label=<h11 state machine: {}> }} """.format(machine_name)) class Edges: def __init__(self): self.edges = [] def e(self, source, target, label, color, italicize=False, weight=1): if italicize: quoted_label = "<{}>".format(label) else: quoted_label = '<{}>'.format(label) self.edges.append( '{source} -> {target} [\n' ' label={quoted_label},\n' ' color="{color}", fontcolor="{color}",\n' ' weight={weight},\n' ']\n' .format(**locals())) def write(self, f): self.edges.sort() f.write("".join(self.edges)) def make_dot_special_state(out_path): with open(out_path, "w") as f: f.write(HEADER) f.write(""" kaT [label=<keep-alive is enabled
initial state>] kaF [label=<keep-alive is disabled>] upF [label=<No potential Upgrade: pending
initial state>] upT [label=<Potential Upgrade: pending>] coF [label=<No potential CONNECT pending
initial state>] coT [label=<Potential CONNECT pending>] """) edges = Edges() for s in ["kaT", "kaF"]: edges.e(s, "kaF", "Request/response with
HTTP/1.0 or Connection: close", color=_EVENT_COLOR, italicize=True) edges.e("upF", "upT", "Request with Upgrade:", color=_EVENT_COLOR, italicize=True) edges.e("upT", "upF", "Response", color=_EVENT_COLOR, italicize=True) edges.e("coF", "coT", "Request with CONNECT", color=_EVENT_COLOR, italicize=True) edges.e("coT", "coF", "Response without 2xx status", color=_EVENT_COLOR, italicize=True) edges.write(f) f.write(finish("special states")) def make_dot(role, out_path): with open(out_path, "w") as f: f.write(HEADER) f.write(""" IDLE [label=start state>] // move ERROR down to the bottom {rank=same CLOSED ERROR} """) # Dot output is sensitive to the order in which the nodes and edges # are listed. We generate them in python's randomized dict iteration # order. So to normalize order, we accumulate and then sort. # Fortunately, this order happens to be one that produces a nice # layout... with other orders I've seen really terrible layouts, and # had to do things like move the server's IDLE->MUST_CLOSE to the top # of the file to fix them. edges = Edges() CORE_EVENTS = {Request, InformationalResponse, Response, Data, EndOfMessage} for (source_state, t) in EVENT_TRIGGERED_TRANSITIONS[role].items(): for (event_type, target_state) in t.items(): weight = 1 color = _EVENT_COLOR italicize = False if (event_type in CORE_EVENTS and source_state is not target_state): weight = 10 # exception if (event_type is Response and source_state is IDLE): weight = 1 if isinstance(event_type, tuple): # The weird special cases #color = _SPECIAL_COLOR if event_type == (Request, CLIENT): name = "client makes Request" weight = 10 elif event_type[1] is _SWITCH_UPGRADE: name = "101 Switching Protocols" weight = 1 elif event_type[1] is _SWITCH_CONNECT: name = "CONNECT accepted" weight = 1 else: assert False else: name = event_type.__name__ edges.e(source_state, target_state, name, color, weight=weight, italicize=italicize) for state_pair, updates in STATE_TRIGGERED_TRANSITIONS.items(): if role not in updates: continue if role is CLIENT: (our_state, their_state) = state_pair else: (their_state, our_state) = state_pair edges.e(our_state, updates[role], "peer in
{}".format(their_state), color=_STATE_COLOR) if role is CLIENT: edges.e(DONE, MIGHT_SWITCH_PROTOCOL, "Potential Upgrade:
or CONNECT pending", _STATE_COLOR, italicize=True) edges.e(MIGHT_SWITCH_PROTOCOL, DONE, "No potential Upgrade:
or CONNECT pending", _STATE_COLOR, italicize=True) edges.e(DONE, MUST_CLOSE, "keep-alive
is disabled", _STATE_COLOR, italicize=True) edges.e(DONE, IDLE, "start_next_cycle()", _SPECIAL_COLOR) edges.write(f) # For some reason labelfontsize doesn't seem to do anything, but this # works f.write(finish(role)) my_dir = os.path.dirname(__file__) out_dir = os.path.join(my_dir, "_static") if not os.path.exists(out_dir): os.path.mkdir(out_dir) for role in (CLIENT, SERVER): dot_path = os.path.join(out_dir, str(role) + ".dot") svg_path = dot_path[:-3] + "svg" make_dot(role, dot_path) subprocess.check_call(["dot", "-Tsvg", dot_path, "-o", svg_path]) dot_path = os.path.join(out_dir, "special-states.dot") svg_path = dot_path[:-3] + "svg" make_dot_special_state(dot_path) subprocess.check_call(["dot", "-Tsvg", dot_path, "-o", svg_path]) h11-0.13.0/docs/source/supported-http.rst000066400000000000000000000100261417207257000201540ustar00rootroot00000000000000Details of our HTTP support for HTTP nerds ========================================== .. currentmodule:: h11 h11 only speaks HTTP/1.1. It can talk to HTTP/1.0 clients and servers, but it itself only does HTTP/1.1. We fully support HTTP/1.1 keep-alive. We have a little bit of support for HTTP/1.1 pipelining -- basically the minimum that's required by the standard. In server mode we can handle pipelined requests in a serial manner, responding completely to each request before reading the next (and our API is designed to make it easy for servers to keep this straight). Client mode doesn't support pipelining at all. As far as I can tell, this matches the state of the art in all the major HTTP implementations: the consensus seems to be that HTTP/1.1 pipelining was a nice try but unworkable in practice, and if you really need pipelining to work then instead of trying to fix HTTP/1.1 you should switch to HTTP/2.0. The HTTP/1.0 ``Connection: keep-alive`` pseudo-standard is currently not supported. (Note that this only affects h11 as a server, because h11 as a client always speaks HTTP/1.1.) Supporting this would be possible, but it's fragile and finicky and I'm suspicious that if we leave it out then no-one will notice or care. HTTP/1.1 is now almost old enough to vote in United States elections. I get that people sometimes write HTTP/1.0 clients because they don't want to deal with annoying stuff like chunked encoding, and I completely sympathize with that, but I'm guessing that you're not going to find too many people these days who care desperately about keep-alive *and at the same time* are too lazy to implement Transfer-Encoding: chunked. Still, this would be my bet as to the missing feature that people are most likely to eventually complain about... Of the headers defined in RFC 7230, the ones h11 knows and has some special-case logic to care about are: ``Connection:``, ``Transfer-Encoding:``, ``Content-Length:``, ``Host:``, ``Upgrade:``, and ``Expect:`` (which is really from `RFC 7231 `_ but whatever). The other headers in RFC 7230 are ``TE:``, ``Trailer:``, and ``Via:``; h11 also supports these in the sense that it ignores them and that's really all it should be doing. Transfer-Encoding support: we only know ``chunked``, not ``gzip`` or ``deflate``. We're in good company in this: node.js at least doesn't handle anything besides ``chunked`` either. So I'm not too worried about this being a problem in practice. But I'm not majorly opposed to adding support for more features here either. A quirk in our :class:`Response` encoding: we don't bother including ascii status messages -- instead of ``200 OK`` we just say ``200``. This is totally legal and no program should care, and it lets us skip carrying around a pointless table of status message strings, but I suppose it might be worth fixing at some point. When parsing chunked encoding, we parse but discard "chunk extensions". This is an extremely obscure feature that allows arbitrary metadata to be interleaved into a chunked transfer stream. This metadata has no standard uses, and proxies are allowed to strip it out. I don't think anyone will notice this lack, but it could be added if someone really wants it; I just ran out of energy for implementing weirdo features no-one uses. Currently we *do* implement support for "obsolete line folding" when reading HTTP headers. This is an optional part of the spec -- conforming HTTP/1.1 implementations MUST NOT send continuation lines, and conforming HTTP/1.1 servers MAY send 400 Bad Request responses back at clients who do send them (`ref `_). I'm tempted to remove this support, since it adds some complicated and ugly code right at the center of the request/response parsing loop, and I'm not sure whether anyone actually needs it. Unfortunately a few major implementations that I spot-checked (node.js, go) do still seem to support reading such headers (but not generating them), so it might or might not be obsolete in practice -- it's hard to know. h11-0.13.0/examples/000077500000000000000000000000001417207257000140075ustar00rootroot00000000000000h11-0.13.0/examples/basic-client.py000066400000000000000000000034521417207257000167220ustar00rootroot00000000000000import socket import ssl import h11 ################################################################ # Setup ################################################################ conn = h11.Connection(our_role=h11.CLIENT) ctx = ssl.create_default_context() sock = ctx.wrap_socket( socket.create_connection(("httpbin.org", 443)), server_hostname="httpbin.org" ) ################################################################ # Sending a request ################################################################ def send(event): print("Sending event:") print(event) print() # Pass the event through h11's state machine and encoding machinery data = conn.send(event) # Send the resulting bytes on the wire sock.sendall(data) send( h11.Request( method="GET", target="/get", headers=[("Host", "httpbin.org"), ("Connection", "close")], ) ) send(h11.EndOfMessage()) ################################################################ # Receiving the response ################################################################ def next_event(): while True: # Check if an event is already available event = conn.next_event() if event is h11.NEED_DATA: # Nope, so fetch some data from the socket... data = sock.recv(2048) # ...and give it to h11 to convert back into events... conn.receive_data(data) # ...and then loop around to try again. continue return event while True: event = next_event() print("Received event:") print(event) print() if type(event) is h11.EndOfMessage: break ################################################################ # Clean up ################################################################ sock.close() h11-0.13.0/examples/trio-server.py000066400000000000000000000332511417207257000166460ustar00rootroot00000000000000# A simple HTTP server implemented using h11 and Trio: # http://trio.readthedocs.io/en/latest/index.html # (so requires python 3.5+). # # All requests get echoed back a JSON document containing information about # the request. # # This is a rather involved example, since it attempts to both be # fully-HTTP-compliant and also demonstrate error handling. # # The main difference between an HTTP client and an HTTP server is that in a # client, if something goes wrong, you can just throw away that connection and # make a new one. In a server, you're expected to handle all kinds of garbage # input and internal errors and recover with grace and dignity. And that's # what this code does. # # I recommend pushing on it to see how it works -- e.g. watch what happens if # you visit http://localhost:8080 in a webbrowser that supports keep-alive, # hit reload a few times, and then wait for the keep-alive to time out on the # server. # # Or try using curl to start a chunked upload and then hit control-C in the # middle of the upload: # # (for CHUNK in $(seq 10); do echo $CHUNK; sleep 1; done) \ # | curl -T - http://localhost:8080/foo # # (Note that curl will send Expect: 100-Continue, too.) # # Or, heck, try letting curl complete successfully ;-). # Some potential improvements, if you wanted to try and extend this to a real # general-purpose HTTP server (and to give you some hints about the many # considerations that go into making a robust HTTP server): # # - The timeout handling is rather crude -- we impose a flat 10 second timeout # on each request (starting from the end of the previous # response). Something finer-grained would be better. Also, if a timeout is # triggered we unconditionally send a 500 Internal Server Error; it would be # better to keep track of whether the timeout is the client's fault, and if # so send a 408 Request Timeout. # # - The error handling policy here is somewhat crude as well. It handles a lot # of cases perfectly, but there are corner cases where the ideal behavior is # more debateable. For example, if a client starts uploading a large # request, uses 100-Continue, and we send an error response, then we'll shut # down the connection immediately (for well-behaved clients) or after # spending TIMEOUT seconds reading and discarding their upload (for # ill-behaved ones that go on and try to upload their request anyway). And # for clients that do this without 100-Continue, we'll send the error # response and then shut them down after TIMEOUT seconds. This might or # might not be your preferred policy, though -- maybe you want to shut such # clients down immediately (even if this risks their not seeing the # response), or maybe you're happy to let them continue sending all the data # and wasting your bandwidth if this is what it takes to guarantee that they # see your error response. Up to you, really. # # - Another example of a debateable choice: if a response handler errors out # without having done *anything* -- hasn't started responding, hasn't read # the request body -- then this connection actually is salvagable, if the # server sends an error response + reads and discards the request body. This # code sends the error response, but it doesn't try to salvage the # connection by reading the request body, it just closes the # connection. This is quite possibly the best option, but again this is a # policy decision. # # - Our error pages always include the exception text. In real life you might # want to log the exception but not send that information to the client. # # - Our error responses perhaps should include Connection: close when we know # we're going to close this connection. # # - We don't support the HEAD method, but ought to. # # - We should probably do something cleverer with buffering responses and # TCP_CORK and suchlike. import json from itertools import count from wsgiref.handlers import format_date_time import trio import h11 MAX_RECV = 2 ** 16 TIMEOUT = 10 ################################################################ # I/O adapter: h11 <-> trio ################################################################ # The core of this could be factored out to be usable for trio-based clients # too, as well as servers. But as a simplified pedagogical example we don't # attempt this here. class TrioHTTPWrapper: _next_id = count() def __init__(self, stream): self.stream = stream self.conn = h11.Connection(h11.SERVER) # Our Server: header self.ident = " ".join( ["h11-example-trio-server/{}".format(h11.__version__), h11.PRODUCT_ID] ).encode("ascii") # A unique id for this connection, to include in debugging output # (useful for understanding what's going on if there are multiple # simultaneous clients). self._obj_id = next(TrioHTTPWrapper._next_id) async def send(self, event): # The code below doesn't send ConnectionClosed, so we don't bother # handling it here either -- it would require that we do something # appropriate when 'data' is None. assert type(event) is not h11.ConnectionClosed data = self.conn.send(event) await self.stream.send_all(data) async def _read_from_peer(self): if self.conn.they_are_waiting_for_100_continue: self.info("Sending 100 Continue") go_ahead = h11.InformationalResponse( status_code=100, headers=self.basic_headers() ) await self.send(go_ahead) try: data = await self.stream.receive_some(MAX_RECV) except ConnectionError: # They've stopped listening. Not much we can do about it here. data = b"" self.conn.receive_data(data) async def next_event(self): while True: event = self.conn.next_event() if event is h11.NEED_DATA: await self._read_from_peer() continue return event async def shutdown_and_clean_up(self): # When this method is called, it's because we definitely want to kill # this connection, either as a clean shutdown or because of some kind # of error or loss-of-sync bug, and we no longer care if that violates # the protocol or not. So we ignore the state of self.conn, and just # go ahead and do the shutdown on the socket directly. (If you're # implementing a client you might prefer to send ConnectionClosed() # and let it raise an exception if that violates the protocol.) # try: await self.stream.send_eof() except trio.BrokenResourceError: # They're already gone, nothing to do return # Wait and read for a bit to give them a chance to see that we closed # things, but eventually give up and just close the socket. # XX FIXME: possibly we should set SO_LINGER to 0 here, so # that in the case where the client has ignored our shutdown and # declined to initiate the close themselves, we do a violent shutdown # (RST) and avoid the TIME_WAIT? # it looks like nginx never does this for keepalive timeouts, and only # does it for regular timeouts (slow clients I guess?) if explicitly # enabled ("Default: reset_timedout_connection off") with trio.move_on_after(TIMEOUT): try: while True: # Attempt to read until EOF got = await self.stream.receive_some(MAX_RECV) if not got: break except trio.BrokenResourceError: pass finally: await self.stream.aclose() def basic_headers(self): # HTTP requires these headers in all responses (client would do # something different here) return [ ("Date", format_date_time(None).encode("ascii")), ("Server", self.ident), ] def info(self, *args): # Little debugging method print("{}:".format(self._obj_id), *args) ################################################################ # Server main loop ################################################################ # General theory: # # If everything goes well: # - we'll get a Request # - our response handler will read the request body and send a full response # - that will either leave us in MUST_CLOSE (if the client doesn't # support keepalive) or DONE/DONE (if the client does). # # But then there are many, many different ways that things can go wrong # here. For example: # - we don't actually get a Request, but rather a ConnectionClosed # - exception is raised from somewhere (naughty client, broken # response handler, whatever) # - depending on what went wrong and where, we might or might not be # able to send an error response, and the connection might or # might not be salvagable after that # - response handler doesn't fully read the request or doesn't send a # full response # # But these all have one thing in common: they involve us leaving the # nice easy path up above. So we can just proceed on the assumption # that the nice easy thing is what's happening, and whenever something # goes wrong do our best to get back onto that path, and h11 will keep # track of how successful we were and raise new errors if things don't work # out. async def http_serve(stream): wrapper = TrioHTTPWrapper(stream) wrapper.info("Got new connection") while True: assert wrapper.conn.states == {h11.CLIENT: h11.IDLE, h11.SERVER: h11.IDLE} try: with trio.fail_after(TIMEOUT): wrapper.info("Server main loop waiting for request") event = await wrapper.next_event() wrapper.info("Server main loop got event:", event) if type(event) is h11.Request: await send_echo_response(wrapper, event) except Exception as exc: wrapper.info("Error during response handler: {!r}".format(exc)) await maybe_send_error_response(wrapper, exc) if wrapper.conn.our_state is h11.MUST_CLOSE: wrapper.info("connection is not reusable, so shutting down") await wrapper.shutdown_and_clean_up() return else: try: wrapper.info("trying to re-use connection") wrapper.conn.start_next_cycle() except h11.ProtocolError: states = wrapper.conn.states wrapper.info("unexpected state", states, "-- bailing out") await maybe_send_error_response( wrapper, RuntimeError("unexpected state {}".format(states)) ) await wrapper.shutdown_and_clean_up() return ################################################################ # Actual response handlers ################################################################ # Helper function async def send_simple_response(wrapper, status_code, content_type, body): wrapper.info("Sending", status_code, "response with", len(body), "bytes") headers = wrapper.basic_headers() headers.append(("Content-Type", content_type)) headers.append(("Content-Length", str(len(body)))) res = h11.Response(status_code=status_code, headers=headers) await wrapper.send(res) await wrapper.send(h11.Data(data=body)) await wrapper.send(h11.EndOfMessage()) async def maybe_send_error_response(wrapper, exc): # If we can't send an error, oh well, nothing to be done wrapper.info("trying to send error response...") if wrapper.conn.our_state not in {h11.IDLE, h11.SEND_RESPONSE}: wrapper.info("...but I can't, because our state is", wrapper.conn.our_state) return try: if isinstance(exc, h11.RemoteProtocolError): status_code = exc.error_status_hint elif isinstance(exc, trio.TooSlowError): status_code = 408 # Request Timeout else: status_code = 500 body = str(exc).encode("utf-8") await send_simple_response( wrapper, status_code, "text/plain; charset=utf-8", body ) except Exception as exc: wrapper.info("error while sending error response:", exc) async def send_echo_response(wrapper, request): wrapper.info("Preparing echo response") if request.method not in {b"GET", b"POST"}: # Laziness: we should send a proper 405 Method Not Allowed with the # appropriate Accept: header, but we don't. raise RuntimeError("unsupported method") response_json = { "method": request.method.decode("ascii"), "target": request.target.decode("ascii"), "headers": [ (name.decode("ascii"), value.decode("ascii")) for (name, value) in request.headers ], "body": "", } while True: event = await wrapper.next_event() if type(event) is h11.EndOfMessage: break assert type(event) is h11.Data response_json["body"] += event.data.decode("ascii") response_body_unicode = json.dumps( response_json, sort_keys=True, indent=4, separators=(",", ": ") ) response_body_bytes = response_body_unicode.encode("utf-8") await send_simple_response( wrapper, 200, "application/json; charset=utf-8", response_body_bytes ) async def serve(port): print("listening on http://localhost:{}".format(port)) try: await trio.serve_tcp(http_serve, port) except KeyboardInterrupt: print("KeyboardInterrupt - shutting down") ################################################################ # Run the server ################################################################ if __name__ == "__main__": trio.run(serve, 8080) h11-0.13.0/fuzz/000077500000000000000000000000001417207257000131675ustar00rootroot00000000000000h11-0.13.0/fuzz/README.rst000066400000000000000000000044071417207257000146630ustar00rootroot00000000000000Some harness code for using `afl `_ and `python-afl `_ to fuzz-test h11. See `Alex Gaynor's tutorial `_, or just: .. code-block:: sh sudo apt install afl pip install python-afl cd fuzz PYTHONPATH=.. py-afl-fuzz -o results -i afl-server-examples/ -- python ./afl-server.py Note 1: You may need to add ``AFL_SKIP_CPUFREQ=1`` if you want to play with it on a laptop and don't want to bother messing with your cpufreq config. Note 2: You may see some false "hangs" due to afl's aggressive default timeouts. I think this might be intentional, and serve to discourage afl from wasting time exploring arbitrarily longer and longer inputs? Or you can set the timeout explicitly with ``-t $MILLISECONDS``. Note 3: `Parallel fuzzing is a good thing `_. Right now we just have a simple test that throws garbage at the server ``receive_data`` and makes sure that it's either accepted or raises ``RemoteProtocolError``, never any other exceptions. (As an example of how even this relatively simple thing can catch bugs, here's a `bug in gunicorn `_ that was found by this approach, and `here's a bug this found in h11 `_... though that one's so simple that even basic fuzz-testing would have found it without any of afl's cleverness.) Ideas for further additions --------------------------- * Teach afl-server.py to watch the state machine and send responses back to get things unpaused, to allow for fuzzing of pipelined requests and unsuccessful protocol switches * Add a client-side fuzzer too * Add a `dictionary `_ tuned for HTTP * add more seed examples: ``Connection: close``? more complicated chunked examples? pipelining and protocol switch examples? * check that the all-at-once and byte-by-byte processing give the same event stream (modulo data splits) * maybe should split apart fancy checks versus non-fancy checks b/c speed is important h11-0.13.0/fuzz/afl-server-examples/000077500000000000000000000000001417207257000170515ustar00rootroot00000000000000h11-0.13.0/fuzz/afl-server-examples/1000066400000000000000000000000541417207257000171330ustar00rootroot00000000000000GET /some-path HTTP/1.1 Host: example.com h11-0.13.0/fuzz/afl-server-examples/2000066400000000000000000000001321417207257000171310ustar00rootroot00000000000000POST /some/path HTTP/1.1 HOST: example.com Transfer-Encoding: chunked 5 abcde 0 h11-0.13.0/fuzz/afl-server-examples/3000066400000000000000000000001071417207257000171340ustar00rootroot00000000000000PUT /asdf HTTP/1.1 Host: a.b.c.d.e Content-length: 10 abcdefghij h11-0.13.0/fuzz/afl-server-examples/4000066400000000000000000000000261417207257000171350ustar00rootroot00000000000000GET /asdf HTTP/1.0 h11-0.13.0/fuzz/afl-server.py000066400000000000000000000023251417207257000156110ustar00rootroot00000000000000# Invariant tested: No matter what random garbage a client throws at us, we # either successfully parse it, or else throw a RemoteProtocolError, never any # other error. import os import sys import afl import h11 def process_all(c): while True: event = c.next_event() if event is h11.NEED_DATA or event is h11.PAUSED: break if type(event) is h11.ConnectionClosed: break afl.init() data = sys.stdin.detach().read() # one big chunk server1 = h11.Connection(h11.SERVER) try: server1.receive_data(data) process_all(server1) server1.receive_data(b"") process_all(server1) except h11.RemoteProtocolError: pass # byte at a time server2 = h11.Connection(h11.SERVER) try: for i in range(len(data)): server2.receive_data(data[i : i + 1]) process_all(server2) server2.receive_data(b"") process_all(server2) except h11.RemoteProtocolError: pass # Suggested by the afl-python docs -- this substantially speeds up fuzzing, at # the risk of missing bugs that would cause the interpreter to crash on # exit. h11 is pure python, so I'm pretty sure h11 doesn't have any bugs that # would cause the interpreter to crash on exit. os._exit(0) h11-0.13.0/h11/000077500000000000000000000000001417207257000125625ustar00rootroot00000000000000h11-0.13.0/h11/__init__.py000066400000000000000000000027431417207257000147010ustar00rootroot00000000000000# A highish-level implementation of the HTTP/1.1 wire protocol (RFC 7230), # containing no networking code at all, loosely modelled on hyper-h2's generic # implementation of HTTP/2 (and in particular the h2.connection.H2Connection # class). There's still a bunch of subtle details you need to get right if you # want to make this actually useful, because it doesn't implement all the # semantics to check that what you're asking to write to the wire is sensible, # but at least it gets you out of dealing with the wire itself. from h11._connection import Connection, NEED_DATA, PAUSED from h11._events import ( ConnectionClosed, Data, EndOfMessage, Event, InformationalResponse, Request, Response, ) from h11._state import ( CLIENT, CLOSED, DONE, ERROR, IDLE, MIGHT_SWITCH_PROTOCOL, MUST_CLOSE, SEND_BODY, SEND_RESPONSE, SERVER, SWITCHED_PROTOCOL, ) from h11._util import LocalProtocolError, ProtocolError, RemoteProtocolError from h11._version import __version__ PRODUCT_ID = "python-h11/" + __version__ __all__ = ( "Connection", "NEED_DATA", "PAUSED", "ConnectionClosed", "Data", "EndOfMessage", "Event", "InformationalResponse", "Request", "Response", "CLIENT", "CLOSED", "DONE", "ERROR", "IDLE", "MUST_CLOSE", "SEND_BODY", "SEND_RESPONSE", "SERVER", "SWITCHED_PROTOCOL", "ProtocolError", "LocalProtocolError", "RemoteProtocolError", ) h11-0.13.0/h11/_abnf.py000066400000000000000000000110351417207257000142010ustar00rootroot00000000000000# We use native strings for all the re patterns, to take advantage of string # formatting, and then convert to bytestrings when compiling the final re # objects. # https://svn.tools.ietf.org/svn/wg/httpbis/specs/rfc7230.html#whitespace # OWS = *( SP / HTAB ) # ; optional whitespace OWS = r"[ \t]*" # https://svn.tools.ietf.org/svn/wg/httpbis/specs/rfc7230.html#rule.token.separators # token = 1*tchar # # tchar = "!" / "#" / "$" / "%" / "&" / "'" / "*" # / "+" / "-" / "." / "^" / "_" / "`" / "|" / "~" # / DIGIT / ALPHA # ; any VCHAR, except delimiters token = r"[-!#$%&'*+.^_`|~0-9a-zA-Z]+" # https://svn.tools.ietf.org/svn/wg/httpbis/specs/rfc7230.html#header.fields # field-name = token field_name = token # The standard says: # # field-value = *( field-content / obs-fold ) # field-content = field-vchar [ 1*( SP / HTAB ) field-vchar ] # field-vchar = VCHAR / obs-text # obs-fold = CRLF 1*( SP / HTAB ) # ; obsolete line folding # ; see Section 3.2.4 # # https://tools.ietf.org/html/rfc5234#appendix-B.1 # # VCHAR = %x21-7E # ; visible (printing) characters # # https://svn.tools.ietf.org/svn/wg/httpbis/specs/rfc7230.html#rule.quoted-string # obs-text = %x80-FF # # However, the standard definition of field-content is WRONG! It disallows # fields containing a single visible character surrounded by whitespace, # e.g. "foo a bar". # # See: https://www.rfc-editor.org/errata_search.php?rfc=7230&eid=4189 # # So our definition of field_content attempts to fix it up... # # Also, we allow lots of control characters, because apparently people assume # that they're legal in practice (e.g., google analytics makes cookies with # \x01 in them!): # https://github.com/python-hyper/h11/issues/57 # We still don't allow NUL or whitespace, because those are often treated as # meta-characters and letting them through can lead to nasty issues like SSRF. vchar = r"[\x21-\x7e]" vchar_or_obs_text = r"[^\x00\s]" field_vchar = vchar_or_obs_text field_content = r"{field_vchar}+(?:[ \t]+{field_vchar}+)*".format(**globals()) # We handle obs-fold at a different level, and our fixed-up field_content # already grows to swallow the whole value, so ? instead of * field_value = r"({field_content})?".format(**globals()) # header-field = field-name ":" OWS field-value OWS header_field = ( r"(?P{field_name})" r":" r"{OWS}" r"(?P{field_value})" r"{OWS}".format(**globals()) ) # https://svn.tools.ietf.org/svn/wg/httpbis/specs/rfc7230.html#request.line # # request-line = method SP request-target SP HTTP-version CRLF # method = token # HTTP-version = HTTP-name "/" DIGIT "." DIGIT # HTTP-name = %x48.54.54.50 ; "HTTP", case-sensitive # # request-target is complicated (see RFC 7230 sec 5.3) -- could be path, full # URL, host+port (for connect), or even "*", but in any case we are guaranteed # that it contists of the visible printing characters. method = token request_target = r"{vchar}+".format(**globals()) http_version = r"HTTP/(?P[0-9]\.[0-9])" request_line = ( r"(?P{method})" r" " r"(?P{request_target})" r" " r"{http_version}".format(**globals()) ) # https://svn.tools.ietf.org/svn/wg/httpbis/specs/rfc7230.html#status.line # # status-line = HTTP-version SP status-code SP reason-phrase CRLF # status-code = 3DIGIT # reason-phrase = *( HTAB / SP / VCHAR / obs-text ) status_code = r"[0-9]{3}" reason_phrase = r"([ \t]|{vchar_or_obs_text})*".format(**globals()) status_line = ( r"{http_version}" r" " r"(?P{status_code})" # However, there are apparently a few too many servers out there that just # leave out the reason phrase: # https://github.com/scrapy/scrapy/issues/345#issuecomment-281756036 # https://github.com/seanmonstar/httparse/issues/29 # so make it optional. ?: is a non-capturing group. r"(?: (?P{reason_phrase}))?".format(**globals()) ) HEXDIG = r"[0-9A-Fa-f]" # Actually # # chunk-size = 1*HEXDIG # # but we impose an upper-limit to avoid ridiculosity. len(str(2**64)) == 20 chunk_size = r"({HEXDIG}){{1,20}}".format(**globals()) # Actually # # chunk-ext = *( ";" chunk-ext-name [ "=" chunk-ext-val ] ) # # but we aren't parsing the things so we don't really care. chunk_ext = r";.*" chunk_header = ( r"(?P{chunk_size})" r"(?P{chunk_ext})?" r"\r\n".format(**globals()) ) h11-0.13.0/h11/_connection.py000066400000000000000000000635601417207257000154440ustar00rootroot00000000000000# This contains the main Connection class. Everything in h11 revolves around # this. from typing import Any, Callable, cast, Dict, List, Optional, Tuple, Type, Union from ._events import ( ConnectionClosed, Data, EndOfMessage, Event, InformationalResponse, Request, Response, ) from ._headers import get_comma_header, has_expect_100_continue, set_comma_header from ._readers import READERS, ReadersType from ._receivebuffer import ReceiveBuffer from ._state import ( _SWITCH_CONNECT, _SWITCH_UPGRADE, CLIENT, ConnectionState, DONE, ERROR, MIGHT_SWITCH_PROTOCOL, SEND_BODY, SERVER, SWITCHED_PROTOCOL, ) from ._util import ( # Import the internal things we need LocalProtocolError, RemoteProtocolError, Sentinel, ) from ._writers import WRITERS, WritersType # Everything in __all__ gets re-exported as part of the h11 public API. __all__ = ["Connection", "NEED_DATA", "PAUSED"] class NEED_DATA(Sentinel, metaclass=Sentinel): pass class PAUSED(Sentinel, metaclass=Sentinel): pass # If we ever have this much buffered without it making a complete parseable # event, we error out. The only time we really buffer is when reading the # request/response line + headers together, so this is effectively the limit on # the size of that. # # Some precedents for defaults: # - node.js: 80 * 1024 # - tomcat: 8 * 1024 # - IIS: 16 * 1024 # - Apache: <8 KiB per line> DEFAULT_MAX_INCOMPLETE_EVENT_SIZE = 16 * 1024 # RFC 7230's rules for connection lifecycles: # - If either side says they want to close the connection, then the connection # must close. # - HTTP/1.1 defaults to keep-alive unless someone says Connection: close # - HTTP/1.0 defaults to close unless both sides say Connection: keep-alive # (and even this is a mess -- e.g. if you're implementing a proxy then # sending Connection: keep-alive is forbidden). # # We simplify life by simply not supporting keep-alive with HTTP/1.0 peers. So # our rule is: # - If someone says Connection: close, we will close # - If someone uses HTTP/1.0, we will close. def _keep_alive(event: Union[Request, Response]) -> bool: connection = get_comma_header(event.headers, b"connection") if b"close" in connection: return False if getattr(event, "http_version", b"1.1") < b"1.1": return False return True def _body_framing( request_method: bytes, event: Union[Request, Response] ) -> Tuple[str, Union[Tuple[()], Tuple[int]]]: # Called when we enter SEND_BODY to figure out framing information for # this body. # # These are the only two events that can trigger a SEND_BODY state: assert type(event) in (Request, Response) # Returns one of: # # ("content-length", count) # ("chunked", ()) # ("http/1.0", ()) # # which are (lookup key, *args) for constructing body reader/writer # objects. # # Reference: https://tools.ietf.org/html/rfc7230#section-3.3.3 # # Step 1: some responses always have an empty body, regardless of what the # headers say. if type(event) is Response: if ( event.status_code in (204, 304) or request_method == b"HEAD" or (request_method == b"CONNECT" and 200 <= event.status_code < 300) ): return ("content-length", (0,)) # Section 3.3.3 also lists another case -- responses with status_code # < 200. For us these are InformationalResponses, not Responses, so # they can't get into this function in the first place. assert event.status_code >= 200 # Step 2: check for Transfer-Encoding (T-E beats C-L): transfer_encodings = get_comma_header(event.headers, b"transfer-encoding") if transfer_encodings: assert transfer_encodings == [b"chunked"] return ("chunked", ()) # Step 3: check for Content-Length content_lengths = get_comma_header(event.headers, b"content-length") if content_lengths: return ("content-length", (int(content_lengths[0]),)) # Step 4: no applicable headers; fallback/default depends on type if type(event) is Request: return ("content-length", (0,)) else: return ("http/1.0", ()) ################################################################ # # The main Connection class # ################################################################ class Connection: """An object encapsulating the state of an HTTP connection. Args: our_role: If you're implementing a client, pass :data:`h11.CLIENT`. If you're implementing a server, pass :data:`h11.SERVER`. max_incomplete_event_size (int): The maximum number of bytes we're willing to buffer of an incomplete event. In practice this mostly sets a limit on the maximum size of the request/response line + headers. If this is exceeded, then :meth:`next_event` will raise :exc:`RemoteProtocolError`. """ def __init__( self, our_role: Type[Sentinel], max_incomplete_event_size: int = DEFAULT_MAX_INCOMPLETE_EVENT_SIZE, ) -> None: self._max_incomplete_event_size = max_incomplete_event_size # State and role tracking if our_role not in (CLIENT, SERVER): raise ValueError("expected CLIENT or SERVER, not {!r}".format(our_role)) self.our_role = our_role self.their_role: Type[Sentinel] if our_role is CLIENT: self.their_role = SERVER else: self.their_role = CLIENT self._cstate = ConnectionState() # Callables for converting data->events or vice-versa given the # current state self._writer = self._get_io_object(self.our_role, None, WRITERS) self._reader = self._get_io_object(self.their_role, None, READERS) # Holds any unprocessed received data self._receive_buffer = ReceiveBuffer() # If this is true, then it indicates that the incoming connection was # closed *after* the end of whatever's in self._receive_buffer: self._receive_buffer_closed = False # Extra bits of state that don't fit into the state machine. # # These two are only used to interpret framing headers for figuring # out how to read/write response bodies. their_http_version is also # made available as a convenient public API. self.their_http_version: Optional[bytes] = None self._request_method: Optional[bytes] = None # This is pure flow-control and doesn't at all affect the set of legal # transitions, so no need to bother ConnectionState with it: self.client_is_waiting_for_100_continue = False @property def states(self) -> Dict[Type[Sentinel], Type[Sentinel]]: """A dictionary like:: {CLIENT: , SERVER: } See :ref:`state-machine` for details. """ return dict(self._cstate.states) @property def our_state(self) -> Type[Sentinel]: """The current state of whichever role we are playing. See :ref:`state-machine` for details. """ return self._cstate.states[self.our_role] @property def their_state(self) -> Type[Sentinel]: """The current state of whichever role we are NOT playing. See :ref:`state-machine` for details. """ return self._cstate.states[self.their_role] @property def they_are_waiting_for_100_continue(self) -> bool: return self.their_role is CLIENT and self.client_is_waiting_for_100_continue def start_next_cycle(self) -> None: """Attempt to reset our connection state for a new request/response cycle. If both client and server are in :data:`DONE` state, then resets them both to :data:`IDLE` state in preparation for a new request/response cycle on this same connection. Otherwise, raises a :exc:`LocalProtocolError`. See :ref:`keepalive-and-pipelining`. """ old_states = dict(self._cstate.states) self._cstate.start_next_cycle() self._request_method = None # self.their_http_version gets left alone, since it presumably lasts # beyond a single request/response cycle assert not self.client_is_waiting_for_100_continue self._respond_to_state_changes(old_states) def _process_error(self, role: Type[Sentinel]) -> None: old_states = dict(self._cstate.states) self._cstate.process_error(role) self._respond_to_state_changes(old_states) def _server_switch_event(self, event: Event) -> Optional[Type[Sentinel]]: if type(event) is InformationalResponse and event.status_code == 101: return _SWITCH_UPGRADE if type(event) is Response: if ( _SWITCH_CONNECT in self._cstate.pending_switch_proposals and 200 <= event.status_code < 300 ): return _SWITCH_CONNECT return None # All events go through here def _process_event(self, role: Type[Sentinel], event: Event) -> None: # First, pass the event through the state machine to make sure it # succeeds. old_states = dict(self._cstate.states) if role is CLIENT and type(event) is Request: if event.method == b"CONNECT": self._cstate.process_client_switch_proposal(_SWITCH_CONNECT) if get_comma_header(event.headers, b"upgrade"): self._cstate.process_client_switch_proposal(_SWITCH_UPGRADE) server_switch_event = None if role is SERVER: server_switch_event = self._server_switch_event(event) self._cstate.process_event(role, type(event), server_switch_event) # Then perform the updates triggered by it. if type(event) is Request: self._request_method = event.method if role is self.their_role and type(event) in ( Request, Response, InformationalResponse, ): event = cast(Union[Request, Response, InformationalResponse], event) self.their_http_version = event.http_version # Keep alive handling # # RFC 7230 doesn't really say what one should do if Connection: close # shows up on a 1xx InformationalResponse. I think the idea is that # this is not supposed to happen. In any case, if it does happen, we # ignore it. if type(event) in (Request, Response) and not _keep_alive( cast(Union[Request, Response], event) ): self._cstate.process_keep_alive_disabled() # 100-continue if type(event) is Request and has_expect_100_continue(event): self.client_is_waiting_for_100_continue = True if type(event) in (InformationalResponse, Response): self.client_is_waiting_for_100_continue = False if role is CLIENT and type(event) in (Data, EndOfMessage): self.client_is_waiting_for_100_continue = False self._respond_to_state_changes(old_states, event) def _get_io_object( self, role: Type[Sentinel], event: Optional[Event], io_dict: Union[ReadersType, WritersType], ) -> Optional[Callable[..., Any]]: # event may be None; it's only used when entering SEND_BODY state = self._cstate.states[role] if state is SEND_BODY: # Special case: the io_dict has a dict of reader/writer factories # that depend on the request/response framing. framing_type, args = _body_framing( cast(bytes, self._request_method), cast(Union[Request, Response], event) ) return io_dict[SEND_BODY][framing_type](*args) # type: ignore[index] else: # General case: the io_dict just has the appropriate reader/writer # for this state return io_dict.get((role, state)) # type: ignore # This must be called after any action that might have caused # self._cstate.states to change. def _respond_to_state_changes( self, old_states: Dict[Type[Sentinel], Type[Sentinel]], event: Optional[Event] = None, ) -> None: # Update reader/writer if self.our_state != old_states[self.our_role]: self._writer = self._get_io_object(self.our_role, event, WRITERS) if self.their_state != old_states[self.their_role]: self._reader = self._get_io_object(self.their_role, event, READERS) @property def trailing_data(self) -> Tuple[bytes, bool]: """Data that has been received, but not yet processed, represented as a tuple with two elements, where the first is a byte-string containing the unprocessed data itself, and the second is a bool that is True if the receive connection was closed. See :ref:`switching-protocols` for discussion of why you'd want this. """ return (bytes(self._receive_buffer), self._receive_buffer_closed) def receive_data(self, data: bytes) -> None: """Add data to our internal receive buffer. This does not actually do any processing on the data, just stores it. To trigger processing, you have to call :meth:`next_event`. Args: data (:term:`bytes-like object`): The new data that was just received. Special case: If *data* is an empty byte-string like ``b""``, then this indicates that the remote side has closed the connection (end of file). Normally this is convenient, because standard Python APIs like :meth:`file.read` or :meth:`socket.recv` use ``b""`` to indicate end-of-file, while other failures to read are indicated using other mechanisms like raising :exc:`TimeoutError`. When using such an API you can just blindly pass through whatever you get from ``read`` to :meth:`receive_data`, and everything will work. But, if you have an API where reading an empty string is a valid non-EOF condition, then you need to be aware of this and make sure to check for such strings and avoid passing them to :meth:`receive_data`. Returns: Nothing, but after calling this you should call :meth:`next_event` to parse the newly received data. Raises: RuntimeError: Raised if you pass an empty *data*, indicating EOF, and then pass a non-empty *data*, indicating more data that somehow arrived after the EOF. (Calling ``receive_data(b"")`` multiple times is fine, and equivalent to calling it once.) """ if data: if self._receive_buffer_closed: raise RuntimeError("received close, then received more data?") self._receive_buffer += data else: self._receive_buffer_closed = True def _extract_next_receive_event(self) -> Union[Event, Type[Sentinel]]: state = self.their_state # We don't pause immediately when they enter DONE, because even in # DONE state we can still process a ConnectionClosed() event. But # if we have data in our buffer, then we definitely aren't getting # a ConnectionClosed() immediately and we need to pause. if state is DONE and self._receive_buffer: return PAUSED if state is MIGHT_SWITCH_PROTOCOL or state is SWITCHED_PROTOCOL: return PAUSED assert self._reader is not None event = self._reader(self._receive_buffer) if event is None: if not self._receive_buffer and self._receive_buffer_closed: # In some unusual cases (basically just HTTP/1.0 bodies), EOF # triggers an actual protocol event; in that case, we want to # return that event, and then the state will change and we'll # get called again to generate the actual ConnectionClosed(). if hasattr(self._reader, "read_eof"): event = self._reader.read_eof() # type: ignore[attr-defined] else: event = ConnectionClosed() if event is None: event = NEED_DATA return event # type: ignore[no-any-return] def next_event(self) -> Union[Event, Type[Sentinel]]: """Parse the next event out of our receive buffer, update our internal state, and return it. This is a mutating operation -- think of it like calling :func:`next` on an iterator. Returns: : One of three things: 1) An event object -- see :ref:`events`. 2) The special constant :data:`NEED_DATA`, which indicates that you need to read more data from your socket and pass it to :meth:`receive_data` before this method will be able to return any more events. 3) The special constant :data:`PAUSED`, which indicates that we are not in a state where we can process incoming data (usually because the peer has finished their part of the current request/response cycle, and you have not yet called :meth:`start_next_cycle`). See :ref:`flow-control` for details. Raises: RemoteProtocolError: The peer has misbehaved. You should close the connection (possibly after sending some kind of 4xx response). Once this method returns :class:`ConnectionClosed` once, then all subsequent calls will also return :class:`ConnectionClosed`. If this method raises any exception besides :exc:`RemoteProtocolError` then that's a bug -- if it happens please file a bug report! If this method raises any exception then it also sets :attr:`Connection.their_state` to :data:`ERROR` -- see :ref:`error-handling` for discussion. """ if self.their_state is ERROR: raise RemoteProtocolError("Can't receive data when peer state is ERROR") try: event = self._extract_next_receive_event() if event not in [NEED_DATA, PAUSED]: self._process_event(self.their_role, cast(Event, event)) if event is NEED_DATA: if len(self._receive_buffer) > self._max_incomplete_event_size: # 431 is "Request header fields too large" which is pretty # much the only situation where we can get here raise RemoteProtocolError( "Receive buffer too long", error_status_hint=431 ) if self._receive_buffer_closed: # We're still trying to complete some event, but that's # never going to happen because no more data is coming raise RemoteProtocolError("peer unexpectedly closed connection") return event except BaseException as exc: self._process_error(self.their_role) if isinstance(exc, LocalProtocolError): exc._reraise_as_remote_protocol_error() else: raise def send(self, event: Event) -> Optional[bytes]: """Convert a high-level event into bytes that can be sent to the peer, while updating our internal state machine. Args: event: The :ref:`event ` to send. Returns: If ``type(event) is ConnectionClosed``, then returns ``None``. Otherwise, returns a :term:`bytes-like object`. Raises: LocalProtocolError: Sending this event at this time would violate our understanding of the HTTP/1.1 protocol. If this method raises any exception then it also sets :attr:`Connection.our_state` to :data:`ERROR` -- see :ref:`error-handling` for discussion. """ data_list = self.send_with_data_passthrough(event) if data_list is None: return None else: return b"".join(data_list) def send_with_data_passthrough(self, event: Event) -> Optional[List[bytes]]: """Identical to :meth:`send`, except that in situations where :meth:`send` returns a single :term:`bytes-like object`, this instead returns a list of them -- and when sending a :class:`Data` event, this list is guaranteed to contain the exact object you passed in as :attr:`Data.data`. See :ref:`sendfile` for discussion. """ if self.our_state is ERROR: raise LocalProtocolError("Can't send data when our state is ERROR") try: if type(event) is Response: event = self._clean_up_response_headers_for_sending(event) # We want to call _process_event before calling the writer, # because if someone tries to do something invalid then this will # give a sensible error message, while our writers all just assume # they will only receive valid events. But, _process_event might # change self._writer. So we have to do a little dance: writer = self._writer self._process_event(self.our_role, event) if type(event) is ConnectionClosed: return None else: # In any situation where writer is None, process_event should # have raised ProtocolError assert writer is not None data_list: List[bytes] = [] writer(event, data_list.append) return data_list except: self._process_error(self.our_role) raise def send_failed(self) -> None: """Notify the state machine that we failed to send the data it gave us. This causes :attr:`Connection.our_state` to immediately become :data:`ERROR` -- see :ref:`error-handling` for discussion. """ self._process_error(self.our_role) # When sending a Response, we take responsibility for a few things: # # - Sometimes you MUST set Connection: close. We take care of those # times. (You can also set it yourself if you want, and if you do then # we'll respect that and close the connection at the right time. But you # don't have to worry about that unless you want to.) # # - The user has to set Content-Length if they want it. Otherwise, for # responses that have bodies (e.g. not HEAD), then we will automatically # select the right mechanism for streaming a body of unknown length, # which depends on depending on the peer's HTTP version. # # This function's *only* responsibility is making sure headers are set up # right -- everything downstream just looks at the headers. There are no # side channels. def _clean_up_response_headers_for_sending(self, response: Response) -> Response: assert type(response) is Response headers = response.headers need_close = False # HEAD requests need some special handling: they always act like they # have Content-Length: 0, and that's how _body_framing treats # them. But their headers are supposed to match what we would send if # the request was a GET. (Technically there is one deviation allowed: # we're allowed to leave out the framing headers -- see # https://tools.ietf.org/html/rfc7231#section-4.3.2 . But it's just as # easy to get them right.) method_for_choosing_headers = cast(bytes, self._request_method) if method_for_choosing_headers == b"HEAD": method_for_choosing_headers = b"GET" framing_type, _ = _body_framing(method_for_choosing_headers, response) if framing_type in ("chunked", "http/1.0"): # This response has a body of unknown length. # If our peer is HTTP/1.1, we use Transfer-Encoding: chunked # If our peer is HTTP/1.0, we use no framing headers, and close the # connection afterwards. # # Make sure to clear Content-Length (in principle user could have # set both and then we ignored Content-Length b/c # Transfer-Encoding overwrote it -- this would be naughty of them, # but the HTTP spec says that if our peer does this then we have # to fix it instead of erroring out, so we'll accord the user the # same respect). headers = set_comma_header(headers, b"content-length", []) if self.their_http_version is None or self.their_http_version < b"1.1": # Either we never got a valid request and are sending back an # error (their_http_version is None), so we assume the worst; # or else we did get a valid HTTP/1.0 request, so we know that # they don't understand chunked encoding. headers = set_comma_header(headers, b"transfer-encoding", []) # This is actually redundant ATM, since currently we # unconditionally disable keep-alive when talking to HTTP/1.0 # peers. But let's be defensive just in case we add # Connection: keep-alive support later: if self._request_method != b"HEAD": need_close = True else: headers = set_comma_header(headers, b"transfer-encoding", [b"chunked"]) if not self._cstate.keep_alive or need_close: # Make sure Connection: close is set connection = set(get_comma_header(headers, b"connection")) connection.discard(b"keep-alive") connection.add(b"close") headers = set_comma_header(headers, b"connection", sorted(connection)) return Response( headers=headers, status_code=response.status_code, http_version=response.http_version, reason=response.reason, ) h11-0.13.0/h11/_events.py000066400000000000000000000270501417207257000146030ustar00rootroot00000000000000# High level events that make up HTTP/1.1 conversations. Loosely inspired by # the corresponding events in hyper-h2: # # http://python-hyper.org/h2/en/stable/api.html#events # # Don't subclass these. Stuff will break. import re from abc import ABC from dataclasses import dataclass, field from typing import Any, cast, Dict, List, Tuple, Union from ._abnf import method, request_target from ._headers import Headers, normalize_and_validate from ._util import bytesify, LocalProtocolError, validate # Everything in __all__ gets re-exported as part of the h11 public API. __all__ = [ "Event", "Request", "InformationalResponse", "Response", "Data", "EndOfMessage", "ConnectionClosed", ] method_re = re.compile(method.encode("ascii")) request_target_re = re.compile(request_target.encode("ascii")) class Event(ABC): """ Base class for h11 events. """ __slots__ = () @dataclass(init=False, frozen=True) class Request(Event): """The beginning of an HTTP request. Fields: .. attribute:: method An HTTP method, e.g. ``b"GET"`` or ``b"POST"``. Always a byte string. :term:`Bytes-like objects ` and native strings containing only ascii characters will be automatically converted to byte strings. .. attribute:: target The target of an HTTP request, e.g. ``b"/index.html"``, or one of the more exotic formats described in `RFC 7320, section 5.3 `_. Always a byte string. :term:`Bytes-like objects ` and native strings containing only ascii characters will be automatically converted to byte strings. .. attribute:: headers Request headers, represented as a list of (name, value) pairs. See :ref:`the header normalization rules ` for details. .. attribute:: http_version The HTTP protocol version, represented as a byte string like ``b"1.1"``. See :ref:`the HTTP version normalization rules ` for details. """ __slots__ = ("method", "headers", "target", "http_version") method: bytes headers: Headers target: bytes http_version: bytes def __init__( self, *, method: Union[bytes, str], headers: Union[Headers, List[Tuple[bytes, bytes]], List[Tuple[str, str]]], target: Union[bytes, str], http_version: Union[bytes, str] = b"1.1", _parsed: bool = False, ) -> None: super().__init__() if isinstance(headers, Headers): object.__setattr__(self, "headers", headers) else: object.__setattr__( self, "headers", normalize_and_validate(headers, _parsed=_parsed) ) if not _parsed: object.__setattr__(self, "method", bytesify(method)) object.__setattr__(self, "target", bytesify(target)) object.__setattr__(self, "http_version", bytesify(http_version)) else: object.__setattr__(self, "method", method) object.__setattr__(self, "target", target) object.__setattr__(self, "http_version", http_version) # "A server MUST respond with a 400 (Bad Request) status code to any # HTTP/1.1 request message that lacks a Host header field and to any # request message that contains more than one Host header field or a # Host header field with an invalid field-value." # -- https://tools.ietf.org/html/rfc7230#section-5.4 host_count = 0 for name, value in self.headers: if name == b"host": host_count += 1 if self.http_version == b"1.1" and host_count == 0: raise LocalProtocolError("Missing mandatory Host: header") if host_count > 1: raise LocalProtocolError("Found multiple Host: headers") validate(method_re, self.method, "Illegal method characters") validate(request_target_re, self.target, "Illegal target characters") # This is an unhashable type. __hash__ = None # type: ignore @dataclass(init=False, frozen=True) class _ResponseBase(Event): __slots__ = ("headers", "http_version", "reason", "status_code") headers: Headers http_version: bytes reason: bytes status_code: int def __init__( self, *, headers: Union[Headers, List[Tuple[bytes, bytes]], List[Tuple[str, str]]], status_code: int, http_version: Union[bytes, str] = b"1.1", reason: Union[bytes, str] = b"", _parsed: bool = False, ) -> None: super().__init__() if isinstance(headers, Headers): object.__setattr__(self, "headers", headers) else: object.__setattr__( self, "headers", normalize_and_validate(headers, _parsed=_parsed) ) if not _parsed: object.__setattr__(self, "reason", bytesify(reason)) object.__setattr__(self, "http_version", bytesify(http_version)) if not isinstance(status_code, int): raise LocalProtocolError("status code must be integer") # Because IntEnum objects are instances of int, but aren't # duck-compatible (sigh), see gh-72. object.__setattr__(self, "status_code", int(status_code)) else: object.__setattr__(self, "reason", reason) object.__setattr__(self, "http_version", http_version) object.__setattr__(self, "status_code", status_code) self.__post_init__() def __post_init__(self) -> None: pass # This is an unhashable type. __hash__ = None # type: ignore @dataclass(init=False, frozen=True) class InformationalResponse(_ResponseBase): """An HTTP informational response. Fields: .. attribute:: status_code The status code of this response, as an integer. For an :class:`InformationalResponse`, this is always in the range [100, 200). .. attribute:: headers Request headers, represented as a list of (name, value) pairs. See :ref:`the header normalization rules ` for details. .. attribute:: http_version The HTTP protocol version, represented as a byte string like ``b"1.1"``. See :ref:`the HTTP version normalization rules ` for details. .. attribute:: reason The reason phrase of this response, as a byte string. For example: ``b"OK"``, or ``b"Not Found"``. """ def __post_init__(self) -> None: if not (100 <= self.status_code < 200): raise LocalProtocolError( "InformationalResponse status_code should be in range " "[100, 200), not {}".format(self.status_code) ) # This is an unhashable type. __hash__ = None # type: ignore @dataclass(init=False, frozen=True) class Response(_ResponseBase): """The beginning of an HTTP response. Fields: .. attribute:: status_code The status code of this response, as an integer. For an :class:`Response`, this is always in the range [200, 1000). .. attribute:: headers Request headers, represented as a list of (name, value) pairs. See :ref:`the header normalization rules ` for details. .. attribute:: http_version The HTTP protocol version, represented as a byte string like ``b"1.1"``. See :ref:`the HTTP version normalization rules ` for details. .. attribute:: reason The reason phrase of this response, as a byte string. For example: ``b"OK"``, or ``b"Not Found"``. """ def __post_init__(self) -> None: if not (200 <= self.status_code < 1000): raise LocalProtocolError( "Response status_code should be in range [200, 1000), not {}".format( self.status_code ) ) # This is an unhashable type. __hash__ = None # type: ignore @dataclass(init=False, frozen=True) class Data(Event): """Part of an HTTP message body. Fields: .. attribute:: data A :term:`bytes-like object` containing part of a message body. Or, if using the ``combine=False`` argument to :meth:`Connection.send`, then any object that your socket writing code knows what to do with, and for which calling :func:`len` returns the number of bytes that will be written -- see :ref:`sendfile` for details. .. attribute:: chunk_start A marker that indicates whether this data object is from the start of a chunked transfer encoding chunk. This field is ignored when when a Data event is provided to :meth:`Connection.send`: it is only valid on events emitted from :meth:`Connection.next_event`. You probably shouldn't use this attribute at all; see :ref:`chunk-delimiters-are-bad` for details. .. attribute:: chunk_end A marker that indicates whether this data object is the last for a given chunked transfer encoding chunk. This field is ignored when when a Data event is provided to :meth:`Connection.send`: it is only valid on events emitted from :meth:`Connection.next_event`. You probably shouldn't use this attribute at all; see :ref:`chunk-delimiters-are-bad` for details. """ __slots__ = ("data", "chunk_start", "chunk_end") data: bytes chunk_start: bool chunk_end: bool def __init__( self, data: bytes, chunk_start: bool = False, chunk_end: bool = False ) -> None: object.__setattr__(self, "data", data) object.__setattr__(self, "chunk_start", chunk_start) object.__setattr__(self, "chunk_end", chunk_end) # This is an unhashable type. __hash__ = None # type: ignore # XX FIXME: "A recipient MUST ignore (or consider as an error) any fields that # are forbidden to be sent in a trailer, since processing them as if they were # present in the header section might bypass external security filters." # https://svn.tools.ietf.org/svn/wg/httpbis/specs/rfc7230.html#chunked.trailer.part # Unfortunately, the list of forbidden fields is long and vague :-/ @dataclass(init=False, frozen=True) class EndOfMessage(Event): """The end of an HTTP message. Fields: .. attribute:: headers Default value: ``[]`` Any trailing headers attached to this message, represented as a list of (name, value) pairs. See :ref:`the header normalization rules ` for details. Must be empty unless ``Transfer-Encoding: chunked`` is in use. """ __slots__ = ("headers",) headers: Headers def __init__( self, *, headers: Union[ Headers, List[Tuple[bytes, bytes]], List[Tuple[str, str]], None ] = None, _parsed: bool = False, ) -> None: super().__init__() if headers is None: headers = Headers([]) elif not isinstance(headers, Headers): headers = normalize_and_validate(headers, _parsed=_parsed) object.__setattr__(self, "headers", headers) # This is an unhashable type. __hash__ = None # type: ignore @dataclass(frozen=True) class ConnectionClosed(Event): """This event indicates that the sender has closed their outgoing connection. Note that this does not necessarily mean that they can't *receive* further data, because TCP connections are composed to two one-way channels which can be closed independently. See :ref:`closing` for details. No fields. """ pass h11-0.13.0/h11/_headers.py000066400000000000000000000237661417207257000147240ustar00rootroot00000000000000import re from typing import AnyStr, cast, List, overload, Sequence, Tuple, TYPE_CHECKING, Union from ._abnf import field_name, field_value from ._util import bytesify, LocalProtocolError, validate if TYPE_CHECKING: from ._events import Request try: from typing import Literal except ImportError: from typing_extensions import Literal # type: ignore # Facts # ----- # # Headers are: # keys: case-insensitive ascii # values: mixture of ascii and raw bytes # # "Historically, HTTP has allowed field content with text in the ISO-8859-1 # charset [ISO-8859-1], supporting other charsets only through use of # [RFC2047] encoding. In practice, most HTTP header field values use only a # subset of the US-ASCII charset [USASCII]. Newly defined header fields SHOULD # limit their field values to US-ASCII octets. A recipient SHOULD treat other # octets in field content (obs-text) as opaque data." # And it deprecates all non-ascii values # # Leading/trailing whitespace in header names is forbidden # # Values get leading/trailing whitespace stripped # # Content-Disposition actually needs to contain unicode semantically; to # accomplish this it has a terrifically weird way of encoding the filename # itself as ascii (and even this still has lots of cross-browser # incompatibilities) # # Order is important: # "a proxy MUST NOT change the order of these field values when forwarding a # message" # (and there are several headers where the order indicates a preference) # # Multiple occurences of the same header: # "A sender MUST NOT generate multiple header fields with the same field name # in a message unless either the entire field value for that header field is # defined as a comma-separated list [or the header is Set-Cookie which gets a # special exception]" - RFC 7230. (cookies are in RFC 6265) # # So every header aside from Set-Cookie can be merged by b", ".join if it # occurs repeatedly. But, of course, they can't necessarily be split by # .split(b","), because quoting. # # Given all this mess (case insensitive, duplicates allowed, order is # important, ...), there doesn't appear to be any standard way to handle # headers in Python -- they're almost like dicts, but... actually just # aren't. For now we punt and just use a super simple representation: headers # are a list of pairs # # [(name1, value1), (name2, value2), ...] # # where all entries are bytestrings, names are lowercase and have no # leading/trailing whitespace, and values are bytestrings with no # leading/trailing whitespace. Searching and updating are done via naive O(n) # methods. # # Maybe a dict-of-lists would be better? _content_length_re = re.compile(br"[0-9]+") _field_name_re = re.compile(field_name.encode("ascii")) _field_value_re = re.compile(field_value.encode("ascii")) class Headers(Sequence[Tuple[bytes, bytes]]): """ A list-like interface that allows iterating over headers as byte-pairs of (lowercased-name, value). Internally we actually store the representation as three-tuples, including both the raw original casing, in order to preserve casing over-the-wire, and the lowercased name, for case-insensitive comparisions. r = Request( method="GET", target="/", headers=[("Host", "example.org"), ("Connection", "keep-alive")], http_version="1.1", ) assert r.headers == [ (b"host", b"example.org"), (b"connection", b"keep-alive") ] assert r.headers.raw_items() == [ (b"Host", b"example.org"), (b"Connection", b"keep-alive") ] """ __slots__ = "_full_items" def __init__(self, full_items: List[Tuple[bytes, bytes, bytes]]) -> None: self._full_items = full_items def __bool__(self) -> bool: return bool(self._full_items) def __eq__(self, other: object) -> bool: return list(self) == list(other) # type: ignore def __len__(self) -> int: return len(self._full_items) def __repr__(self) -> str: return "" % repr(list(self)) def __getitem__(self, idx: int) -> Tuple[bytes, bytes]: # type: ignore[override] _, name, value = self._full_items[idx] return (name, value) def raw_items(self) -> List[Tuple[bytes, bytes]]: return [(raw_name, value) for raw_name, _, value in self._full_items] HeaderTypes = Union[ List[Tuple[bytes, bytes]], List[Tuple[bytes, str]], List[Tuple[str, bytes]], List[Tuple[str, str]], ] @overload def normalize_and_validate(headers: Headers, _parsed: Literal[True]) -> Headers: ... @overload def normalize_and_validate(headers: HeaderTypes, _parsed: Literal[False]) -> Headers: ... @overload def normalize_and_validate( headers: Union[Headers, HeaderTypes], _parsed: bool = False ) -> Headers: ... def normalize_and_validate( headers: Union[Headers, HeaderTypes], _parsed: bool = False ) -> Headers: new_headers = [] seen_content_length = None saw_transfer_encoding = False for name, value in headers: # For headers coming out of the parser, we can safely skip some steps, # because it always returns bytes and has already run these regexes # over the data: if not _parsed: name = bytesify(name) value = bytesify(value) validate(_field_name_re, name, "Illegal header name {!r}", name) validate(_field_value_re, value, "Illegal header value {!r}", value) assert isinstance(name, bytes) assert isinstance(value, bytes) raw_name = name name = name.lower() if name == b"content-length": lengths = {length.strip() for length in value.split(b",")} if len(lengths) != 1: raise LocalProtocolError("conflicting Content-Length headers") value = lengths.pop() validate(_content_length_re, value, "bad Content-Length") if seen_content_length is None: seen_content_length = value new_headers.append((raw_name, name, value)) elif seen_content_length != value: raise LocalProtocolError("conflicting Content-Length headers") elif name == b"transfer-encoding": # "A server that receives a request message with a transfer coding # it does not understand SHOULD respond with 501 (Not # Implemented)." # https://tools.ietf.org/html/rfc7230#section-3.3.1 if saw_transfer_encoding: raise LocalProtocolError( "multiple Transfer-Encoding headers", error_status_hint=501 ) # "All transfer-coding names are case-insensitive" # -- https://tools.ietf.org/html/rfc7230#section-4 value = value.lower() if value != b"chunked": raise LocalProtocolError( "Only Transfer-Encoding: chunked is supported", error_status_hint=501, ) saw_transfer_encoding = True new_headers.append((raw_name, name, value)) else: new_headers.append((raw_name, name, value)) return Headers(new_headers) def get_comma_header(headers: Headers, name: bytes) -> List[bytes]: # Should only be used for headers whose value is a list of # comma-separated, case-insensitive values. # # The header name `name` is expected to be lower-case bytes. # # Connection: meets these criteria (including cast insensitivity). # # Content-Length: technically is just a single value (1*DIGIT), but the # standard makes reference to implementations that do multiple values, and # using this doesn't hurt. Ditto, case insensitivity doesn't things either # way. # # Transfer-Encoding: is more complex (allows for quoted strings), so # splitting on , is actually wrong. For example, this is legal: # # Transfer-Encoding: foo; options="1,2", chunked # # and should be parsed as # # foo; options="1,2" # chunked # # but this naive function will parse it as # # foo; options="1 # 2" # chunked # # However, this is okay because the only thing we are going to do with # any Transfer-Encoding is reject ones that aren't just "chunked", so # both of these will be treated the same anyway. # # Expect: the only legal value is the literal string # "100-continue". Splitting on commas is harmless. Case insensitive. # out: List[bytes] = [] for _, found_name, found_raw_value in headers._full_items: if found_name == name: found_raw_value = found_raw_value.lower() for found_split_value in found_raw_value.split(b","): found_split_value = found_split_value.strip() if found_split_value: out.append(found_split_value) return out def set_comma_header(headers: Headers, name: bytes, new_values: List[bytes]) -> Headers: # The header name `name` is expected to be lower-case bytes. # # Note that when we store the header we use title casing for the header # names, in order to match the conventional HTTP header style. # # Simply calling `.title()` is a blunt approach, but it's correct # here given the cases where we're using `set_comma_header`... # # Connection, Content-Length, Transfer-Encoding. new_headers: List[Tuple[bytes, bytes]] = [] for found_raw_name, found_name, found_raw_value in headers._full_items: if found_name != name: new_headers.append((found_raw_name, found_raw_value)) for new_value in new_values: new_headers.append((name.title(), new_value)) return normalize_and_validate(new_headers) def has_expect_100_continue(request: "Request") -> bool: # https://tools.ietf.org/html/rfc7231#section-5.1.1 # "A server that receives a 100-continue expectation in an HTTP/1.0 request # MUST ignore that expectation." if request.http_version < b"1.1": return False expect = get_comma_header(request.headers, b"expect") return b"100-continue" in expect h11-0.13.0/h11/_readers.py000066400000000000000000000202621417207257000147220ustar00rootroot00000000000000# Code to read HTTP data # # Strategy: each reader is a callable which takes a ReceiveBuffer object, and # either: # 1) consumes some of it and returns an Event # 2) raises a LocalProtocolError (for consistency -- e.g. we call validate() # and it might raise a LocalProtocolError, so simpler just to always use # this) # 3) returns None, meaning "I need more data" # # If they have a .read_eof attribute, then this will be called if an EOF is # received -- but this is optional. Either way, the actual ConnectionClosed # event will be generated afterwards. # # READERS is a dict describing how to pick a reader. It maps states to either: # - a reader # - or, for body readers, a dict of per-framing reader factories import re from typing import Any, Callable, Dict, Iterable, NoReturn, Optional, Tuple, Type, Union from ._abnf import chunk_header, header_field, request_line, status_line from ._events import Data, EndOfMessage, InformationalResponse, Request, Response from ._receivebuffer import ReceiveBuffer from ._state import ( CLIENT, CLOSED, DONE, IDLE, MUST_CLOSE, SEND_BODY, SEND_RESPONSE, SERVER, ) from ._util import LocalProtocolError, RemoteProtocolError, Sentinel, validate __all__ = ["READERS"] header_field_re = re.compile(header_field.encode("ascii")) # Remember that this has to run in O(n) time -- so e.g. the bytearray cast is # critical. obs_fold_re = re.compile(br"[ \t]+") def _obsolete_line_fold(lines: Iterable[bytes]) -> Iterable[bytes]: it = iter(lines) last: Optional[bytes] = None for line in it: match = obs_fold_re.match(line) if match: if last is None: raise LocalProtocolError("continuation line at start of headers") if not isinstance(last, bytearray): last = bytearray(last) last += b" " last += line[match.end() :] else: if last is not None: yield last last = line if last is not None: yield last def _decode_header_lines( lines: Iterable[bytes], ) -> Iterable[Tuple[bytes, bytes]]: for line in _obsolete_line_fold(lines): matches = validate(header_field_re, line, "illegal header line: {!r}", line) yield (matches["field_name"], matches["field_value"]) request_line_re = re.compile(request_line.encode("ascii")) def maybe_read_from_IDLE_client(buf: ReceiveBuffer) -> Optional[Request]: lines = buf.maybe_extract_lines() if lines is None: if buf.is_next_line_obviously_invalid_request_line(): raise LocalProtocolError("illegal request line") return None if not lines: raise LocalProtocolError("no request line received") matches = validate( request_line_re, lines[0], "illegal request line: {!r}", lines[0] ) return Request( headers=list(_decode_header_lines(lines[1:])), _parsed=True, **matches ) status_line_re = re.compile(status_line.encode("ascii")) def maybe_read_from_SEND_RESPONSE_server( buf: ReceiveBuffer, ) -> Union[InformationalResponse, Response, None]: lines = buf.maybe_extract_lines() if lines is None: if buf.is_next_line_obviously_invalid_request_line(): raise LocalProtocolError("illegal request line") return None if not lines: raise LocalProtocolError("no response line received") matches = validate(status_line_re, lines[0], "illegal status line: {!r}", lines[0]) http_version = ( b"1.1" if matches["http_version"] is None else matches["http_version"] ) reason = b"" if matches["reason"] is None else matches["reason"] status_code = int(matches["status_code"]) class_: Union[Type[InformationalResponse], Type[Response]] = ( InformationalResponse if status_code < 200 else Response ) return class_( headers=list(_decode_header_lines(lines[1:])), _parsed=True, status_code=status_code, reason=reason, http_version=http_version, ) class ContentLengthReader: def __init__(self, length: int) -> None: self._length = length self._remaining = length def __call__(self, buf: ReceiveBuffer) -> Union[Data, EndOfMessage, None]: if self._remaining == 0: return EndOfMessage() data = buf.maybe_extract_at_most(self._remaining) if data is None: return None self._remaining -= len(data) return Data(data=data) def read_eof(self) -> NoReturn: raise RemoteProtocolError( "peer closed connection without sending complete message body " "(received {} bytes, expected {})".format( self._length - self._remaining, self._length ) ) chunk_header_re = re.compile(chunk_header.encode("ascii")) class ChunkedReader: def __init__(self) -> None: self._bytes_in_chunk = 0 # After reading a chunk, we have to throw away the trailing \r\n; if # this is >0 then we discard that many bytes before resuming regular # de-chunkification. self._bytes_to_discard = 0 self._reading_trailer = False def __call__(self, buf: ReceiveBuffer) -> Union[Data, EndOfMessage, None]: if self._reading_trailer: lines = buf.maybe_extract_lines() if lines is None: return None return EndOfMessage(headers=list(_decode_header_lines(lines))) if self._bytes_to_discard > 0: data = buf.maybe_extract_at_most(self._bytes_to_discard) if data is None: return None self._bytes_to_discard -= len(data) if self._bytes_to_discard > 0: return None # else, fall through and read some more assert self._bytes_to_discard == 0 if self._bytes_in_chunk == 0: # We need to refill our chunk count chunk_header = buf.maybe_extract_next_line() if chunk_header is None: return None matches = validate( chunk_header_re, chunk_header, "illegal chunk header: {!r}", chunk_header, ) # XX FIXME: we discard chunk extensions. Does anyone care? self._bytes_in_chunk = int(matches["chunk_size"], base=16) if self._bytes_in_chunk == 0: self._reading_trailer = True return self(buf) chunk_start = True else: chunk_start = False assert self._bytes_in_chunk > 0 data = buf.maybe_extract_at_most(self._bytes_in_chunk) if data is None: return None self._bytes_in_chunk -= len(data) if self._bytes_in_chunk == 0: self._bytes_to_discard = 2 chunk_end = True else: chunk_end = False return Data(data=data, chunk_start=chunk_start, chunk_end=chunk_end) def read_eof(self) -> NoReturn: raise RemoteProtocolError( "peer closed connection without sending complete message body " "(incomplete chunked read)" ) class Http10Reader: def __call__(self, buf: ReceiveBuffer) -> Optional[Data]: data = buf.maybe_extract_at_most(999999999) if data is None: return None return Data(data=data) def read_eof(self) -> EndOfMessage: return EndOfMessage() def expect_nothing(buf: ReceiveBuffer) -> None: if buf: raise LocalProtocolError("Got data when expecting EOF") return None ReadersType = Dict[ Union[Sentinel, Tuple[Sentinel, Sentinel]], Union[Callable[..., Any], Dict[str, Callable[..., Any]]], ] READERS: ReadersType = { (CLIENT, IDLE): maybe_read_from_IDLE_client, (SERVER, IDLE): maybe_read_from_SEND_RESPONSE_server, (SERVER, SEND_RESPONSE): maybe_read_from_SEND_RESPONSE_server, (CLIENT, DONE): expect_nothing, (CLIENT, MUST_CLOSE): expect_nothing, (CLIENT, CLOSED): expect_nothing, (SERVER, DONE): expect_nothing, (SERVER, MUST_CLOSE): expect_nothing, (SERVER, CLOSED): expect_nothing, SEND_BODY: { "chunked": ChunkedReader, "content-length": ContentLengthReader, "http/1.0": Http10Reader, }, } h11-0.13.0/h11/_receivebuffer.py000066400000000000000000000122041417207257000161060ustar00rootroot00000000000000import re import sys from typing import List, Optional, Union __all__ = ["ReceiveBuffer"] # Operations we want to support: # - find next \r\n or \r\n\r\n (\n or \n\n are also acceptable), # or wait until there is one # - read at-most-N bytes # Goals: # - on average, do this fast # - worst case, do this in O(n) where n is the number of bytes processed # Plan: # - store bytearray, offset, how far we've searched for a separator token # - use the how-far-we've-searched data to avoid rescanning # - while doing a stream of uninterrupted processing, advance offset instead # of constantly copying # WARNING: # - I haven't benchmarked or profiled any of this yet. # # Note that starting in Python 3.4, deleting the initial n bytes from a # bytearray is amortized O(n), thanks to some excellent work by Antoine # Martin: # # https://bugs.python.org/issue19087 # # This means that if we only supported 3.4+, we could get rid of the code here # involving self._start and self.compress, because it's doing exactly the same # thing that bytearray now does internally. # # BUT unfortunately, we still support 2.7, and reading short segments out of a # long buffer MUST be O(bytes read) to avoid DoS issues, so we can't actually # delete this code. Yet: # # https://pythonclock.org/ # # (Two things to double-check first though: make sure PyPy also has the # optimization, and benchmark to make sure it's a win, since we do have a # slightly clever thing where we delay calling compress() until we've # processed a whole event, which could in theory be slightly more efficient # than the internal bytearray support.) blank_line_regex = re.compile(b"\n\r?\n", re.MULTILINE) class ReceiveBuffer: def __init__(self) -> None: self._data = bytearray() self._next_line_search = 0 self._multiple_lines_search = 0 def __iadd__(self, byteslike: Union[bytes, bytearray]) -> "ReceiveBuffer": self._data += byteslike return self def __bool__(self) -> bool: return bool(len(self)) def __len__(self) -> int: return len(self._data) # for @property unprocessed_data def __bytes__(self) -> bytes: return bytes(self._data) def _extract(self, count: int) -> bytearray: # extracting an initial slice of the data buffer and return it out = self._data[:count] del self._data[:count] self._next_line_search = 0 self._multiple_lines_search = 0 return out def maybe_extract_at_most(self, count: int) -> Optional[bytearray]: """ Extract a fixed number of bytes from the buffer. """ out = self._data[:count] if not out: return None return self._extract(count) def maybe_extract_next_line(self) -> Optional[bytearray]: """ Extract the first line, if it is completed in the buffer. """ # Only search in buffer space that we've not already looked at. search_start_index = max(0, self._next_line_search - 1) partial_idx = self._data.find(b"\r\n", search_start_index) if partial_idx == -1: self._next_line_search = len(self._data) return None # + 2 is to compensate len(b"\r\n") idx = partial_idx + 2 return self._extract(idx) def maybe_extract_lines(self) -> Optional[List[bytearray]]: """ Extract everything up to the first blank line, and return a list of lines. """ # Handle the case where we have an immediate empty line. if self._data[:1] == b"\n": self._extract(1) return [] if self._data[:2] == b"\r\n": self._extract(2) return [] # Only search in buffer space that we've not already looked at. match = blank_line_regex.search(self._data, self._multiple_lines_search) if match is None: self._multiple_lines_search = max(0, len(self._data) - 2) return None # Truncate the buffer and return it. idx = match.span(0)[-1] out = self._extract(idx) lines = out.split(b"\n") for line in lines: if line.endswith(b"\r"): del line[-1] assert lines[-2] == lines[-1] == b"" del lines[-2:] return lines # In theory we should wait until `\r\n` before starting to validate # incoming data. However it's interesting to detect (very) invalid data # early given they might not even contain `\r\n` at all (hence only # timeout will get rid of them). # This is not a 100% effective detection but more of a cheap sanity check # allowing for early abort in some useful cases. # This is especially interesting when peer is messing up with HTTPS and # sent us a TLS stream where we were expecting plain HTTP given all # versions of TLS so far start handshake with a 0x16 message type code. def is_next_line_obviously_invalid_request_line(self) -> bool: try: # HTTP header line must not contain non-printable characters # and should not start with a space return self._data[0] < 0x21 except IndexError: return False h11-0.13.0/h11/_state.py000066400000000000000000000316001417207257000144130ustar00rootroot00000000000000################################################################ # The core state machine ################################################################ # # Rule 1: everything that affects the state machine and state transitions must # live here in this file. As much as possible goes into the table-based # representation, but for the bits that don't quite fit, the actual code and # state must nonetheless live here. # # Rule 2: this file does not know about what role we're playing; it only knows # about HTTP request/response cycles in the abstract. This ensures that we # don't cheat and apply different rules to local and remote parties. # # # Theory of operation # =================== # # Possibly the simplest way to think about this is that we actually have 5 # different state machines here. Yes, 5. These are: # # 1) The client state, with its complicated automaton (see the docs) # 2) The server state, with its complicated automaton (see the docs) # 3) The keep-alive state, with possible states {True, False} # 4) The SWITCH_CONNECT state, with possible states {False, True} # 5) The SWITCH_UPGRADE state, with possible states {False, True} # # For (3)-(5), the first state listed is the initial state. # # (1)-(3) are stored explicitly in member variables. The last # two are stored implicitly in the pending_switch_proposals set as: # (state of 4) == (_SWITCH_CONNECT in pending_switch_proposals) # (state of 5) == (_SWITCH_UPGRADE in pending_switch_proposals) # # And each of these machines has two different kinds of transitions: # # a) Event-triggered # b) State-triggered # # Event triggered is the obvious thing that you'd think it is: some event # happens, and if it's the right event at the right time then a transition # happens. But there are somewhat complicated rules for which machines can # "see" which events. (As a rule of thumb, if a machine "sees" an event, this # means two things: the event can affect the machine, and if the machine is # not in a state where it expects that event then it's an error.) These rules # are: # # 1) The client machine sees all h11.events objects emitted by the client. # # 2) The server machine sees all h11.events objects emitted by the server. # # It also sees the client's Request event. # # And sometimes, server events are annotated with a _SWITCH_* event. For # example, we can have a (Response, _SWITCH_CONNECT) event, which is # different from a regular Response event. # # 3) The keep-alive machine sees the process_keep_alive_disabled() event # (which is derived from Request/Response events), and this event # transitions it from True -> False, or from False -> False. There's no way # to transition back. # # 4&5) The _SWITCH_* machines transition from False->True when we get a # Request that proposes the relevant type of switch (via # process_client_switch_proposals), and they go from True->False when we # get a Response that has no _SWITCH_* annotation. # # So that's event-triggered transitions. # # State-triggered transitions are less standard. What they do here is couple # the machines together. The way this works is, when certain *joint* # configurations of states are achieved, then we automatically transition to a # new *joint* state. So, for example, if we're ever in a joint state with # # client: DONE # keep-alive: False # # then the client state immediately transitions to: # # client: MUST_CLOSE # # This is fundamentally different from an event-based transition, because it # doesn't matter how we arrived at the {client: DONE, keep-alive: False} state # -- maybe the client transitioned SEND_BODY -> DONE, or keep-alive # transitioned True -> False. Either way, once this precondition is satisfied, # this transition is immediately triggered. # # What if two conflicting state-based transitions get enabled at the same # time? In practice there's only one case where this arises (client DONE -> # MIGHT_SWITCH_PROTOCOL versus DONE -> MUST_CLOSE), and we resolve it by # explicitly prioritizing the DONE -> MIGHT_SWITCH_PROTOCOL transition. # # Implementation # -------------- # # The event-triggered transitions for the server and client machines are all # stored explicitly in a table. Ditto for the state-triggered transitions that # involve just the server and client state. # # The transitions for the other machines, and the state-triggered transitions # that involve the other machines, are written out as explicit Python code. # # It'd be nice if there were some cleaner way to do all this. This isn't # *too* terrible, but I feel like it could probably be better. # # WARNING # ------- # # The script that generates the state machine diagrams for the docs knows how # to read out the EVENT_TRIGGERED_TRANSITIONS and STATE_TRIGGERED_TRANSITIONS # tables. But it can't automatically read the transitions that are written # directly in Python code. So if you touch those, you need to also update the # script to keep it in sync! from typing import cast, Dict, Optional, Set, Tuple, Type, Union from ._events import * from ._util import LocalProtocolError, Sentinel # Everything in __all__ gets re-exported as part of the h11 public API. __all__ = [ "CLIENT", "SERVER", "IDLE", "SEND_RESPONSE", "SEND_BODY", "DONE", "MUST_CLOSE", "CLOSED", "MIGHT_SWITCH_PROTOCOL", "SWITCHED_PROTOCOL", "ERROR", ] class CLIENT(Sentinel, metaclass=Sentinel): pass class SERVER(Sentinel, metaclass=Sentinel): pass # States class IDLE(Sentinel, metaclass=Sentinel): pass class SEND_RESPONSE(Sentinel, metaclass=Sentinel): pass class SEND_BODY(Sentinel, metaclass=Sentinel): pass class DONE(Sentinel, metaclass=Sentinel): pass class MUST_CLOSE(Sentinel, metaclass=Sentinel): pass class CLOSED(Sentinel, metaclass=Sentinel): pass class ERROR(Sentinel, metaclass=Sentinel): pass # Switch types class MIGHT_SWITCH_PROTOCOL(Sentinel, metaclass=Sentinel): pass class SWITCHED_PROTOCOL(Sentinel, metaclass=Sentinel): pass class _SWITCH_UPGRADE(Sentinel, metaclass=Sentinel): pass class _SWITCH_CONNECT(Sentinel, metaclass=Sentinel): pass EventTransitionType = Dict[ Type[Sentinel], Dict[ Type[Sentinel], Dict[Union[Type[Event], Tuple[Type[Event], Type[Sentinel]]], Type[Sentinel]], ], ] EVENT_TRIGGERED_TRANSITIONS: EventTransitionType = { CLIENT: { IDLE: {Request: SEND_BODY, ConnectionClosed: CLOSED}, SEND_BODY: {Data: SEND_BODY, EndOfMessage: DONE}, DONE: {ConnectionClosed: CLOSED}, MUST_CLOSE: {ConnectionClosed: CLOSED}, CLOSED: {ConnectionClosed: CLOSED}, MIGHT_SWITCH_PROTOCOL: {}, SWITCHED_PROTOCOL: {}, ERROR: {}, }, SERVER: { IDLE: { ConnectionClosed: CLOSED, Response: SEND_BODY, # Special case: server sees client Request events, in this form (Request, CLIENT): SEND_RESPONSE, }, SEND_RESPONSE: { InformationalResponse: SEND_RESPONSE, Response: SEND_BODY, (InformationalResponse, _SWITCH_UPGRADE): SWITCHED_PROTOCOL, (Response, _SWITCH_CONNECT): SWITCHED_PROTOCOL, }, SEND_BODY: {Data: SEND_BODY, EndOfMessage: DONE}, DONE: {ConnectionClosed: CLOSED}, MUST_CLOSE: {ConnectionClosed: CLOSED}, CLOSED: {ConnectionClosed: CLOSED}, SWITCHED_PROTOCOL: {}, ERROR: {}, }, } # NB: there are also some special-case state-triggered transitions hard-coded # into _fire_state_triggered_transitions below. STATE_TRIGGERED_TRANSITIONS = { # (Client state, Server state) -> new states # Protocol negotiation (MIGHT_SWITCH_PROTOCOL, SWITCHED_PROTOCOL): {CLIENT: SWITCHED_PROTOCOL}, # Socket shutdown (CLOSED, DONE): {SERVER: MUST_CLOSE}, (CLOSED, IDLE): {SERVER: MUST_CLOSE}, (ERROR, DONE): {SERVER: MUST_CLOSE}, (DONE, CLOSED): {CLIENT: MUST_CLOSE}, (IDLE, CLOSED): {CLIENT: MUST_CLOSE}, (DONE, ERROR): {CLIENT: MUST_CLOSE}, } class ConnectionState: def __init__(self) -> None: # Extra bits of state that don't quite fit into the state model. # If this is False then it enables the automatic DONE -> MUST_CLOSE # transition. Don't set this directly; call .keep_alive_disabled() self.keep_alive = True # This is a subset of {UPGRADE, CONNECT}, containing the proposals # made by the client for switching protocols. self.pending_switch_proposals: Set[Type[Sentinel]] = set() self.states: Dict[Type[Sentinel], Type[Sentinel]] = {CLIENT: IDLE, SERVER: IDLE} def process_error(self, role: Type[Sentinel]) -> None: self.states[role] = ERROR self._fire_state_triggered_transitions() def process_keep_alive_disabled(self) -> None: self.keep_alive = False self._fire_state_triggered_transitions() def process_client_switch_proposal(self, switch_event: Type[Sentinel]) -> None: self.pending_switch_proposals.add(switch_event) self._fire_state_triggered_transitions() def process_event( self, role: Type[Sentinel], event_type: Type[Event], server_switch_event: Optional[Type[Sentinel]] = None, ) -> None: _event_type: Union[Type[Event], Tuple[Type[Event], Type[Sentinel]]] = event_type if server_switch_event is not None: assert role is SERVER if server_switch_event not in self.pending_switch_proposals: raise LocalProtocolError( "Received server {} event without a pending proposal".format( server_switch_event ) ) _event_type = (event_type, server_switch_event) if server_switch_event is None and _event_type is Response: self.pending_switch_proposals = set() self._fire_event_triggered_transitions(role, _event_type) # Special case: the server state does get to see Request # events. if _event_type is Request: assert role is CLIENT self._fire_event_triggered_transitions(SERVER, (Request, CLIENT)) self._fire_state_triggered_transitions() def _fire_event_triggered_transitions( self, role: Type[Sentinel], event_type: Union[Type[Event], Tuple[Type[Event], Type[Sentinel]]], ) -> None: state = self.states[role] try: new_state = EVENT_TRIGGERED_TRANSITIONS[role][state][event_type] except KeyError: event_type = cast(Type[Event], event_type) raise LocalProtocolError( "can't handle event type {} when role={} and state={}".format( event_type.__name__, role, self.states[role] ) ) from None self.states[role] = new_state def _fire_state_triggered_transitions(self) -> None: # We apply these rules repeatedly until converging on a fixed point while True: start_states = dict(self.states) # It could happen that both these special-case transitions are # enabled at the same time: # # DONE -> MIGHT_SWITCH_PROTOCOL # DONE -> MUST_CLOSE # # For example, this will always be true of a HTTP/1.0 client # requesting CONNECT. If this happens, the protocol switch takes # priority. From there the client will either go to # SWITCHED_PROTOCOL, in which case it's none of our business when # they close the connection, or else the server will deny the # request, in which case the client will go back to DONE and then # from there to MUST_CLOSE. if self.pending_switch_proposals: if self.states[CLIENT] is DONE: self.states[CLIENT] = MIGHT_SWITCH_PROTOCOL if not self.pending_switch_proposals: if self.states[CLIENT] is MIGHT_SWITCH_PROTOCOL: self.states[CLIENT] = DONE if not self.keep_alive: for role in (CLIENT, SERVER): if self.states[role] is DONE: self.states[role] = MUST_CLOSE # Tabular state-triggered transitions joint_state = (self.states[CLIENT], self.states[SERVER]) changes = STATE_TRIGGERED_TRANSITIONS.get(joint_state, {}) self.states.update(changes) # type: ignore if self.states == start_states: # Fixed point reached return def start_next_cycle(self) -> None: if self.states != {CLIENT: DONE, SERVER: DONE}: raise LocalProtocolError( "not in a reusable state. self.states={}".format(self.states) ) # Can't reach DONE/DONE with any of these active, but still, let's be # sure. assert self.keep_alive assert not self.pending_switch_proposals self.states = {CLIENT: IDLE, SERVER: IDLE} h11-0.13.0/h11/_util.py000066400000000000000000000114301417207257000142470ustar00rootroot00000000000000from typing import Any, Dict, NoReturn, Pattern, Tuple, Type, TypeVar, Union __all__ = [ "ProtocolError", "LocalProtocolError", "RemoteProtocolError", "validate", "bytesify", ] class ProtocolError(Exception): """Exception indicating a violation of the HTTP/1.1 protocol. This as an abstract base class, with two concrete base classes: :exc:`LocalProtocolError`, which indicates that you tried to do something that HTTP/1.1 says is illegal, and :exc:`RemoteProtocolError`, which indicates that the remote peer tried to do something that HTTP/1.1 says is illegal. See :ref:`error-handling` for details. In addition to the normal :exc:`Exception` features, it has one attribute: .. attribute:: error_status_hint This gives a suggestion as to what status code a server might use if this error occurred as part of a request. For a :exc:`RemoteProtocolError`, this is useful as a suggestion for how you might want to respond to a misbehaving peer, if you're implementing a server. For a :exc:`LocalProtocolError`, this can be taken as a suggestion for how your peer might have responded to *you* if h11 had allowed you to continue. The default is 400 Bad Request, a generic catch-all for protocol violations. """ def __init__(self, msg: str, error_status_hint: int = 400) -> None: if type(self) is ProtocolError: raise TypeError("tried to directly instantiate ProtocolError") Exception.__init__(self, msg) self.error_status_hint = error_status_hint # Strategy: there are a number of public APIs where a LocalProtocolError can # be raised (send(), all the different event constructors, ...), and only one # public API where RemoteProtocolError can be raised # (receive_data()). Therefore we always raise LocalProtocolError internally, # and then receive_data will translate this into a RemoteProtocolError. # # Internally: # LocalProtocolError is the generic "ProtocolError". # Externally: # LocalProtocolError is for local errors and RemoteProtocolError is for # remote errors. class LocalProtocolError(ProtocolError): def _reraise_as_remote_protocol_error(self) -> NoReturn: # After catching a LocalProtocolError, use this method to re-raise it # as a RemoteProtocolError. This method must be called from inside an # except: block. # # An easy way to get an equivalent RemoteProtocolError is just to # modify 'self' in place. self.__class__ = RemoteProtocolError # type: ignore # But the re-raising is somewhat non-trivial -- you might think that # now that we've modified the in-flight exception object, that just # doing 'raise' to re-raise it would be enough. But it turns out that # this doesn't work, because Python tracks the exception type # (exc_info[0]) separately from the exception object (exc_info[1]), # and we only modified the latter. So we really do need to re-raise # the new type explicitly. # On py3, the traceback is part of the exception object, so our # in-place modification preserved it and we can just re-raise: raise self class RemoteProtocolError(ProtocolError): pass def validate( regex: Pattern[bytes], data: bytes, msg: str = "malformed data", *format_args: Any ) -> Dict[str, bytes]: match = regex.fullmatch(data) if not match: if format_args: msg = msg.format(*format_args) raise LocalProtocolError(msg) return match.groupdict() # Sentinel values # # - Inherit identity-based comparison and hashing from object # - Have a nice repr # - Have a *bonus property*: type(sentinel) is sentinel # # The bonus property is useful if you want to take the return value from # next_event() and do some sort of dispatch based on type(event). _T_Sentinel = TypeVar("_T_Sentinel", bound="Sentinel") class Sentinel(type): def __new__( cls: Type[_T_Sentinel], name: str, bases: Tuple[type, ...], namespace: Dict[str, Any], **kwds: Any ) -> _T_Sentinel: assert bases == (Sentinel,) v = super().__new__(cls, name, bases, namespace, **kwds) v.__class__ = v # type: ignore return v def __repr__(self) -> str: return self.__name__ # Used for methods, request targets, HTTP versions, header names, and header # values. Accepts ascii-strings, or bytes/bytearray/memoryview/..., and always # returns bytes. def bytesify(s: Union[bytes, bytearray, memoryview, int, str]) -> bytes: # Fast-path: if type(s) is bytes: return s if isinstance(s, str): s = s.encode("ascii") if isinstance(s, int): raise TypeError("expected bytes-like object, not int") return bytes(s) h11-0.13.0/h11/_version.py000066400000000000000000000012561417207257000147640ustar00rootroot00000000000000# This file must be kept very simple, because it is consumed from several # places -- it is imported by h11/__init__.py, execfile'd by setup.py, etc. # We use a simple scheme: # 1.0.0 -> 1.0.0+dev -> 1.1.0 -> 1.1.0+dev # where the +dev versions are never released into the wild, they're just what # we stick into the VCS in between releases. # # This is compatible with PEP 440: # http://legacy.python.org/dev/peps/pep-0440/ # via the use of the "local suffix" "+dev", which is disallowed on index # servers and causes 1.0.0+dev to sort after plain 1.0.0, which is what we # want. (Contrast with the special suffix 1.0.0.dev, which sorts *before* # 1.0.0.) __version__ = "0.13.0" h11-0.13.0/h11/_writers.py000066400000000000000000000117071417207257000150000ustar00rootroot00000000000000# Code to read HTTP data # # Strategy: each writer takes an event + a write-some-bytes function, which is # calls. # # WRITERS is a dict describing how to pick a reader. It maps states to either: # - a writer # - or, for body writers, a dict of framin-dependent writer factories from typing import Any, Callable, Dict, List, Tuple, Type, Union from ._events import Data, EndOfMessage, Event, InformationalResponse, Request, Response from ._headers import Headers from ._state import CLIENT, IDLE, SEND_BODY, SEND_RESPONSE, SERVER from ._util import LocalProtocolError, Sentinel __all__ = ["WRITERS"] Writer = Callable[[bytes], Any] def write_headers(headers: Headers, write: Writer) -> None: # "Since the Host field-value is critical information for handling a # request, a user agent SHOULD generate Host as the first header field # following the request-line." - RFC 7230 raw_items = headers._full_items for raw_name, name, value in raw_items: if name == b"host": write(b"%s: %s\r\n" % (raw_name, value)) for raw_name, name, value in raw_items: if name != b"host": write(b"%s: %s\r\n" % (raw_name, value)) write(b"\r\n") def write_request(request: Request, write: Writer) -> None: if request.http_version != b"1.1": raise LocalProtocolError("I only send HTTP/1.1") write(b"%s %s HTTP/1.1\r\n" % (request.method, request.target)) write_headers(request.headers, write) # Shared between InformationalResponse and Response def write_any_response( response: Union[InformationalResponse, Response], write: Writer ) -> None: if response.http_version != b"1.1": raise LocalProtocolError("I only send HTTP/1.1") status_bytes = str(response.status_code).encode("ascii") # We don't bother sending ascii status messages like "OK"; they're # optional and ignored by the protocol. (But the space after the numeric # status code is mandatory.) # # XX FIXME: could at least make an effort to pull out the status message # from stdlib's http.HTTPStatus table. Or maybe just steal their enums # (either by import or copy/paste). We already accept them as status codes # since they're of type IntEnum < int. write(b"HTTP/1.1 %s %s\r\n" % (status_bytes, response.reason)) write_headers(response.headers, write) class BodyWriter: def __call__(self, event: Event, write: Writer) -> None: if type(event) is Data: self.send_data(event.data, write) elif type(event) is EndOfMessage: self.send_eom(event.headers, write) else: # pragma: no cover assert False def send_data(self, data: bytes, write: Writer) -> None: pass def send_eom(self, headers: Headers, write: Writer) -> None: pass # # These are all careful not to do anything to 'data' except call len(data) and # write(data). This allows us to transparently pass-through funny objects, # like placeholder objects referring to files on disk that will be sent via # sendfile(2). # class ContentLengthWriter(BodyWriter): def __init__(self, length: int) -> None: self._length = length def send_data(self, data: bytes, write: Writer) -> None: self._length -= len(data) if self._length < 0: raise LocalProtocolError("Too much data for declared Content-Length") write(data) def send_eom(self, headers: Headers, write: Writer) -> None: if self._length != 0: raise LocalProtocolError("Too little data for declared Content-Length") if headers: raise LocalProtocolError("Content-Length and trailers don't mix") class ChunkedWriter(BodyWriter): def send_data(self, data: bytes, write: Writer) -> None: # if we encoded 0-length data in the naive way, it would look like an # end-of-message. if not data: return write(b"%x\r\n" % len(data)) write(data) write(b"\r\n") def send_eom(self, headers: Headers, write: Writer) -> None: write(b"0\r\n") write_headers(headers, write) class Http10Writer(BodyWriter): def send_data(self, data: bytes, write: Writer) -> None: write(data) def send_eom(self, headers: Headers, write: Writer) -> None: if headers: raise LocalProtocolError("can't send trailers to HTTP/1.0 client") # no need to close the socket ourselves, that will be taken care of by # Connection: close machinery WritersType = Dict[ Union[Tuple[Sentinel, Sentinel], Sentinel], Union[ Dict[str, Type[BodyWriter]], Callable[[Union[InformationalResponse, Response], Writer], None], Callable[[Request, Writer], None], ], ] WRITERS: WritersType = { (CLIENT, IDLE): write_request, (SERVER, IDLE): write_any_response, (SERVER, SEND_RESPONSE): write_any_response, SEND_BODY: { "chunked": ChunkedWriter, "content-length": ContentLengthWriter, "http/1.0": Http10Writer, }, } h11-0.13.0/h11/py.typed000066400000000000000000000000071417207257000142560ustar00rootroot00000000000000Marker h11-0.13.0/h11/tests/000077500000000000000000000000001417207257000137245ustar00rootroot00000000000000h11-0.13.0/h11/tests/__init__.py000066400000000000000000000000001417207257000160230ustar00rootroot00000000000000h11-0.13.0/h11/tests/data/000077500000000000000000000000001417207257000146355ustar00rootroot00000000000000h11-0.13.0/h11/tests/data/test-file000066400000000000000000000001011417207257000164440ustar00rootroot0000000000000092b12bc045050b55b848d37167a1a63947c364579889ce1d39788e45e9fac9e5 h11-0.13.0/h11/tests/helpers.py000066400000000000000000000064331417207257000157460ustar00rootroot00000000000000from typing import cast, List, Type, Union, ValuesView from .._connection import Connection, NEED_DATA, PAUSED from .._events import ( ConnectionClosed, Data, EndOfMessage, Event, InformationalResponse, Request, Response, ) from .._state import CLIENT, CLOSED, DONE, MUST_CLOSE, SERVER from .._util import Sentinel try: from typing import Literal except ImportError: from typing_extensions import Literal # type: ignore def get_all_events(conn: Connection) -> List[Event]: got_events = [] while True: event = conn.next_event() if event in (NEED_DATA, PAUSED): break event = cast(Event, event) got_events.append(event) if type(event) is ConnectionClosed: break return got_events def receive_and_get(conn: Connection, data: bytes) -> List[Event]: conn.receive_data(data) return get_all_events(conn) # Merges adjacent Data events, converts payloads to bytestrings, and removes # chunk boundaries. def normalize_data_events(in_events: List[Event]) -> List[Event]: out_events: List[Event] = [] for event in in_events: if type(event) is Data: event = Data(data=bytes(event.data), chunk_start=False, chunk_end=False) if out_events and type(out_events[-1]) is type(event) is Data: out_events[-1] = Data( data=out_events[-1].data + event.data, chunk_start=out_events[-1].chunk_start, chunk_end=out_events[-1].chunk_end, ) else: out_events.append(event) return out_events # Given that we want to write tests that push some events through a Connection # and check that its state updates appropriately... we might as make a habit # of pushing them through two Connections with a fake network link in # between. class ConnectionPair: def __init__(self) -> None: self.conn = {CLIENT: Connection(CLIENT), SERVER: Connection(SERVER)} self.other = {CLIENT: SERVER, SERVER: CLIENT} @property def conns(self) -> ValuesView[Connection]: return self.conn.values() # expect="match" if expect=send_events; expect=[...] to say what expected def send( self, role: Type[Sentinel], send_events: Union[List[Event], Event], expect: Union[List[Event], Event, Literal["match"]] = "match", ) -> bytes: if not isinstance(send_events, list): send_events = [send_events] data = b"" closed = False for send_event in send_events: new_data = self.conn[role].send(send_event) if new_data is None: closed = True else: data += new_data # send uses b"" to mean b"", and None to mean closed # receive uses b"" to mean closed, and None to mean "try again" # so we have to translate between the two conventions if data: self.conn[self.other[role]].receive_data(data) if closed: self.conn[self.other[role]].receive_data(b"") got_events = get_all_events(self.conn[self.other[role]]) if expect == "match": expect = send_events if not isinstance(expect, list): expect = [expect] assert got_events == expect return data h11-0.13.0/h11/tests/test_against_stdlib_http.py000066400000000000000000000076331417207257000213740ustar00rootroot00000000000000import json import os.path import socket import socketserver import threading from contextlib import closing, contextmanager from http.server import SimpleHTTPRequestHandler from typing import Callable, Generator from urllib.request import urlopen import h11 @contextmanager def socket_server( handler: Callable[..., socketserver.BaseRequestHandler] ) -> Generator[socketserver.TCPServer, None, None]: httpd = socketserver.TCPServer(("127.0.0.1", 0), handler) thread = threading.Thread( target=httpd.serve_forever, kwargs={"poll_interval": 0.01} ) thread.daemon = True try: thread.start() yield httpd finally: httpd.shutdown() test_file_path = os.path.join(os.path.dirname(__file__), "data/test-file") with open(test_file_path, "rb") as f: test_file_data = f.read() class SingleMindedRequestHandler(SimpleHTTPRequestHandler): def translate_path(self, path: str) -> str: return test_file_path def test_h11_as_client() -> None: with socket_server(SingleMindedRequestHandler) as httpd: with closing(socket.create_connection(httpd.server_address)) as s: c = h11.Connection(h11.CLIENT) s.sendall( c.send( # type: ignore[arg-type] h11.Request( method="GET", target="/foo", headers=[("Host", "localhost")] ) ) ) s.sendall(c.send(h11.EndOfMessage())) # type: ignore[arg-type] data = bytearray() while True: event = c.next_event() print(event) if event is h11.NEED_DATA: # Use a small read buffer to make things more challenging # and exercise more paths :-) c.receive_data(s.recv(10)) continue if type(event) is h11.Response: assert event.status_code == 200 if type(event) is h11.Data: data += event.data if type(event) is h11.EndOfMessage: break assert bytes(data) == test_file_data class H11RequestHandler(socketserver.BaseRequestHandler): def handle(self) -> None: with closing(self.request) as s: c = h11.Connection(h11.SERVER) request = None while True: event = c.next_event() if event is h11.NEED_DATA: # Use a small read buffer to make things more challenging # and exercise more paths :-) c.receive_data(s.recv(10)) continue if type(event) is h11.Request: request = event if type(event) is h11.EndOfMessage: break assert request is not None info = json.dumps( { "method": request.method.decode("ascii"), "target": request.target.decode("ascii"), "headers": { name.decode("ascii"): value.decode("ascii") for (name, value) in request.headers }, } ) s.sendall(c.send(h11.Response(status_code=200, headers=[]))) # type: ignore[arg-type] s.sendall(c.send(h11.Data(data=info.encode("ascii")))) s.sendall(c.send(h11.EndOfMessage())) def test_h11_as_server() -> None: with socket_server(H11RequestHandler) as httpd: host, port = httpd.server_address url = "http://{}:{}/some-path".format(host, port) with closing(urlopen(url)) as f: assert f.getcode() == 200 data = f.read() info = json.loads(data.decode("ascii")) print(info) assert info["method"] == "GET" assert info["target"] == "/some-path" assert "urllib" in info["headers"]["user-agent"] h11-0.13.0/h11/tests/test_connection.py000066400000000000000000001135001417207257000174740ustar00rootroot00000000000000from typing import Any, cast, Dict, List, Optional, Tuple, Type import pytest from .._connection import _body_framing, _keep_alive, Connection, NEED_DATA, PAUSED from .._events import ( ConnectionClosed, Data, EndOfMessage, Event, InformationalResponse, Request, Response, ) from .._state import ( CLIENT, CLOSED, DONE, ERROR, IDLE, MIGHT_SWITCH_PROTOCOL, MUST_CLOSE, SEND_BODY, SEND_RESPONSE, SERVER, SWITCHED_PROTOCOL, ) from .._util import LocalProtocolError, RemoteProtocolError, Sentinel from .helpers import ConnectionPair, get_all_events, receive_and_get def test__keep_alive() -> None: assert _keep_alive( Request(method="GET", target="/", headers=[("Host", "Example.com")]) ) assert not _keep_alive( Request( method="GET", target="/", headers=[("Host", "Example.com"), ("Connection", "close")], ) ) assert not _keep_alive( Request( method="GET", target="/", headers=[("Host", "Example.com"), ("Connection", "a, b, cLOse, foo")], ) ) assert not _keep_alive( Request(method="GET", target="/", headers=[], http_version="1.0") # type: ignore[arg-type] ) assert _keep_alive(Response(status_code=200, headers=[])) # type: ignore[arg-type] assert not _keep_alive(Response(status_code=200, headers=[("Connection", "close")])) assert not _keep_alive( Response(status_code=200, headers=[("Connection", "a, b, cLOse, foo")]) ) assert not _keep_alive(Response(status_code=200, headers=[], http_version="1.0")) # type: ignore[arg-type] def test__body_framing() -> None: def headers(cl: Optional[int], te: bool) -> List[Tuple[str, str]]: headers = [] if cl is not None: headers.append(("Content-Length", str(cl))) if te: headers.append(("Transfer-Encoding", "chunked")) return headers def resp( status_code: int = 200, cl: Optional[int] = None, te: bool = False ) -> Response: return Response(status_code=status_code, headers=headers(cl, te)) def req(cl: Optional[int] = None, te: bool = False) -> Request: h = headers(cl, te) h += [("Host", "example.com")] return Request(method="GET", target="/", headers=h) # Special cases where the headers are ignored: for kwargs in [{}, {"cl": 100}, {"te": True}, {"cl": 100, "te": True}]: kwargs = cast(Dict[str, Any], kwargs) for meth, r in [ (b"HEAD", resp(**kwargs)), (b"GET", resp(status_code=204, **kwargs)), (b"GET", resp(status_code=304, **kwargs)), ]: assert _body_framing(meth, r) == ("content-length", (0,)) # Transfer-encoding for kwargs in [{"te": True}, {"cl": 100, "te": True}]: kwargs = cast(Dict[str, Any], kwargs) for meth, r in [(None, req(**kwargs)), (b"GET", resp(**kwargs))]: # type: ignore assert _body_framing(meth, r) == ("chunked", ()) # Content-Length for meth, r in [(None, req(cl=100)), (b"GET", resp(cl=100))]: # type: ignore assert _body_framing(meth, r) == ("content-length", (100,)) # No headers assert _body_framing(None, req()) == ("content-length", (0,)) # type: ignore assert _body_framing(b"GET", resp()) == ("http/1.0", ()) def test_Connection_basics_and_content_length() -> None: with pytest.raises(ValueError): Connection("CLIENT") # type: ignore p = ConnectionPair() assert p.conn[CLIENT].our_role is CLIENT assert p.conn[CLIENT].their_role is SERVER assert p.conn[SERVER].our_role is SERVER assert p.conn[SERVER].their_role is CLIENT data = p.send( CLIENT, Request( method="GET", target="/", headers=[("Host", "example.com"), ("Content-Length", "10")], ), ) assert data == ( b"GET / HTTP/1.1\r\n" b"Host: example.com\r\n" b"Content-Length: 10\r\n\r\n" ) for conn in p.conns: assert conn.states == {CLIENT: SEND_BODY, SERVER: SEND_RESPONSE} assert p.conn[CLIENT].our_state is SEND_BODY assert p.conn[CLIENT].their_state is SEND_RESPONSE assert p.conn[SERVER].our_state is SEND_RESPONSE assert p.conn[SERVER].their_state is SEND_BODY assert p.conn[CLIENT].their_http_version is None assert p.conn[SERVER].their_http_version == b"1.1" data = p.send(SERVER, InformationalResponse(status_code=100, headers=[])) # type: ignore[arg-type] assert data == b"HTTP/1.1 100 \r\n\r\n" data = p.send(SERVER, Response(status_code=200, headers=[("Content-Length", "11")])) assert data == b"HTTP/1.1 200 \r\nContent-Length: 11\r\n\r\n" for conn in p.conns: assert conn.states == {CLIENT: SEND_BODY, SERVER: SEND_BODY} assert p.conn[CLIENT].their_http_version == b"1.1" assert p.conn[SERVER].their_http_version == b"1.1" data = p.send(CLIENT, Data(data=b"12345")) assert data == b"12345" data = p.send( CLIENT, Data(data=b"67890"), expect=[Data(data=b"67890"), EndOfMessage()] ) assert data == b"67890" data = p.send(CLIENT, EndOfMessage(), expect=[]) assert data == b"" for conn in p.conns: assert conn.states == {CLIENT: DONE, SERVER: SEND_BODY} data = p.send(SERVER, Data(data=b"1234567890")) assert data == b"1234567890" data = p.send(SERVER, Data(data=b"1"), expect=[Data(data=b"1"), EndOfMessage()]) assert data == b"1" data = p.send(SERVER, EndOfMessage(), expect=[]) assert data == b"" for conn in p.conns: assert conn.states == {CLIENT: DONE, SERVER: DONE} def test_chunked() -> None: p = ConnectionPair() p.send( CLIENT, Request( method="GET", target="/", headers=[("Host", "example.com"), ("Transfer-Encoding", "chunked")], ), ) data = p.send(CLIENT, Data(data=b"1234567890", chunk_start=True, chunk_end=True)) assert data == b"a\r\n1234567890\r\n" data = p.send(CLIENT, Data(data=b"abcde", chunk_start=True, chunk_end=True)) assert data == b"5\r\nabcde\r\n" data = p.send(CLIENT, Data(data=b""), expect=[]) assert data == b"" data = p.send(CLIENT, EndOfMessage(headers=[("hello", "there")])) assert data == b"0\r\nhello: there\r\n\r\n" p.send( SERVER, Response(status_code=200, headers=[("Transfer-Encoding", "chunked")]) ) p.send(SERVER, Data(data=b"54321", chunk_start=True, chunk_end=True)) p.send(SERVER, Data(data=b"12345", chunk_start=True, chunk_end=True)) p.send(SERVER, EndOfMessage()) for conn in p.conns: assert conn.states == {CLIENT: DONE, SERVER: DONE} def test_chunk_boundaries() -> None: conn = Connection(our_role=SERVER) request = ( b"POST / HTTP/1.1\r\n" b"Host: example.com\r\n" b"Transfer-Encoding: chunked\r\n" b"\r\n" ) conn.receive_data(request) assert conn.next_event() == Request( method="POST", target="/", headers=[("Host", "example.com"), ("Transfer-Encoding", "chunked")], ) assert conn.next_event() is NEED_DATA conn.receive_data(b"5\r\nhello\r\n") assert conn.next_event() == Data(data=b"hello", chunk_start=True, chunk_end=True) conn.receive_data(b"5\r\nhel") assert conn.next_event() == Data(data=b"hel", chunk_start=True, chunk_end=False) conn.receive_data(b"l") assert conn.next_event() == Data(data=b"l", chunk_start=False, chunk_end=False) conn.receive_data(b"o\r\n") assert conn.next_event() == Data(data=b"o", chunk_start=False, chunk_end=True) conn.receive_data(b"5\r\nhello") assert conn.next_event() == Data(data=b"hello", chunk_start=True, chunk_end=True) conn.receive_data(b"\r\n") assert conn.next_event() == NEED_DATA conn.receive_data(b"0\r\n\r\n") assert conn.next_event() == EndOfMessage() def test_client_talking_to_http10_server() -> None: c = Connection(CLIENT) c.send(Request(method="GET", target="/", headers=[("Host", "example.com")])) c.send(EndOfMessage()) assert c.our_state is DONE # No content-length, so Http10 framing for body assert receive_and_get(c, b"HTTP/1.0 200 OK\r\n\r\n") == [ Response(status_code=200, headers=[], http_version="1.0", reason=b"OK") # type: ignore[arg-type] ] assert c.our_state is MUST_CLOSE assert receive_and_get(c, b"12345") == [Data(data=b"12345")] assert receive_and_get(c, b"67890") == [Data(data=b"67890")] assert receive_and_get(c, b"") == [EndOfMessage(), ConnectionClosed()] assert c.their_state is CLOSED def test_server_talking_to_http10_client() -> None: c = Connection(SERVER) # No content-length, so no body # NB: no host header assert receive_and_get(c, b"GET / HTTP/1.0\r\n\r\n") == [ Request(method="GET", target="/", headers=[], http_version="1.0"), # type: ignore[arg-type] EndOfMessage(), ] assert c.their_state is MUST_CLOSE # We automatically Connection: close back at them assert ( c.send(Response(status_code=200, headers=[])) # type: ignore[arg-type] == b"HTTP/1.1 200 \r\nConnection: close\r\n\r\n" ) assert c.send(Data(data=b"12345")) == b"12345" assert c.send(EndOfMessage()) == b"" assert c.our_state is MUST_CLOSE # Check that it works if they do send Content-Length c = Connection(SERVER) # NB: no host header assert receive_and_get(c, b"POST / HTTP/1.0\r\nContent-Length: 10\r\n\r\n1") == [ Request( method="POST", target="/", headers=[("Content-Length", "10")], http_version="1.0", ), Data(data=b"1"), ] assert receive_and_get(c, b"234567890") == [Data(data=b"234567890"), EndOfMessage()] assert c.their_state is MUST_CLOSE assert receive_and_get(c, b"") == [ConnectionClosed()] def test_automatic_transfer_encoding_in_response() -> None: # Check that in responses, the user can specify either Transfer-Encoding: # chunked or no framing at all, and in both cases we automatically select # the right option depending on whether the peer speaks HTTP/1.0 or # HTTP/1.1 for user_headers in [ [("Transfer-Encoding", "chunked")], [], # In fact, this even works if Content-Length is set, # because if both are set then Transfer-Encoding wins [("Transfer-Encoding", "chunked"), ("Content-Length", "100")], ]: user_headers = cast(List[Tuple[str, str]], user_headers) p = ConnectionPair() p.send( CLIENT, [ Request(method="GET", target="/", headers=[("Host", "example.com")]), EndOfMessage(), ], ) # When speaking to HTTP/1.1 client, all of the above cases get # normalized to Transfer-Encoding: chunked p.send( SERVER, Response(status_code=200, headers=user_headers), expect=Response( status_code=200, headers=[("Transfer-Encoding", "chunked")] ), ) # When speaking to HTTP/1.0 client, all of the above cases get # normalized to no-framing-headers c = Connection(SERVER) receive_and_get(c, b"GET / HTTP/1.0\r\n\r\n") assert ( c.send(Response(status_code=200, headers=user_headers)) == b"HTTP/1.1 200 \r\nConnection: close\r\n\r\n" ) assert c.send(Data(data=b"12345")) == b"12345" def test_automagic_connection_close_handling() -> None: p = ConnectionPair() # If the user explicitly sets Connection: close, then we notice and # respect it p.send( CLIENT, [ Request( method="GET", target="/", headers=[("Host", "example.com"), ("Connection", "close")], ), EndOfMessage(), ], ) for conn in p.conns: assert conn.states[CLIENT] is MUST_CLOSE # And if the client sets it, the server automatically echoes it back p.send( SERVER, # no header here... [Response(status_code=204, headers=[]), EndOfMessage()], # type: ignore[arg-type] # ...but oh look, it arrived anyway expect=[ Response(status_code=204, headers=[("connection", "close")]), EndOfMessage(), ], ) for conn in p.conns: assert conn.states == {CLIENT: MUST_CLOSE, SERVER: MUST_CLOSE} def test_100_continue() -> None: def setup() -> ConnectionPair: p = ConnectionPair() p.send( CLIENT, Request( method="GET", target="/", headers=[ ("Host", "example.com"), ("Content-Length", "100"), ("Expect", "100-continue"), ], ), ) for conn in p.conns: assert conn.client_is_waiting_for_100_continue assert not p.conn[CLIENT].they_are_waiting_for_100_continue assert p.conn[SERVER].they_are_waiting_for_100_continue return p # Disabled by 100 Continue p = setup() p.send(SERVER, InformationalResponse(status_code=100, headers=[])) # type: ignore[arg-type] for conn in p.conns: assert not conn.client_is_waiting_for_100_continue assert not conn.they_are_waiting_for_100_continue # Disabled by a real response p = setup() p.send( SERVER, Response(status_code=200, headers=[("Transfer-Encoding", "chunked")]) ) for conn in p.conns: assert not conn.client_is_waiting_for_100_continue assert not conn.they_are_waiting_for_100_continue # Disabled by the client going ahead and sending stuff anyway p = setup() p.send(CLIENT, Data(data=b"12345")) for conn in p.conns: assert not conn.client_is_waiting_for_100_continue assert not conn.they_are_waiting_for_100_continue def test_max_incomplete_event_size_countermeasure() -> None: # Infinitely long headers are definitely not okay c = Connection(SERVER) c.receive_data(b"GET / HTTP/1.0\r\nEndless: ") assert c.next_event() is NEED_DATA with pytest.raises(RemoteProtocolError): while True: c.receive_data(b"a" * 1024) c.next_event() # Checking that the same header is accepted / rejected depending on the # max_incomplete_event_size setting: c = Connection(SERVER, max_incomplete_event_size=5000) c.receive_data(b"GET / HTTP/1.0\r\nBig: ") c.receive_data(b"a" * 4000) c.receive_data(b"\r\n\r\n") assert get_all_events(c) == [ Request( method="GET", target="/", http_version="1.0", headers=[("big", "a" * 4000)] ), EndOfMessage(), ] c = Connection(SERVER, max_incomplete_event_size=4000) c.receive_data(b"GET / HTTP/1.0\r\nBig: ") c.receive_data(b"a" * 4000) with pytest.raises(RemoteProtocolError): c.next_event() # Temporarily exceeding the size limit is fine, as long as its done with # complete events: c = Connection(SERVER, max_incomplete_event_size=5000) c.receive_data(b"GET / HTTP/1.0\r\nContent-Length: 10000") c.receive_data(b"\r\n\r\n" + b"a" * 10000) assert get_all_events(c) == [ Request( method="GET", target="/", http_version="1.0", headers=[("Content-Length", "10000")], ), Data(data=b"a" * 10000), EndOfMessage(), ] c = Connection(SERVER, max_incomplete_event_size=100) # Two pipelined requests to create a way-too-big receive buffer... but # it's fine because we're not checking c.receive_data( b"GET /1 HTTP/1.1\r\nHost: a\r\n\r\n" b"GET /2 HTTP/1.1\r\nHost: b\r\n\r\n" + b"X" * 1000 ) assert get_all_events(c) == [ Request(method="GET", target="/1", headers=[("host", "a")]), EndOfMessage(), ] # Even more data comes in, still no problem c.receive_data(b"X" * 1000) # We can respond and reuse to get the second pipelined request c.send(Response(status_code=200, headers=[])) # type: ignore[arg-type] c.send(EndOfMessage()) c.start_next_cycle() assert get_all_events(c) == [ Request(method="GET", target="/2", headers=[("host", "b")]), EndOfMessage(), ] # But once we unpause and try to read the next message, and find that it's # incomplete and the buffer is *still* way too large, then *that's* a # problem: c.send(Response(status_code=200, headers=[])) # type: ignore[arg-type] c.send(EndOfMessage()) c.start_next_cycle() with pytest.raises(RemoteProtocolError): c.next_event() def test_reuse_simple() -> None: p = ConnectionPair() p.send( CLIENT, [Request(method="GET", target="/", headers=[("Host", "a")]), EndOfMessage()], ) p.send( SERVER, [ Response(status_code=200, headers=[(b"transfer-encoding", b"chunked")]), EndOfMessage(), ], ) for conn in p.conns: assert conn.states == {CLIENT: DONE, SERVER: DONE} conn.start_next_cycle() p.send( CLIENT, [ Request(method="DELETE", target="/foo", headers=[("Host", "a")]), EndOfMessage(), ], ) p.send( SERVER, [ Response(status_code=404, headers=[(b"transfer-encoding", b"chunked")]), EndOfMessage(), ], ) def test_pipelining() -> None: # Client doesn't support pipelining, so we have to do this by hand c = Connection(SERVER) assert c.next_event() is NEED_DATA # 3 requests all bunched up c.receive_data( b"GET /1 HTTP/1.1\r\nHost: a.com\r\nContent-Length: 5\r\n\r\n" b"12345" b"GET /2 HTTP/1.1\r\nHost: a.com\r\nContent-Length: 5\r\n\r\n" b"67890" b"GET /3 HTTP/1.1\r\nHost: a.com\r\n\r\n" ) assert get_all_events(c) == [ Request( method="GET", target="/1", headers=[("Host", "a.com"), ("Content-Length", "5")], ), Data(data=b"12345"), EndOfMessage(), ] assert c.their_state is DONE assert c.our_state is SEND_RESPONSE assert c.next_event() is PAUSED c.send(Response(status_code=200, headers=[])) # type: ignore[arg-type] c.send(EndOfMessage()) assert c.their_state is DONE assert c.our_state is DONE c.start_next_cycle() assert get_all_events(c) == [ Request( method="GET", target="/2", headers=[("Host", "a.com"), ("Content-Length", "5")], ), Data(data=b"67890"), EndOfMessage(), ] assert c.next_event() is PAUSED c.send(Response(status_code=200, headers=[])) # type: ignore[arg-type] c.send(EndOfMessage()) c.start_next_cycle() assert get_all_events(c) == [ Request(method="GET", target="/3", headers=[("Host", "a.com")]), EndOfMessage(), ] # Doesn't pause this time, no trailing data assert c.next_event() is NEED_DATA c.send(Response(status_code=200, headers=[])) # type: ignore[arg-type] c.send(EndOfMessage()) # Arrival of more data triggers pause assert c.next_event() is NEED_DATA c.receive_data(b"SADF") assert c.next_event() is PAUSED assert c.trailing_data == (b"SADF", False) # If EOF arrives while paused, we don't see that either: c.receive_data(b"") assert c.trailing_data == (b"SADF", True) assert c.next_event() is PAUSED c.receive_data(b"") assert c.next_event() is PAUSED # Can't call receive_data with non-empty buf after closing it with pytest.raises(RuntimeError): c.receive_data(b"FDSA") def test_protocol_switch() -> None: for (req, deny, accept) in [ ( Request( method="CONNECT", target="example.com:443", headers=[("Host", "foo"), ("Content-Length", "1")], ), Response(status_code=404, headers=[(b"transfer-encoding", b"chunked")]), Response(status_code=200, headers=[(b"transfer-encoding", b"chunked")]), ), ( Request( method="GET", target="/", headers=[("Host", "foo"), ("Content-Length", "1"), ("Upgrade", "a, b")], ), Response(status_code=200, headers=[(b"transfer-encoding", b"chunked")]), InformationalResponse(status_code=101, headers=[("Upgrade", "a")]), ), ( Request( method="CONNECT", target="example.com:443", headers=[("Host", "foo"), ("Content-Length", "1"), ("Upgrade", "a, b")], ), Response(status_code=404, headers=[(b"transfer-encoding", b"chunked")]), # Accept CONNECT, not upgrade Response(status_code=200, headers=[(b"transfer-encoding", b"chunked")]), ), ( Request( method="CONNECT", target="example.com:443", headers=[("Host", "foo"), ("Content-Length", "1"), ("Upgrade", "a, b")], ), Response(status_code=404, headers=[(b"transfer-encoding", b"chunked")]), # Accept Upgrade, not CONNECT InformationalResponse(status_code=101, headers=[("Upgrade", "b")]), ), ]: def setup() -> ConnectionPair: p = ConnectionPair() p.send(CLIENT, req) # No switch-related state change stuff yet; the client has to # finish the request before that kicks in for conn in p.conns: assert conn.states[CLIENT] is SEND_BODY p.send(CLIENT, [Data(data=b"1"), EndOfMessage()]) for conn in p.conns: assert conn.states[CLIENT] is MIGHT_SWITCH_PROTOCOL assert p.conn[SERVER].next_event() is PAUSED return p # Test deny case p = setup() p.send(SERVER, deny) for conn in p.conns: assert conn.states == {CLIENT: DONE, SERVER: SEND_BODY} p.send(SERVER, EndOfMessage()) # Check that re-use is still allowed after a denial for conn in p.conns: conn.start_next_cycle() # Test accept case p = setup() p.send(SERVER, accept) for conn in p.conns: assert conn.states == {CLIENT: SWITCHED_PROTOCOL, SERVER: SWITCHED_PROTOCOL} conn.receive_data(b"123") assert conn.next_event() is PAUSED conn.receive_data(b"456") assert conn.next_event() is PAUSED assert conn.trailing_data == (b"123456", False) # Pausing in might-switch, then recovery # (weird artificial case where the trailing data actually is valid # HTTP for some reason, because this makes it easier to test the state # logic) p = setup() sc = p.conn[SERVER] sc.receive_data(b"GET / HTTP/1.0\r\n\r\n") assert sc.next_event() is PAUSED assert sc.trailing_data == (b"GET / HTTP/1.0\r\n\r\n", False) sc.send(deny) assert sc.next_event() is PAUSED sc.send(EndOfMessage()) sc.start_next_cycle() assert get_all_events(sc) == [ Request(method="GET", target="/", headers=[], http_version="1.0"), # type: ignore[arg-type] EndOfMessage(), ] # When we're DONE, have no trailing data, and the connection gets # closed, we report ConnectionClosed(). When we're in might-switch or # switched, we don't. p = setup() sc = p.conn[SERVER] sc.receive_data(b"") assert sc.next_event() is PAUSED assert sc.trailing_data == (b"", True) p.send(SERVER, accept) assert sc.next_event() is PAUSED p = setup() sc = p.conn[SERVER] sc.receive_data(b"") assert sc.next_event() is PAUSED sc.send(deny) assert sc.next_event() == ConnectionClosed() # You can't send after switching protocols, or while waiting for a # protocol switch p = setup() with pytest.raises(LocalProtocolError): p.conn[CLIENT].send( Request(method="GET", target="/", headers=[("Host", "a")]) ) p = setup() p.send(SERVER, accept) with pytest.raises(LocalProtocolError): p.conn[SERVER].send(Data(data=b"123")) def test_close_simple() -> None: # Just immediately closing a new connection without anything having # happened yet. for (who_shot_first, who_shot_second) in [(CLIENT, SERVER), (SERVER, CLIENT)]: def setup() -> ConnectionPair: p = ConnectionPair() p.send(who_shot_first, ConnectionClosed()) for conn in p.conns: assert conn.states == { who_shot_first: CLOSED, who_shot_second: MUST_CLOSE, } return p # You can keep putting b"" into a closed connection, and you keep # getting ConnectionClosed() out: p = setup() assert p.conn[who_shot_second].next_event() == ConnectionClosed() assert p.conn[who_shot_second].next_event() == ConnectionClosed() p.conn[who_shot_second].receive_data(b"") assert p.conn[who_shot_second].next_event() == ConnectionClosed() # Second party can close... p = setup() p.send(who_shot_second, ConnectionClosed()) for conn in p.conns: assert conn.our_state is CLOSED assert conn.their_state is CLOSED # But trying to receive new data on a closed connection is a # RuntimeError (not ProtocolError, because the problem here isn't # violation of HTTP, it's violation of physics) p = setup() with pytest.raises(RuntimeError): p.conn[who_shot_second].receive_data(b"123") # And receiving new data on a MUST_CLOSE connection is a ProtocolError p = setup() p.conn[who_shot_first].receive_data(b"GET") with pytest.raises(RemoteProtocolError): p.conn[who_shot_first].next_event() def test_close_different_states() -> None: req = [ Request(method="GET", target="/foo", headers=[("Host", "a")]), EndOfMessage(), ] resp = [ Response(status_code=200, headers=[(b"transfer-encoding", b"chunked")]), EndOfMessage(), ] # Client before request p = ConnectionPair() p.send(CLIENT, ConnectionClosed()) for conn in p.conns: assert conn.states == {CLIENT: CLOSED, SERVER: MUST_CLOSE} # Client after request p = ConnectionPair() p.send(CLIENT, req) p.send(CLIENT, ConnectionClosed()) for conn in p.conns: assert conn.states == {CLIENT: CLOSED, SERVER: SEND_RESPONSE} # Server after request -> not allowed p = ConnectionPair() p.send(CLIENT, req) with pytest.raises(LocalProtocolError): p.conn[SERVER].send(ConnectionClosed()) p.conn[CLIENT].receive_data(b"") with pytest.raises(RemoteProtocolError): p.conn[CLIENT].next_event() # Server after response p = ConnectionPair() p.send(CLIENT, req) p.send(SERVER, resp) p.send(SERVER, ConnectionClosed()) for conn in p.conns: assert conn.states == {CLIENT: MUST_CLOSE, SERVER: CLOSED} # Both after closing (ConnectionClosed() is idempotent) p = ConnectionPair() p.send(CLIENT, req) p.send(SERVER, resp) p.send(CLIENT, ConnectionClosed()) p.send(SERVER, ConnectionClosed()) p.send(CLIENT, ConnectionClosed()) p.send(SERVER, ConnectionClosed()) # In the middle of sending -> not allowed p = ConnectionPair() p.send( CLIENT, Request( method="GET", target="/", headers=[("Host", "a"), ("Content-Length", "10")] ), ) with pytest.raises(LocalProtocolError): p.conn[CLIENT].send(ConnectionClosed()) p.conn[SERVER].receive_data(b"") with pytest.raises(RemoteProtocolError): p.conn[SERVER].next_event() # Receive several requests and then client shuts down their side of the # connection; we can respond to each def test_pipelined_close() -> None: c = Connection(SERVER) # 2 requests then a close c.receive_data( b"GET /1 HTTP/1.1\r\nHost: a.com\r\nContent-Length: 5\r\n\r\n" b"12345" b"GET /2 HTTP/1.1\r\nHost: a.com\r\nContent-Length: 5\r\n\r\n" b"67890" ) c.receive_data(b"") assert get_all_events(c) == [ Request( method="GET", target="/1", headers=[("host", "a.com"), ("content-length", "5")], ), Data(data=b"12345"), EndOfMessage(), ] assert c.states[CLIENT] is DONE c.send(Response(status_code=200, headers=[])) # type: ignore[arg-type] c.send(EndOfMessage()) assert c.states[SERVER] is DONE c.start_next_cycle() assert get_all_events(c) == [ Request( method="GET", target="/2", headers=[("host", "a.com"), ("content-length", "5")], ), Data(data=b"67890"), EndOfMessage(), ConnectionClosed(), ] assert c.states == {CLIENT: CLOSED, SERVER: SEND_RESPONSE} c.send(Response(status_code=200, headers=[])) # type: ignore[arg-type] c.send(EndOfMessage()) assert c.states == {CLIENT: CLOSED, SERVER: MUST_CLOSE} c.send(ConnectionClosed()) assert c.states == {CLIENT: CLOSED, SERVER: CLOSED} def test_sendfile() -> None: class SendfilePlaceholder: def __len__(self) -> int: return 10 placeholder = SendfilePlaceholder() def setup( header: Tuple[str, str], http_version: str ) -> Tuple[Connection, Optional[List[bytes]]]: c = Connection(SERVER) receive_and_get( c, "GET / HTTP/{}\r\nHost: a\r\n\r\n".format(http_version).encode("ascii") ) headers = [] if header: headers.append(header) c.send(Response(status_code=200, headers=headers)) return c, c.send_with_data_passthrough(Data(data=placeholder)) # type: ignore c, data = setup(("Content-Length", "10"), "1.1") assert data == [placeholder] # type: ignore # Raises an error if the connection object doesn't think we've sent # exactly 10 bytes c.send(EndOfMessage()) _, data = setup(("Transfer-Encoding", "chunked"), "1.1") assert placeholder in data # type: ignore data[data.index(placeholder)] = b"x" * 10 # type: ignore assert b"".join(data) == b"a\r\nxxxxxxxxxx\r\n" # type: ignore c, data = setup(None, "1.0") # type: ignore assert data == [placeholder] # type: ignore assert c.our_state is SEND_BODY def test_errors() -> None: # After a receive error, you can't receive for role in [CLIENT, SERVER]: c = Connection(our_role=role) c.receive_data(b"gibberish\r\n\r\n") with pytest.raises(RemoteProtocolError): c.next_event() # Now any attempt to receive continues to raise assert c.their_state is ERROR assert c.our_state is not ERROR print(c._cstate.states) with pytest.raises(RemoteProtocolError): c.next_event() # But we can still yell at the client for sending us gibberish if role is SERVER: assert ( c.send(Response(status_code=400, headers=[])) # type: ignore[arg-type] == b"HTTP/1.1 400 \r\nConnection: close\r\n\r\n" ) # After an error sending, you can no longer send # (This is especially important for things like content-length errors, # where there's complex internal state being modified) def conn(role: Type[Sentinel]) -> Connection: c = Connection(our_role=role) if role is SERVER: # Put it into the state where it *could* send a response... receive_and_get(c, b"GET / HTTP/1.0\r\n\r\n") assert c.our_state is SEND_RESPONSE return c for role in [CLIENT, SERVER]: if role is CLIENT: # This HTTP/1.0 request won't be detected as bad until after we go # through the state machine and hit the writing code good = Request(method="GET", target="/", headers=[("Host", "example.com")]) bad = Request( method="GET", target="/", headers=[("Host", "example.com")], http_version="1.0", ) elif role is SERVER: good = Response(status_code=200, headers=[]) # type: ignore[arg-type,assignment] bad = Response(status_code=200, headers=[], http_version="1.0") # type: ignore[arg-type,assignment] # Make sure 'good' actually is good c = conn(role) c.send(good) assert c.our_state is not ERROR # Do that again, but this time sending 'bad' first c = conn(role) with pytest.raises(LocalProtocolError): c.send(bad) assert c.our_state is ERROR assert c.their_state is not ERROR # Now 'good' is not so good with pytest.raises(LocalProtocolError): c.send(good) # And check send_failed() too c = conn(role) c.send_failed() assert c.our_state is ERROR assert c.their_state is not ERROR # This is idempotent c.send_failed() assert c.our_state is ERROR assert c.their_state is not ERROR def test_idle_receive_nothing() -> None: # At one point this incorrectly raised an error for role in [CLIENT, SERVER]: c = Connection(role) assert c.next_event() is NEED_DATA def test_connection_drop() -> None: c = Connection(SERVER) c.receive_data(b"GET /") assert c.next_event() is NEED_DATA c.receive_data(b"") with pytest.raises(RemoteProtocolError): c.next_event() def test_408_request_timeout() -> None: # Should be able to send this spontaneously as a server without seeing # anything from client p = ConnectionPair() p.send(SERVER, Response(status_code=408, headers=[(b"connection", b"close")])) # This used to raise IndexError def test_empty_request() -> None: c = Connection(SERVER) c.receive_data(b"\r\n") with pytest.raises(RemoteProtocolError): c.next_event() # This used to raise IndexError def test_empty_response() -> None: c = Connection(CLIENT) c.send(Request(method="GET", target="/", headers=[("Host", "a")])) c.receive_data(b"\r\n") with pytest.raises(RemoteProtocolError): c.next_event() @pytest.mark.parametrize( "data", [ b"\x00", b"\x20", b"\x16\x03\x01\x00\xa5", # Typical start of a TLS Client Hello ], ) def test_early_detection_of_invalid_request(data: bytes) -> None: c = Connection(SERVER) # Early detection should occur before even receiving a `\r\n` c.receive_data(data) with pytest.raises(RemoteProtocolError): c.next_event() @pytest.mark.parametrize( "data", [ b"\x00", b"\x20", b"\x16\x03\x03\x00\x31", # Typical start of a TLS Server Hello ], ) def test_early_detection_of_invalid_response(data: bytes) -> None: c = Connection(CLIENT) # Early detection should occur before even receiving a `\r\n` c.receive_data(data) with pytest.raises(RemoteProtocolError): c.next_event() # This used to give different headers for HEAD and GET. # The correct way to handle HEAD is to put whatever headers we *would* have # put if it were a GET -- even though we know that for HEAD, those headers # will be ignored. def test_HEAD_framing_headers() -> None: def setup(method: bytes, http_version: bytes) -> Connection: c = Connection(SERVER) c.receive_data( method + b" / HTTP/" + http_version + b"\r\n" + b"Host: example.com\r\n\r\n" ) assert type(c.next_event()) is Request assert type(c.next_event()) is EndOfMessage return c for method in [b"GET", b"HEAD"]: # No Content-Length, HTTP/1.1 peer, should use chunked c = setup(method, b"1.1") assert ( c.send(Response(status_code=200, headers=[])) == b"HTTP/1.1 200 \r\n" # type: ignore[arg-type] b"Transfer-Encoding: chunked\r\n\r\n" ) # No Content-Length, HTTP/1.0 peer, frame with connection: close c = setup(method, b"1.0") assert ( c.send(Response(status_code=200, headers=[])) == b"HTTP/1.1 200 \r\n" # type: ignore[arg-type] b"Connection: close\r\n\r\n" ) # Content-Length + Transfer-Encoding, TE wins c = setup(method, b"1.1") assert ( c.send( Response( status_code=200, headers=[ ("Content-Length", "100"), ("Transfer-Encoding", "chunked"), ], ) ) == b"HTTP/1.1 200 \r\n" b"Transfer-Encoding: chunked\r\n\r\n" ) def test_special_exceptions_for_lost_connection_in_message_body() -> None: c = Connection(SERVER) c.receive_data( b"POST / HTTP/1.1\r\n" b"Host: example.com\r\n" b"Content-Length: 100\r\n\r\n" ) assert type(c.next_event()) is Request assert c.next_event() is NEED_DATA c.receive_data(b"12345") assert c.next_event() == Data(data=b"12345") c.receive_data(b"") with pytest.raises(RemoteProtocolError) as excinfo: c.next_event() assert "received 5 bytes" in str(excinfo.value) assert "expected 100" in str(excinfo.value) c = Connection(SERVER) c.receive_data( b"POST / HTTP/1.1\r\n" b"Host: example.com\r\n" b"Transfer-Encoding: chunked\r\n\r\n" ) assert type(c.next_event()) is Request assert c.next_event() is NEED_DATA c.receive_data(b"8\r\n012345") assert c.next_event().data == b"012345" # type: ignore c.receive_data(b"") with pytest.raises(RemoteProtocolError) as excinfo: c.next_event() assert "incomplete chunked read" in str(excinfo.value) h11-0.13.0/h11/tests/test_events.py000066400000000000000000000110611417207257000166400ustar00rootroot00000000000000from http import HTTPStatus import pytest from .. import _events from .._events import ( ConnectionClosed, Data, EndOfMessage, Event, InformationalResponse, Request, Response, ) from .._util import LocalProtocolError def test_events() -> None: with pytest.raises(LocalProtocolError): # Missing Host: req = Request( method="GET", target="/", headers=[("a", "b")], http_version="1.1" ) # But this is okay (HTTP/1.0) req = Request(method="GET", target="/", headers=[("a", "b")], http_version="1.0") # fields are normalized assert req.method == b"GET" assert req.target == b"/" assert req.headers == [(b"a", b"b")] assert req.http_version == b"1.0" # This is also okay -- has a Host (with weird capitalization, which is ok) req = Request( method="GET", target="/", headers=[("a", "b"), ("hOSt", "example.com")], http_version="1.1", ) # we normalize header capitalization assert req.headers == [(b"a", b"b"), (b"host", b"example.com")] # Multiple host is bad too with pytest.raises(LocalProtocolError): req = Request( method="GET", target="/", headers=[("Host", "a"), ("Host", "a")], http_version="1.1", ) # Even for HTTP/1.0 with pytest.raises(LocalProtocolError): req = Request( method="GET", target="/", headers=[("Host", "a"), ("Host", "a")], http_version="1.0", ) # Header values are validated for bad_char in "\x00\r\n\f\v": with pytest.raises(LocalProtocolError): req = Request( method="GET", target="/", headers=[("Host", "a"), ("Foo", "asd" + bad_char)], http_version="1.0", ) # But for compatibility we allow non-whitespace control characters, even # though they're forbidden by the spec. Request( method="GET", target="/", headers=[("Host", "a"), ("Foo", "asd\x01\x02\x7f")], http_version="1.0", ) # Request target is validated for bad_byte in b"\x00\x20\x7f\xee": target = bytearray(b"/") target.append(bad_byte) with pytest.raises(LocalProtocolError): Request( method="GET", target=target, headers=[("Host", "a")], http_version="1.1" ) # Request method is validated with pytest.raises(LocalProtocolError): Request( method="GET / HTTP/1.1", target=target, headers=[("Host", "a")], http_version="1.1", ) ir = InformationalResponse(status_code=100, headers=[("Host", "a")]) assert ir.status_code == 100 assert ir.headers == [(b"host", b"a")] assert ir.http_version == b"1.1" with pytest.raises(LocalProtocolError): InformationalResponse(status_code=200, headers=[("Host", "a")]) resp = Response(status_code=204, headers=[], http_version="1.0") # type: ignore[arg-type] assert resp.status_code == 204 assert resp.headers == [] assert resp.http_version == b"1.0" with pytest.raises(LocalProtocolError): resp = Response(status_code=100, headers=[], http_version="1.0") # type: ignore[arg-type] with pytest.raises(LocalProtocolError): Response(status_code="100", headers=[], http_version="1.0") # type: ignore[arg-type] with pytest.raises(LocalProtocolError): InformationalResponse(status_code=b"100", headers=[], http_version="1.0") # type: ignore[arg-type] d = Data(data=b"asdf") assert d.data == b"asdf" eom = EndOfMessage() assert eom.headers == [] cc = ConnectionClosed() assert repr(cc) == "ConnectionClosed()" def test_intenum_status_code() -> None: # https://github.com/python-hyper/h11/issues/72 r = Response(status_code=HTTPStatus.OK, headers=[], http_version="1.0") # type: ignore[arg-type] assert r.status_code == HTTPStatus.OK assert type(r.status_code) is not type(HTTPStatus.OK) assert type(r.status_code) is int def test_header_casing() -> None: r = Request( method="GET", target="/", headers=[("Host", "example.org"), ("Connection", "keep-alive")], http_version="1.1", ) assert len(r.headers) == 2 assert r.headers[0] == (b"host", b"example.org") assert r.headers == [(b"host", b"example.org"), (b"connection", b"keep-alive")] assert r.headers.raw_items() == [ (b"Host", b"example.org"), (b"Connection", b"keep-alive"), ] h11-0.13.0/h11/tests/test_headers.py000066400000000000000000000127541417207257000167610ustar00rootroot00000000000000import pytest from .._events import Request from .._headers import ( get_comma_header, has_expect_100_continue, Headers, normalize_and_validate, set_comma_header, ) from .._util import LocalProtocolError def test_normalize_and_validate() -> None: assert normalize_and_validate([("foo", "bar")]) == [(b"foo", b"bar")] assert normalize_and_validate([(b"foo", b"bar")]) == [(b"foo", b"bar")] # no leading/trailing whitespace in names with pytest.raises(LocalProtocolError): normalize_and_validate([(b"foo ", "bar")]) with pytest.raises(LocalProtocolError): normalize_and_validate([(b" foo", "bar")]) # no weird characters in names with pytest.raises(LocalProtocolError) as excinfo: normalize_and_validate([(b"foo bar", b"baz")]) assert "foo bar" in str(excinfo.value) with pytest.raises(LocalProtocolError): normalize_and_validate([(b"foo\x00bar", b"baz")]) # Not even 8-bit characters: with pytest.raises(LocalProtocolError): normalize_and_validate([(b"foo\xffbar", b"baz")]) # And not even the control characters we allow in values: with pytest.raises(LocalProtocolError): normalize_and_validate([(b"foo\x01bar", b"baz")]) # no return or NUL characters in values with pytest.raises(LocalProtocolError) as excinfo: normalize_and_validate([("foo", "bar\rbaz")]) assert "bar\\rbaz" in str(excinfo.value) with pytest.raises(LocalProtocolError): normalize_and_validate([("foo", "bar\nbaz")]) with pytest.raises(LocalProtocolError): normalize_and_validate([("foo", "bar\x00baz")]) # no leading/trailing whitespace with pytest.raises(LocalProtocolError): normalize_and_validate([("foo", "barbaz ")]) with pytest.raises(LocalProtocolError): normalize_and_validate([("foo", " barbaz")]) with pytest.raises(LocalProtocolError): normalize_and_validate([("foo", "barbaz\t")]) with pytest.raises(LocalProtocolError): normalize_and_validate([("foo", "\tbarbaz")]) # content-length assert normalize_and_validate([("Content-Length", "1")]) == [ (b"content-length", b"1") ] with pytest.raises(LocalProtocolError): normalize_and_validate([("Content-Length", "asdf")]) with pytest.raises(LocalProtocolError): normalize_and_validate([("Content-Length", "1x")]) with pytest.raises(LocalProtocolError): normalize_and_validate([("Content-Length", "1"), ("Content-Length", "2")]) assert normalize_and_validate( [("Content-Length", "0"), ("Content-Length", "0")] ) == [(b"content-length", b"0")] assert normalize_and_validate([("Content-Length", "0 , 0")]) == [ (b"content-length", b"0") ] with pytest.raises(LocalProtocolError): normalize_and_validate( [("Content-Length", "1"), ("Content-Length", "1"), ("Content-Length", "2")] ) with pytest.raises(LocalProtocolError): normalize_and_validate([("Content-Length", "1 , 1,2")]) # transfer-encoding assert normalize_and_validate([("Transfer-Encoding", "chunked")]) == [ (b"transfer-encoding", b"chunked") ] assert normalize_and_validate([("Transfer-Encoding", "cHuNkEd")]) == [ (b"transfer-encoding", b"chunked") ] with pytest.raises(LocalProtocolError) as excinfo: normalize_and_validate([("Transfer-Encoding", "gzip")]) assert excinfo.value.error_status_hint == 501 # Not Implemented with pytest.raises(LocalProtocolError) as excinfo: normalize_and_validate( [("Transfer-Encoding", "chunked"), ("Transfer-Encoding", "gzip")] ) assert excinfo.value.error_status_hint == 501 # Not Implemented def test_get_set_comma_header() -> None: headers = normalize_and_validate( [ ("Connection", "close"), ("whatever", "something"), ("connectiON", "fOo,, , BAR"), ] ) assert get_comma_header(headers, b"connection") == [b"close", b"foo", b"bar"] headers = set_comma_header(headers, b"newthing", ["a", "b"]) # type: ignore with pytest.raises(LocalProtocolError): set_comma_header(headers, b"newthing", [" a", "b"]) # type: ignore assert headers == [ (b"connection", b"close"), (b"whatever", b"something"), (b"connection", b"fOo,, , BAR"), (b"newthing", b"a"), (b"newthing", b"b"), ] headers = set_comma_header(headers, b"whatever", ["different thing"]) # type: ignore assert headers == [ (b"connection", b"close"), (b"connection", b"fOo,, , BAR"), (b"newthing", b"a"), (b"newthing", b"b"), (b"whatever", b"different thing"), ] def test_has_100_continue() -> None: assert has_expect_100_continue( Request( method="GET", target="/", headers=[("Host", "example.com"), ("Expect", "100-continue")], ) ) assert not has_expect_100_continue( Request(method="GET", target="/", headers=[("Host", "example.com")]) ) # Case insensitive assert has_expect_100_continue( Request( method="GET", target="/", headers=[("Host", "example.com"), ("Expect", "100-Continue")], ) ) # Doesn't work in HTTP/1.0 assert not has_expect_100_continue( Request( method="GET", target="/", headers=[("Host", "example.com"), ("Expect", "100-continue")], http_version="1.0", ) ) h11-0.13.0/h11/tests/test_helpers.py000066400000000000000000000014321417207257000167770ustar00rootroot00000000000000from .._events import ( ConnectionClosed, Data, EndOfMessage, Event, InformationalResponse, Request, Response, ) from .helpers import normalize_data_events def test_normalize_data_events() -> None: assert normalize_data_events( [ Data(data=bytearray(b"1")), Data(data=b"2"), Response(status_code=200, headers=[]), # type: ignore[arg-type] Data(data=b"3"), Data(data=b"4"), EndOfMessage(), Data(data=b"5"), Data(data=b"6"), Data(data=b"7"), ] ) == [ Data(data=b"12"), Response(status_code=200, headers=[]), # type: ignore[arg-type] Data(data=b"34"), EndOfMessage(), Data(data=b"567"), ] h11-0.13.0/h11/tests/test_io.py000066400000000000000000000375631417207257000157620ustar00rootroot00000000000000from typing import Any, Callable, Generator, List import pytest from .._events import ( ConnectionClosed, Data, EndOfMessage, Event, InformationalResponse, Request, Response, ) from .._headers import Headers, normalize_and_validate from .._readers import ( _obsolete_line_fold, ChunkedReader, ContentLengthReader, Http10Reader, READERS, ) from .._receivebuffer import ReceiveBuffer from .._state import ( CLIENT, CLOSED, DONE, IDLE, MIGHT_SWITCH_PROTOCOL, MUST_CLOSE, SEND_BODY, SEND_RESPONSE, SERVER, SWITCHED_PROTOCOL, ) from .._util import LocalProtocolError from .._writers import ( ChunkedWriter, ContentLengthWriter, Http10Writer, write_any_response, write_headers, write_request, WRITERS, ) from .helpers import normalize_data_events SIMPLE_CASES = [ ( (CLIENT, IDLE), Request( method="GET", target="/a", headers=[("Host", "foo"), ("Connection", "close")], ), b"GET /a HTTP/1.1\r\nHost: foo\r\nConnection: close\r\n\r\n", ), ( (SERVER, SEND_RESPONSE), Response(status_code=200, headers=[("Connection", "close")], reason=b"OK"), b"HTTP/1.1 200 OK\r\nConnection: close\r\n\r\n", ), ( (SERVER, SEND_RESPONSE), Response(status_code=200, headers=[], reason=b"OK"), # type: ignore[arg-type] b"HTTP/1.1 200 OK\r\n\r\n", ), ( (SERVER, SEND_RESPONSE), InformationalResponse( status_code=101, headers=[("Upgrade", "websocket")], reason=b"Upgrade" ), b"HTTP/1.1 101 Upgrade\r\nUpgrade: websocket\r\n\r\n", ), ( (SERVER, SEND_RESPONSE), InformationalResponse(status_code=101, headers=[], reason=b"Upgrade"), # type: ignore[arg-type] b"HTTP/1.1 101 Upgrade\r\n\r\n", ), ] def dowrite(writer: Callable[..., None], obj: Any) -> bytes: got_list: List[bytes] = [] writer(obj, got_list.append) return b"".join(got_list) def tw(writer: Any, obj: Any, expected: Any) -> None: got = dowrite(writer, obj) assert got == expected def makebuf(data: bytes) -> ReceiveBuffer: buf = ReceiveBuffer() buf += data return buf def tr(reader: Any, data: bytes, expected: Any) -> None: def check(got: Any) -> None: assert got == expected # Headers should always be returned as bytes, not e.g. bytearray # https://github.com/python-hyper/wsproto/pull/54#issuecomment-377709478 for name, value in getattr(got, "headers", []): assert type(name) is bytes assert type(value) is bytes # Simple: consume whole thing buf = makebuf(data) check(reader(buf)) assert not buf # Incrementally growing buffer buf = ReceiveBuffer() for i in range(len(data)): assert reader(buf) is None buf += data[i : i + 1] check(reader(buf)) # Trailing data buf = makebuf(data) buf += b"trailing" check(reader(buf)) assert bytes(buf) == b"trailing" def test_writers_simple() -> None: for ((role, state), event, binary) in SIMPLE_CASES: tw(WRITERS[role, state], event, binary) def test_readers_simple() -> None: for ((role, state), event, binary) in SIMPLE_CASES: tr(READERS[role, state], binary, event) def test_writers_unusual() -> None: # Simple test of the write_headers utility routine tw( write_headers, normalize_and_validate([("foo", "bar"), ("baz", "quux")]), b"foo: bar\r\nbaz: quux\r\n\r\n", ) tw(write_headers, Headers([]), b"\r\n") # We understand HTTP/1.0, but we don't speak it with pytest.raises(LocalProtocolError): tw( write_request, Request( method="GET", target="/", headers=[("Host", "foo"), ("Connection", "close")], http_version="1.0", ), None, ) with pytest.raises(LocalProtocolError): tw( write_any_response, Response( status_code=200, headers=[("Connection", "close")], http_version="1.0" ), None, ) def test_readers_unusual() -> None: # Reading HTTP/1.0 tr( READERS[CLIENT, IDLE], b"HEAD /foo HTTP/1.0\r\nSome: header\r\n\r\n", Request( method="HEAD", target="/foo", headers=[("Some", "header")], http_version="1.0", ), ) # check no-headers, since it's only legal with HTTP/1.0 tr( READERS[CLIENT, IDLE], b"HEAD /foo HTTP/1.0\r\n\r\n", Request(method="HEAD", target="/foo", headers=[], http_version="1.0"), # type: ignore[arg-type] ) tr( READERS[SERVER, SEND_RESPONSE], b"HTTP/1.0 200 OK\r\nSome: header\r\n\r\n", Response( status_code=200, headers=[("Some", "header")], http_version="1.0", reason=b"OK", ), ) # single-character header values (actually disallowed by the ABNF in RFC # 7230 -- this is a bug in the standard that we originally copied...) tr( READERS[SERVER, SEND_RESPONSE], b"HTTP/1.0 200 OK\r\n" b"Foo: a a a a a \r\n\r\n", Response( status_code=200, headers=[("Foo", "a a a a a")], http_version="1.0", reason=b"OK", ), ) # Empty headers -- also legal tr( READERS[SERVER, SEND_RESPONSE], b"HTTP/1.0 200 OK\r\n" b"Foo:\r\n\r\n", Response( status_code=200, headers=[("Foo", "")], http_version="1.0", reason=b"OK" ), ) tr( READERS[SERVER, SEND_RESPONSE], b"HTTP/1.0 200 OK\r\n" b"Foo: \t \t \r\n\r\n", Response( status_code=200, headers=[("Foo", "")], http_version="1.0", reason=b"OK" ), ) # Tolerate broken servers that leave off the response code tr( READERS[SERVER, SEND_RESPONSE], b"HTTP/1.0 200\r\n" b"Foo: bar\r\n\r\n", Response( status_code=200, headers=[("Foo", "bar")], http_version="1.0", reason=b"" ), ) # Tolerate headers line endings (\r\n and \n) # \n\r\b between headers and body tr( READERS[SERVER, SEND_RESPONSE], b"HTTP/1.1 200 OK\r\nSomeHeader: val\n\r\n", Response( status_code=200, headers=[("SomeHeader", "val")], http_version="1.1", reason="OK", ), ) # delimited only with \n tr( READERS[SERVER, SEND_RESPONSE], b"HTTP/1.1 200 OK\nSomeHeader1: val1\nSomeHeader2: val2\n\n", Response( status_code=200, headers=[("SomeHeader1", "val1"), ("SomeHeader2", "val2")], http_version="1.1", reason="OK", ), ) # mixed \r\n and \n tr( READERS[SERVER, SEND_RESPONSE], b"HTTP/1.1 200 OK\r\nSomeHeader1: val1\nSomeHeader2: val2\n\r\n", Response( status_code=200, headers=[("SomeHeader1", "val1"), ("SomeHeader2", "val2")], http_version="1.1", reason="OK", ), ) # obsolete line folding tr( READERS[CLIENT, IDLE], b"HEAD /foo HTTP/1.1\r\n" b"Host: example.com\r\n" b"Some: multi-line\r\n" b" header\r\n" b"\tnonsense\r\n" b" \t \t\tI guess\r\n" b"Connection: close\r\n" b"More-nonsense: in the\r\n" b" last header \r\n\r\n", Request( method="HEAD", target="/foo", headers=[ ("Host", "example.com"), ("Some", "multi-line header nonsense I guess"), ("Connection", "close"), ("More-nonsense", "in the last header"), ], ), ) with pytest.raises(LocalProtocolError): tr( READERS[CLIENT, IDLE], b"HEAD /foo HTTP/1.1\r\n" b" folded: line\r\n\r\n", None, ) with pytest.raises(LocalProtocolError): tr( READERS[CLIENT, IDLE], b"HEAD /foo HTTP/1.1\r\n" b"foo : line\r\n\r\n", None, ) with pytest.raises(LocalProtocolError): tr( READERS[CLIENT, IDLE], b"HEAD /foo HTTP/1.1\r\n" b"foo\t: line\r\n\r\n", None, ) with pytest.raises(LocalProtocolError): tr( READERS[CLIENT, IDLE], b"HEAD /foo HTTP/1.1\r\n" b"foo\t: line\r\n\r\n", None, ) with pytest.raises(LocalProtocolError): tr(READERS[CLIENT, IDLE], b"HEAD /foo HTTP/1.1\r\n" b": line\r\n\r\n", None) def test__obsolete_line_fold_bytes() -> None: # _obsolete_line_fold has a defensive cast to bytearray, which is # necessary to protect against O(n^2) behavior in case anyone ever passes # in regular bytestrings... but right now we never pass in regular # bytestrings. so this test just exists to get some coverage on that # defensive cast. assert list(_obsolete_line_fold([b"aaa", b"bbb", b" ccc", b"ddd"])) == [ b"aaa", bytearray(b"bbb ccc"), b"ddd", ] def _run_reader_iter( reader: Any, buf: bytes, do_eof: bool ) -> Generator[Any, None, None]: while True: event = reader(buf) if event is None: break yield event # body readers have undefined behavior after returning EndOfMessage, # because this changes the state so they don't get called again if type(event) is EndOfMessage: break if do_eof: assert not buf yield reader.read_eof() def _run_reader(*args: Any) -> List[Event]: events = list(_run_reader_iter(*args)) return normalize_data_events(events) def t_body_reader(thunk: Any, data: bytes, expected: Any, do_eof: bool = False) -> None: # Simple: consume whole thing print("Test 1") buf = makebuf(data) assert _run_reader(thunk(), buf, do_eof) == expected # Incrementally growing buffer print("Test 2") reader = thunk() buf = ReceiveBuffer() events = [] for i in range(len(data)): events += _run_reader(reader, buf, False) buf += data[i : i + 1] events += _run_reader(reader, buf, do_eof) assert normalize_data_events(events) == expected is_complete = any(type(event) is EndOfMessage for event in expected) if is_complete and not do_eof: buf = makebuf(data + b"trailing") assert _run_reader(thunk(), buf, False) == expected def test_ContentLengthReader() -> None: t_body_reader(lambda: ContentLengthReader(0), b"", [EndOfMessage()]) t_body_reader( lambda: ContentLengthReader(10), b"0123456789", [Data(data=b"0123456789"), EndOfMessage()], ) def test_Http10Reader() -> None: t_body_reader(Http10Reader, b"", [EndOfMessage()], do_eof=True) t_body_reader(Http10Reader, b"asdf", [Data(data=b"asdf")], do_eof=False) t_body_reader( Http10Reader, b"asdf", [Data(data=b"asdf"), EndOfMessage()], do_eof=True ) def test_ChunkedReader() -> None: t_body_reader(ChunkedReader, b"0\r\n\r\n", [EndOfMessage()]) t_body_reader( ChunkedReader, b"0\r\nSome: header\r\n\r\n", [EndOfMessage(headers=[("Some", "header")])], ) t_body_reader( ChunkedReader, b"5\r\n01234\r\n" + b"10\r\n0123456789abcdef\r\n" + b"0\r\n" + b"Some: header\r\n\r\n", [ Data(data=b"012340123456789abcdef"), EndOfMessage(headers=[("Some", "header")]), ], ) t_body_reader( ChunkedReader, b"5\r\n01234\r\n" + b"10\r\n0123456789abcdef\r\n" + b"0\r\n\r\n", [Data(data=b"012340123456789abcdef"), EndOfMessage()], ) # handles upper and lowercase hex t_body_reader( ChunkedReader, b"aA\r\n" + b"x" * 0xAA + b"\r\n" + b"0\r\n\r\n", [Data(data=b"x" * 0xAA), EndOfMessage()], ) # refuses arbitrarily long chunk integers with pytest.raises(LocalProtocolError): # Technically this is legal HTTP/1.1, but we refuse to process chunk # sizes that don't fit into 20 characters of hex t_body_reader(ChunkedReader, b"9" * 100 + b"\r\nxxx", [Data(data=b"xxx")]) # refuses garbage in the chunk count with pytest.raises(LocalProtocolError): t_body_reader(ChunkedReader, b"10\x00\r\nxxx", None) # handles (and discards) "chunk extensions" omg wtf t_body_reader( ChunkedReader, b"5; hello=there\r\n" + b"xxxxx" + b"\r\n" + b'0; random="junk"; some=more; canbe=lonnnnngg\r\n\r\n', [Data(data=b"xxxxx"), EndOfMessage()], ) def test_ContentLengthWriter() -> None: w = ContentLengthWriter(5) assert dowrite(w, Data(data=b"123")) == b"123" assert dowrite(w, Data(data=b"45")) == b"45" assert dowrite(w, EndOfMessage()) == b"" w = ContentLengthWriter(5) with pytest.raises(LocalProtocolError): dowrite(w, Data(data=b"123456")) w = ContentLengthWriter(5) dowrite(w, Data(data=b"123")) with pytest.raises(LocalProtocolError): dowrite(w, Data(data=b"456")) w = ContentLengthWriter(5) dowrite(w, Data(data=b"123")) with pytest.raises(LocalProtocolError): dowrite(w, EndOfMessage()) w = ContentLengthWriter(5) dowrite(w, Data(data=b"123")) == b"123" dowrite(w, Data(data=b"45")) == b"45" with pytest.raises(LocalProtocolError): dowrite(w, EndOfMessage(headers=[("Etag", "asdf")])) def test_ChunkedWriter() -> None: w = ChunkedWriter() assert dowrite(w, Data(data=b"aaa")) == b"3\r\naaa\r\n" assert dowrite(w, Data(data=b"a" * 20)) == b"14\r\n" + b"a" * 20 + b"\r\n" assert dowrite(w, Data(data=b"")) == b"" assert dowrite(w, EndOfMessage()) == b"0\r\n\r\n" assert ( dowrite(w, EndOfMessage(headers=[("Etag", "asdf"), ("a", "b")])) == b"0\r\nEtag: asdf\r\na: b\r\n\r\n" ) def test_Http10Writer() -> None: w = Http10Writer() assert dowrite(w, Data(data=b"1234")) == b"1234" assert dowrite(w, EndOfMessage()) == b"" with pytest.raises(LocalProtocolError): dowrite(w, EndOfMessage(headers=[("Etag", "asdf")])) def test_reject_garbage_after_request_line() -> None: with pytest.raises(LocalProtocolError): tr(READERS[SERVER, SEND_RESPONSE], b"HTTP/1.0 200 OK\x00xxxx\r\n\r\n", None) def test_reject_garbage_after_response_line() -> None: with pytest.raises(LocalProtocolError): tr( READERS[CLIENT, IDLE], b"HEAD /foo HTTP/1.1 xxxxxx\r\n" b"Host: a\r\n\r\n", None, ) def test_reject_garbage_in_header_line() -> None: with pytest.raises(LocalProtocolError): tr( READERS[CLIENT, IDLE], b"HEAD /foo HTTP/1.1\r\n" b"Host: foo\x00bar\r\n\r\n", None, ) def test_reject_non_vchar_in_path() -> None: for bad_char in b"\x00\x20\x7f\xee": message = bytearray(b"HEAD /") message.append(bad_char) message.extend(b" HTTP/1.1\r\nHost: foobar\r\n\r\n") with pytest.raises(LocalProtocolError): tr(READERS[CLIENT, IDLE], message, None) # https://github.com/python-hyper/h11/issues/57 def test_allow_some_garbage_in_cookies() -> None: tr( READERS[CLIENT, IDLE], b"HEAD /foo HTTP/1.1\r\n" b"Host: foo\r\n" b"Set-Cookie: ___utmvafIumyLc=kUd\x01UpAt; path=/; Max-Age=900\r\n" b"\r\n", Request( method="HEAD", target="/foo", headers=[ ("Host", "foo"), ("Set-Cookie", "___utmvafIumyLc=kUd\x01UpAt; path=/; Max-Age=900"), ], ), ) def test_host_comes_first() -> None: tw( write_headers, normalize_and_validate([("foo", "bar"), ("Host", "example.com")]), b"Host: example.com\r\nfoo: bar\r\n\r\n", ) h11-0.13.0/h11/tests/test_receivebuffer.py000066400000000000000000000065761417207257000201670ustar00rootroot00000000000000import re from typing import Tuple import pytest from .._receivebuffer import ReceiveBuffer def test_receivebuffer() -> None: b = ReceiveBuffer() assert not b assert len(b) == 0 assert bytes(b) == b"" b += b"123" assert b assert len(b) == 3 assert bytes(b) == b"123" assert bytes(b) == b"123" assert b.maybe_extract_at_most(2) == b"12" assert b assert len(b) == 1 assert bytes(b) == b"3" assert bytes(b) == b"3" assert b.maybe_extract_at_most(10) == b"3" assert bytes(b) == b"" assert b.maybe_extract_at_most(10) is None assert not b ################################################################ # maybe_extract_until_next ################################################################ b += b"123\n456\r\n789\r\n" assert b.maybe_extract_next_line() == b"123\n456\r\n" assert bytes(b) == b"789\r\n" assert b.maybe_extract_next_line() == b"789\r\n" assert bytes(b) == b"" b += b"12\r" assert b.maybe_extract_next_line() is None assert bytes(b) == b"12\r" b += b"345\n\r" assert b.maybe_extract_next_line() is None assert bytes(b) == b"12\r345\n\r" # here we stopped at the middle of b"\r\n" delimiter b += b"\n6789aaa123\r\n" assert b.maybe_extract_next_line() == b"12\r345\n\r\n" assert b.maybe_extract_next_line() == b"6789aaa123\r\n" assert b.maybe_extract_next_line() is None assert bytes(b) == b"" ################################################################ # maybe_extract_lines ################################################################ b += b"123\r\na: b\r\nfoo:bar\r\n\r\ntrailing" lines = b.maybe_extract_lines() assert lines == [b"123", b"a: b", b"foo:bar"] assert bytes(b) == b"trailing" assert b.maybe_extract_lines() is None b += b"\r\n\r" assert b.maybe_extract_lines() is None assert b.maybe_extract_at_most(100) == b"trailing\r\n\r" assert not b # Empty body case (as happens at the end of chunked encoding if there are # no trailing headers, e.g.) b += b"\r\ntrailing" assert b.maybe_extract_lines() == [] assert bytes(b) == b"trailing" @pytest.mark.parametrize( "data", [ pytest.param( ( b"HTTP/1.1 200 OK\r\n", b"Content-type: text/plain\r\n", b"Connection: close\r\n", b"\r\n", b"Some body", ), id="with_crlf_delimiter", ), pytest.param( ( b"HTTP/1.1 200 OK\n", b"Content-type: text/plain\n", b"Connection: close\n", b"\n", b"Some body", ), id="with_lf_only_delimiter", ), pytest.param( ( b"HTTP/1.1 200 OK\n", b"Content-type: text/plain\r\n", b"Connection: close\n", b"\n", b"Some body", ), id="with_mixed_crlf_and_lf", ), ], ) def test_receivebuffer_for_invalid_delimiter(data: Tuple[bytes]) -> None: b = ReceiveBuffer() for line in data: b += line lines = b.maybe_extract_lines() assert lines == [ b"HTTP/1.1 200 OK", b"Content-type: text/plain", b"Connection: close", ] assert bytes(b) == b"Some body" h11-0.13.0/h11/tests/test_state.py000066400000000000000000000213401417207257000164550ustar00rootroot00000000000000import pytest from .._events import ( ConnectionClosed, Data, EndOfMessage, Event, InformationalResponse, Request, Response, ) from .._state import ( _SWITCH_CONNECT, _SWITCH_UPGRADE, CLIENT, CLOSED, ConnectionState, DONE, IDLE, MIGHT_SWITCH_PROTOCOL, MUST_CLOSE, SEND_BODY, SEND_RESPONSE, SERVER, SWITCHED_PROTOCOL, ) from .._util import LocalProtocolError def test_ConnectionState() -> None: cs = ConnectionState() # Basic event-triggered transitions assert cs.states == {CLIENT: IDLE, SERVER: IDLE} cs.process_event(CLIENT, Request) # The SERVER-Request special case: assert cs.states == {CLIENT: SEND_BODY, SERVER: SEND_RESPONSE} # Illegal transitions raise an error and nothing happens with pytest.raises(LocalProtocolError): cs.process_event(CLIENT, Request) assert cs.states == {CLIENT: SEND_BODY, SERVER: SEND_RESPONSE} cs.process_event(SERVER, InformationalResponse) assert cs.states == {CLIENT: SEND_BODY, SERVER: SEND_RESPONSE} cs.process_event(SERVER, Response) assert cs.states == {CLIENT: SEND_BODY, SERVER: SEND_BODY} cs.process_event(CLIENT, EndOfMessage) cs.process_event(SERVER, EndOfMessage) assert cs.states == {CLIENT: DONE, SERVER: DONE} # State-triggered transition cs.process_event(SERVER, ConnectionClosed) assert cs.states == {CLIENT: MUST_CLOSE, SERVER: CLOSED} def test_ConnectionState_keep_alive() -> None: # keep_alive = False cs = ConnectionState() cs.process_event(CLIENT, Request) cs.process_keep_alive_disabled() cs.process_event(CLIENT, EndOfMessage) assert cs.states == {CLIENT: MUST_CLOSE, SERVER: SEND_RESPONSE} cs.process_event(SERVER, Response) cs.process_event(SERVER, EndOfMessage) assert cs.states == {CLIENT: MUST_CLOSE, SERVER: MUST_CLOSE} def test_ConnectionState_keep_alive_in_DONE() -> None: # Check that if keep_alive is disabled when the CLIENT is already in DONE, # then this is sufficient to immediately trigger the DONE -> MUST_CLOSE # transition cs = ConnectionState() cs.process_event(CLIENT, Request) cs.process_event(CLIENT, EndOfMessage) assert cs.states[CLIENT] is DONE cs.process_keep_alive_disabled() assert cs.states[CLIENT] is MUST_CLOSE def test_ConnectionState_switch_denied() -> None: for switch_type in (_SWITCH_CONNECT, _SWITCH_UPGRADE): for deny_early in (True, False): cs = ConnectionState() cs.process_client_switch_proposal(switch_type) cs.process_event(CLIENT, Request) cs.process_event(CLIENT, Data) assert cs.states == {CLIENT: SEND_BODY, SERVER: SEND_RESPONSE} assert switch_type in cs.pending_switch_proposals if deny_early: # before client reaches DONE cs.process_event(SERVER, Response) assert not cs.pending_switch_proposals cs.process_event(CLIENT, EndOfMessage) if deny_early: assert cs.states == {CLIENT: DONE, SERVER: SEND_BODY} else: assert cs.states == { CLIENT: MIGHT_SWITCH_PROTOCOL, SERVER: SEND_RESPONSE, } cs.process_event(SERVER, InformationalResponse) assert cs.states == { CLIENT: MIGHT_SWITCH_PROTOCOL, SERVER: SEND_RESPONSE, } cs.process_event(SERVER, Response) assert cs.states == {CLIENT: DONE, SERVER: SEND_BODY} assert not cs.pending_switch_proposals _response_type_for_switch = { _SWITCH_UPGRADE: InformationalResponse, _SWITCH_CONNECT: Response, None: Response, } def test_ConnectionState_protocol_switch_accepted() -> None: for switch_event in [_SWITCH_UPGRADE, _SWITCH_CONNECT]: cs = ConnectionState() cs.process_client_switch_proposal(switch_event) cs.process_event(CLIENT, Request) cs.process_event(CLIENT, Data) assert cs.states == {CLIENT: SEND_BODY, SERVER: SEND_RESPONSE} cs.process_event(CLIENT, EndOfMessage) assert cs.states == {CLIENT: MIGHT_SWITCH_PROTOCOL, SERVER: SEND_RESPONSE} cs.process_event(SERVER, InformationalResponse) assert cs.states == {CLIENT: MIGHT_SWITCH_PROTOCOL, SERVER: SEND_RESPONSE} cs.process_event(SERVER, _response_type_for_switch[switch_event], switch_event) assert cs.states == {CLIENT: SWITCHED_PROTOCOL, SERVER: SWITCHED_PROTOCOL} def test_ConnectionState_double_protocol_switch() -> None: # CONNECT + Upgrade is legal! Very silly, but legal. So we support # it. Because sometimes doing the silly thing is easier than not. for server_switch in [None, _SWITCH_UPGRADE, _SWITCH_CONNECT]: cs = ConnectionState() cs.process_client_switch_proposal(_SWITCH_UPGRADE) cs.process_client_switch_proposal(_SWITCH_CONNECT) cs.process_event(CLIENT, Request) cs.process_event(CLIENT, EndOfMessage) assert cs.states == {CLIENT: MIGHT_SWITCH_PROTOCOL, SERVER: SEND_RESPONSE} cs.process_event( SERVER, _response_type_for_switch[server_switch], server_switch ) if server_switch is None: assert cs.states == {CLIENT: DONE, SERVER: SEND_BODY} else: assert cs.states == {CLIENT: SWITCHED_PROTOCOL, SERVER: SWITCHED_PROTOCOL} def test_ConnectionState_inconsistent_protocol_switch() -> None: for client_switches, server_switch in [ ([], _SWITCH_CONNECT), ([], _SWITCH_UPGRADE), ([_SWITCH_UPGRADE], _SWITCH_CONNECT), ([_SWITCH_CONNECT], _SWITCH_UPGRADE), ]: cs = ConnectionState() for client_switch in client_switches: # type: ignore[attr-defined] cs.process_client_switch_proposal(client_switch) cs.process_event(CLIENT, Request) with pytest.raises(LocalProtocolError): cs.process_event(SERVER, Response, server_switch) def test_ConnectionState_keepalive_protocol_switch_interaction() -> None: # keep_alive=False + pending_switch_proposals cs = ConnectionState() cs.process_client_switch_proposal(_SWITCH_UPGRADE) cs.process_event(CLIENT, Request) cs.process_keep_alive_disabled() cs.process_event(CLIENT, Data) assert cs.states == {CLIENT: SEND_BODY, SERVER: SEND_RESPONSE} # the protocol switch "wins" cs.process_event(CLIENT, EndOfMessage) assert cs.states == {CLIENT: MIGHT_SWITCH_PROTOCOL, SERVER: SEND_RESPONSE} # but when the server denies the request, keep_alive comes back into play cs.process_event(SERVER, Response) assert cs.states == {CLIENT: MUST_CLOSE, SERVER: SEND_BODY} def test_ConnectionState_reuse() -> None: cs = ConnectionState() with pytest.raises(LocalProtocolError): cs.start_next_cycle() cs.process_event(CLIENT, Request) cs.process_event(CLIENT, EndOfMessage) with pytest.raises(LocalProtocolError): cs.start_next_cycle() cs.process_event(SERVER, Response) cs.process_event(SERVER, EndOfMessage) cs.start_next_cycle() assert cs.states == {CLIENT: IDLE, SERVER: IDLE} # No keepalive cs.process_event(CLIENT, Request) cs.process_keep_alive_disabled() cs.process_event(CLIENT, EndOfMessage) cs.process_event(SERVER, Response) cs.process_event(SERVER, EndOfMessage) with pytest.raises(LocalProtocolError): cs.start_next_cycle() # One side closed cs = ConnectionState() cs.process_event(CLIENT, Request) cs.process_event(CLIENT, EndOfMessage) cs.process_event(CLIENT, ConnectionClosed) cs.process_event(SERVER, Response) cs.process_event(SERVER, EndOfMessage) with pytest.raises(LocalProtocolError): cs.start_next_cycle() # Succesful protocol switch cs = ConnectionState() cs.process_client_switch_proposal(_SWITCH_UPGRADE) cs.process_event(CLIENT, Request) cs.process_event(CLIENT, EndOfMessage) cs.process_event(SERVER, InformationalResponse, _SWITCH_UPGRADE) with pytest.raises(LocalProtocolError): cs.start_next_cycle() # Failed protocol switch cs = ConnectionState() cs.process_client_switch_proposal(_SWITCH_UPGRADE) cs.process_event(CLIENT, Request) cs.process_event(CLIENT, EndOfMessage) cs.process_event(SERVER, Response) cs.process_event(SERVER, EndOfMessage) cs.start_next_cycle() assert cs.states == {CLIENT: IDLE, SERVER: IDLE} def test_server_request_is_illegal() -> None: # There used to be a bug in how we handled the Request special case that # made this allowed... cs = ConnectionState() with pytest.raises(LocalProtocolError): cs.process_event(SERVER, Request) h11-0.13.0/h11/tests/test_util.py000066400000000000000000000056321417207257000163200ustar00rootroot00000000000000import re import sys import traceback from typing import NoReturn import pytest from .._util import ( bytesify, LocalProtocolError, ProtocolError, RemoteProtocolError, Sentinel, validate, ) def test_ProtocolError() -> None: with pytest.raises(TypeError): ProtocolError("abstract base class") def test_LocalProtocolError() -> None: try: raise LocalProtocolError("foo") except LocalProtocolError as e: assert str(e) == "foo" assert e.error_status_hint == 400 try: raise LocalProtocolError("foo", error_status_hint=418) except LocalProtocolError as e: assert str(e) == "foo" assert e.error_status_hint == 418 def thunk() -> NoReturn: raise LocalProtocolError("a", error_status_hint=420) try: try: thunk() except LocalProtocolError as exc1: orig_traceback = "".join(traceback.format_tb(sys.exc_info()[2])) exc1._reraise_as_remote_protocol_error() except RemoteProtocolError as exc2: assert type(exc2) is RemoteProtocolError assert exc2.args == ("a",) assert exc2.error_status_hint == 420 new_traceback = "".join(traceback.format_tb(sys.exc_info()[2])) assert new_traceback.endswith(orig_traceback) def test_validate() -> None: my_re = re.compile(br"(?P[0-9]+)\.(?P[0-9]+)") with pytest.raises(LocalProtocolError): validate(my_re, b"0.") groups = validate(my_re, b"0.1") assert groups == {"group1": b"0", "group2": b"1"} # successful partial matches are an error - must match whole string with pytest.raises(LocalProtocolError): validate(my_re, b"0.1xx") with pytest.raises(LocalProtocolError): validate(my_re, b"0.1\n") def test_validate_formatting() -> None: my_re = re.compile(br"foo") with pytest.raises(LocalProtocolError) as excinfo: validate(my_re, b"", "oops") assert "oops" in str(excinfo.value) with pytest.raises(LocalProtocolError) as excinfo: validate(my_re, b"", "oops {}") assert "oops {}" in str(excinfo.value) with pytest.raises(LocalProtocolError) as excinfo: validate(my_re, b"", "oops {} xx", 10) assert "oops 10 xx" in str(excinfo.value) def test_make_sentinel() -> None: class S(Sentinel, metaclass=Sentinel): pass assert repr(S) == "S" assert S == S assert type(S).__name__ == "S" assert S in {S} assert type(S) is S class S2(Sentinel, metaclass=Sentinel): pass assert repr(S2) == "S2" assert S != S2 assert S not in {S2} assert type(S) is not type(S2) def test_bytesify() -> None: assert bytesify(b"123") == b"123" assert bytesify(bytearray(b"123")) == b"123" assert bytesify("123") == b"123" with pytest.raises(UnicodeEncodeError): bytesify("\u1234") with pytest.raises(TypeError): bytesify(10) h11-0.13.0/newsfragments/000077500000000000000000000000001417207257000150545ustar00rootroot00000000000000h11-0.13.0/newsfragments/.gitkeep000066400000000000000000000000001417207257000164730ustar00rootroot00000000000000h11-0.13.0/newsfragments/README.rst000066400000000000000000000020511417207257000165410ustar00rootroot00000000000000This directory collects "newsfragments": short files that each contain a snippet of ReST-formatted text that will be added to the next release notes. This should be a description of aspects of the change (if any) that are relevant to users. (This contrasts with your commit message and PR description, which are a description of the change as relevant to people working on the code itself.) Each file should be named like ``..rst``, where ```` is an issue numbers, and ```` is one of: * ``feature`` * ``bugfix`` * ``doc`` * ``removal`` * ``misc`` So for example: ``123.feature.rst``, ``456.bugfix.rst`` If your PR fixes an issue, use that number here. If there is no issue, then after you submit the PR and get the PR number you can add a newsfragment using that instead. Note that the ``towncrier`` tool will automatically reflow your text, so don't try to do any fancy formatting. You can install ``towncrier`` and then run ``towncrier --draft`` if you want to get a preview of how your change will look in the final release notes. h11-0.13.0/notes.org000066400000000000000000000164041417207257000140370ustar00rootroot00000000000000Possible API breaking changes: - pondering moving headers to be (default)dict of lowercase bytestrings -> ordered lists of bytestrings I guess we should get some benchmarks/profiles first, since one of the motivations would be to eliminate all these linear scans and reallocations we use when dealing with headers - orrrrr... join most headers on "," and join Set-Cookie on ";" (HTTP/2 spec explicitly allows this!), and then we can just use a freakin' (case insensitive) dict. Terrible idea? or awesome idea? - argh, no, HTTP/2 allows joining *Cookie:* on ";". Set-Cookie header syntax makes it impossible to join them in any way :-( - pondering whether to adopt the HTTP/2 style of sticking request/response line information directly into the header dict. Advantages: - code that wants to handle HTTP/2 will need to handle this anyway, might make it easier to write dual-stack clients/servers - provides a more useful downstream representation for request targets that are in full-fledged http://... form. I'm of mixed mind about how much these matter though -- HTTP/1.1 servers are supposedly required to support them, but HTTP/1.1 clients are forbidden to send them, and in practice the transition that the HTTP/1.1 spec envisions to clients sending these all the time is... just never going to happen. So I like following specs, but in reality servers never have and never will need to support these, making it feel a bit silly. They do get sent to proxies, though -- maybe someone wants to use h11 to implement a proxy? for better tests: https://github.com/kevin1024/pytest-httpbin http://pathod.net/ XX TODO: A server MUST NOT send a Transfer-Encoding header field in any response with a status code of 1xx (Informational) or 204 (No Content). A server MUST NOT send a Transfer-Encoding header field in any 2xx (Successful) response to a CONNECT request (Section 4.3.6 of [RFC7231]). A server MUST NOT send a Content-Length header field in any response with a status code of 1xx (Informational) or 204 (No Content). A server MUST NOT send a Content-Length header field in any 2xx (Successful) response to a CONNECT request (Section 4.3.6 of [RFC7231]). http://coad.measurement-factory.com/details.html * notes on URLs there are multiple not fully consistent specs [[https://tools.ietf.org/html/rfc3986][RFC 3986]] is the basic spec that RFC 7230 refers to RFC 3987 adds "internationalized" support RFC 6874 revises RFC 3986 a bit for "IPv6 zone support" -- golang has some code to handle this and then there's the [[https://url.spec.whatwg.org/][WHATWG URL spec]] some commentary on this: https://daniel.haxx.se/blog/2016/05/11/my-url-isnt-your-url/ note that curl has been forced to handle non-RFC 3986-compliant (but WHATWG URL-compliant) URLs in Location: headers -- specifically ones containing weird numbers of slashes, and ones containing spaces (!), and maybe UTF-8 and other such fun https://news.ycombinator.com/item?id=11673058 "I don't think cURL implements this percent encoding yet - instead, it sends out binary paths on UTF-8 locale and Linux likewise." -- https://news.ycombinator.com/item?id=11674778 also: https://github.com/bagder/docs/blob/master/URL-interop.md "This document is an attempt to describe where and how RFC 3986 (86), RFC 3987 (87) and the WHATWG URL Specification (TWUS) differ. This might be useful input when trying to interop with URLs on the modern Internet." ** looking at the go http parser spaces in HTTP/1.1 request-lines are definitely verboten -- e.g. here's the go http server code for splitting a request line (parseRequestLine), which assumes the second space represents the end of the target: https://golang.org/src/net/http/request.go#L680 OTOH if we scroll down to readRequest, we see that they have a special case where for CONNECT targets, they accept either host:port OR /path/with/slash (wtf): // CONNECT requests are used two different ways, and neither uses a full URL: // The standard use is to tunnel HTTPS through an HTTP proxy. // It looks like "CONNECT www.google.com:443 HTTP/1.1", and the parameter is // just the authority section of a URL. This information should go in req.URL.Host. // // The net/rpc package also uses CONNECT, but there the parameter is a path // that starts with a slash. It can be parsed with the regular URL parser, // and the path will end up in req.URL.Path, where it needs to be in order for // RPC to work. other interesting things: - they have a special removeZone function to handle [[https://tools.ietf.org/html/rfc6874][RFC 6874]], which revises RFC 3986 - they provide both a parsed URL and a raw string containing whatever was in the request line ** experiment to check how firefox handles UTF-8 in URLs: $ socat - TCP-LISTEN:12345 then browse to http://localhost:12345/✔ produces: GET /%E2%9C%94 HTTP/1.1 Host: localhost:12345 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:47.0) Gecko/20100101 Firefox/47.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-US,en;q=0.5 Accept-Encoding: gzip, deflate DNT: 1 Connection: keep-alive * notes for building something on top of this headers to consider auto-supporting at the high-level: - Date: https://svn.tools.ietf.org/svn/wg/httpbis/specs/rfc7231.html#header.date MUST be sent by origin servers who know what time it is (clients don't bother) - Server - automagic compression should let handlers control timeouts ################################################################ Higher level stuff: - Timeouts: waiting for 100-continue, killing idle keepalive connections, killing idle connections in general basically just need a timeout when we block on read, and if it times out then we close. should be settable in the APIs that block on read (e.g. iterating over body). - Expect: https://svn.tools.ietf.org/svn/wg/httpbis/specs/rfc7231.html#rfc.section.5.1.1 This is tightly integrated with flow control, not a lot we can do, except maybe provide a method to be called before blocking waiting for the request body? - Sending an error when things go wrong (esp. 400 Bad Request) Connection shutdown is tricky. Quoth RFC 7230: "If a server performs an immediate close of a TCP connection, there is a significant risk that the client will not be able to read the last HTTP response. If the server receives additional data from the client on a fully closed connection, such as another request that was sent by the client before receiving the server's response, the server's TCP stack will send a reset packet to the client; unfortunately, the reset packet might erase the client's unacknowledged input buffers before they can be read and interpreted by the client's HTTP parser. "To avoid the TCP reset problem, servers typically close a connection in stages. First, the server performs a half-close by closing only the write side of the read/write connection. The server then continues to read from the connection until it receives a corresponding close by the client, or until the server is reasonably certain that its own TCP stack has received the client's acknowledgement of the packet(s) containing the server's last response. Finally, the server fully closes the connection." So this needs shutdown(2). This is what data_to_send's close means -- this complicated close dance. h11-0.13.0/pyproject.toml000066400000000000000000000020041417207257000151010ustar00rootroot00000000000000[tool.towncrier] # Usage: # - PRs should drop a file like "issuenumber.feature" in newsfragments # (or "bugfix", "doc", "removal", "misc"; misc gets no text, we can # customize this) # - At release time after bumping version number, run: towncrier # (or towncrier --draft) package = "h11" filename = "docs/source/changes.rst" directory = "newsfragments" underlines = ["-", "~", "^"] issue_format = "`#{issue} `__" # Unfortunately there's no way to simply override # tool.towncrier.type.misc.showcontent [[tool.towncrier.type]] directory = "feature" name = "Features" showcontent = true [[tool.towncrier.type]] directory = "bugfix" name = "Bugfixes" showcontent = true [[tool.towncrier.type]] directory = "doc" name = "Improved Documentation" showcontent = true [[tool.towncrier.type]] directory = "removal" name = "Deprecations and Removals" showcontent = true [[tool.towncrier.type]] directory = "misc" name = "Miscellaneous internal changes" showcontent = true h11-0.13.0/setup.cfg000066400000000000000000000001431417207257000140100ustar00rootroot00000000000000[mypy] strict = true warn_unused_configs = true warn_unused_ignores = true show_error_codes = true h11-0.13.0/setup.py000066400000000000000000000027551417207257000137140ustar00rootroot00000000000000from setuptools import setup, find_packages # defines __version__ exec(open("h11/_version.py").read()) setup( name="h11", version=__version__, description= "A pure-Python, bring-your-own-I/O implementation of HTTP/1.1", long_description=open("README.rst").read(), author="Nathaniel J. Smith", author_email="njs@pobox.com", license="MIT", packages=find_packages(), package_data={'h11': ['py.typed']}, url="https://github.com/python-hyper/h11", # This means, just install *everything* you see under h11/, even if it # doesn't look like a source file, so long as it appears in MANIFEST.in: include_package_data=True, python_requires=">=3.6", install_requires=[ "dataclasses; python_version < '3.7'", "typing_extensions; python_version < '3.8'", ], classifiers=[ "Development Status :: 3 - Alpha", "Intended Audience :: Developers", "License :: OSI Approved :: MIT License", "Programming Language :: Python :: Implementation :: CPython", "Programming Language :: Python :: Implementation :: PyPy", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3 :: Only", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.9", "Topic :: Internet :: WWW/HTTP", "Topic :: System :: Networking", ], ) h11-0.13.0/test-requirements.txt000066400000000000000000000000221417207257000164240ustar00rootroot00000000000000pytest pytest-cov h11-0.13.0/tox.ini000066400000000000000000000010761417207257000135100ustar00rootroot00000000000000[tox] envlist = format, py36, py37, py38, py39, pypy3, mypy [gh-actions] python = 3.6: py36 3.7: py37 3.8: py38, format, mypy 3.9: py39 pypy3: pypy3 [testenv] deps = -r{toxinidir}/test-requirements.txt commands = pytest --cov=h11 --cov-config=.coveragerc h11 [testenv:format] basepython = python3.8 deps = black isort commands = black --check --diff h11/ bench/ examples/ fuzz/ isort --check --diff --profile black --dt h11 bench examples fuzz [testenv:mypy] basepython = python3.8 deps = mypy pytest commands = mypy h11/