pax_global_header 0000666 0000000 0000000 00000000064 13447420756 0014526 g ustar 00root root 0000000 0000000 52 comment=9065848247e49330560a2f73b044fb8c02338b48
sbws-1.1.0/ 0000775 0000000 0000000 00000000000 13447420756 0012503 5 ustar 00root root 0000000 0000000 sbws-1.1.0/.coveragerc 0000664 0000000 0000000 00000000033 13447420756 0014620 0 ustar 00root root 0000000 0000000 [run]
omit = */__init__.py
sbws-1.1.0/.editorconfig 0000664 0000000 0000000 00000000735 13447420756 0015165 0 ustar 00root root 0000000 0000000 # this should work for all editors that support .editorconfig!
#
# on debian, emacs users should install elpa-editorconfig and vim
# users should install vim-editorconfig.
root = true
[*]
indent_style = space
# this remove EOF new line with some editors
# insert_final_newline = true
trim_trailing_whitespace = true
end_of_line = lf
charset = utf-8
max_line_length = 79
[*.py]
indent_size = 4
[Makefile]
indent_style = tab
[*.ini]
indent_size = 4
[*.yml]
indent_size = 2
sbws-1.1.0/.gitignore 0000664 0000000 0000000 00000000200 13447420756 0014463 0 ustar 00root root 0000000 0000000 *.pyc
__pycache__/
venv*/
passwords.txt
*.swp
*.egg-info/
docs/build
.tox
.coverage
htmlcov
.pytest_cache
dist
build
*.lockfile
sbws-1.1.0/.readthedocs.yml 0000664 0000000 0000000 00000000305 13447420756 0015567 0 ustar 00root root 0000000 0000000 # .readthedocs.yml
build:
image: latest
python:
version: 3.5
# To run "pip install ." in rtfd.io
pip_install: true
# To run "pip install .[docs]" in rtfd.io
extra_requirements:
- docs
sbws-1.1.0/.travis.yml 0000664 0000000 0000000 00000001501 13447420756 0014611 0 ustar 00root root 0000000 0000000 sudo: true
dist: xenial
language: python
python:
- "3.5"
- "3.6"
install: pip install tox-travis
script:
- tox
# This is not in included in the tox envlist, in order to don't need Internet
# when running tox
# - tox -e doclinks
- tox -e clean
before_install:
- gpg --version
- sudo apt-key add tests/deb.torproject.org.asc
- echo "deb https://deb.torproject.org/torproject.org xenial main" | sudo tee -a /etc/apt/sources.list
- sudo apt-get update -qq
- sudo apt-get install tor -y
notifications:
irc:
channels:
- "irc.oftc.net#tor-ci"
template:
- "%{repository_slug} %{branch} %{commit} - %{author}: %{commit_subject}"
- "Build #%{build_number} %{result}. Details: %{build_url}"
# To build the docs
addons:
apt:
packages:
- texlive-latex-extra
- dvipng
sbws-1.1.0/AUTHORS.md 0000664 0000000 0000000 00000000272 13447420756 0014153 0 ustar 00root root 0000000 0000000 # Authors
The following people have contributed to Simple Bandwidth Scanner.
Thank you for helping make Tor better.
* juga0
* Matt Traudt
* teor
*Last updated: 2018-09-14 on 27766cc*
sbws-1.1.0/CHANGELOG.rst 0000664 0000000 0000000 00000037471 13447420756 0014540 0 ustar 00root root 0000000 0000000 Changelog
=========
All notable changes to this project will be documented in this file.
The format is based on `Keep a
Changelog `__ and this project
adheres to `Semantic Versioning `__.
v1.1.0 (2019-03-27)
-------------------
New
~~~
- V3bwfile: Report excluded relays.
Closes: #28565.
- V3bwfile: Add time to report half network.
Closes: #28983
- Destination: Recover destination when it failed.
Closes: #29589.
- V3bwfile: Report relays that fail to be measured.
Closes: #28567.
- V3bwfile: Report relays that are not measured measured.
Closes: #28566
- V3bwfile: Add KeyValues to monitor relays.
Closes: #29591.
- Docs: document that authorities are not measured.
Closes: #29722
- Scanner: Warn when there is no progress.
Closes: #28652
Fix
~~~
- v3bwfile: Report relays even when they don't reach a minimum number.
Closes: #29853.
- Minor fixes. Closes #29891.
- Relaylist: Convert consensus bandwidth to bytes.
v1.0.5 (2019-03-06)
-------------------
- Release v1.0.5.
this time with the correct version
v1.0.4 (2019-03-06)
-------------------
- Release v1.0.4.
because there was a commit missing between `1.0.3` and `1.0.4-dev0`
and what is released as `1.0.3` has version `1.0.4-dev0` and it
can not be fixed now.
v1.0.3 (2019-02-28)
-------------------
Fixed
~~~~~~
- scanner: check that ResultDump queue is not full
Fixes bug #28866. Bugfix v0.1.0.
- config: set stdout log level to cli argument. Closes: #29199
- cleanup: Use getpath to get configuration paths. Bugfix v0.7.0.
- destination: stop running twice usability tests.
Fixes bug #28897. Bugfix v0.3.0
- globals, stem: explain where torrc options are.
Fixes bug #28646. Bugfix v0.4.0
- stem: disable pad connections. Fixes bug 28692. Bugfix v0.4.0
- generate: Load all results, including error ones.
Closes #29568. Bugfix v0.4.0 (line introduced in v0.1.0).
- relayprioritizer: Stop prioritizing relays that tend to fail.
Fixes bug #28868. Bugfix v0.1.0
- circuitbuilder: Stop building the circuit 3 times.
Fixes bug #29295. Bugfix v0.1.0.
- docs: add verify option to man and example.
Closes bug #28788. Bugfix v0.4.0.
- CI: run scanner using the test network. Fixes bug #28933. Bugfix v0.1.0.
- scanner: catch SIGINT in the main loop. Fixes bug #28869. Bugfix v0.1.0.
- Stop including tests network as binary blob. Fixes bug #28590. Bugfix v0.4.0.
- relaylist: remove assertions that fail measurement.
Closes #28870. Bugfix v0.4.0
- config: Use configuration provided as argument.
Fixes bug #28724. Bugfix v0.7.0.
- stem: parse torrc options that are only a key.
Fixes bug #28715. Bugfix v0.1.1
- stem: Stop merging multiple torrc options with the same name.
Fixes bug #28738. Bugfix v0.1.1
- docs: add note about syslog when running systemd.
Closes bug #28761. Bugfix v0.6.0
- CI: include deb.torproject.org key.
Closes #28922. Bugfix v1.0.3-dev0
- config: stop allowing http servers without tls.
Fixes bug #28789. Bugfix v0.2.0.
- Make info level logs more clear and consistent.
Closes bug #28736. Bugfix v0.3.0.
- CI: check broken links in the docs. Closes #28670.
- docs: add scanner and destination requirements.
Closes bug #28647. Bugfix v0.4.0
- generate: use round_digs variable name in methods.
Closes bug #28602. Bugfix 1.0.3-dev0
- docs: Change old broken links in the documentation. Closes #28662.
- docs: replace http by https in links. Closes #28661.
- Fix git repository link. Fixes bug #28762. Bugfix v1.0.0.
- docs: add example destination in DEPLOY. Closes #28649.
- docs: Change links to be interpreted by ReST. Closes #28648.
- Force rtfd.io to install the package. Closes bug #28601.
- config: continue when the file is not found. Closes: #28550.
- Stop resolving domains locally and check same flags for the 2nd hop.
Closes bug #28458, #28471. Bugfix 1.0.4.
- Limit the relays' bandwidth to their consensus bandwidth. Closes #28598.
- globals: add torrc logging options. Closes #28645. Bugfix v0.2.0.
- Limit bandwidth to the relay MaxAdvertisedBandwidth
Fixes bug #28588. Bugfix 0.8.0.
- Exclude results, then check for the minimum number. Closes bug 28572.
- Make sbws round to 3 significant figures in torflow rounding mode.
Bugfix on 27337 in sbws 1.0. Part of 28442.
Changed
~~~~~~~~
- tests: remove unused testnets. Fixes bug #29046. Bugfix v0.4.0.
- scanner, destination: Log all possible exceptions.
- docs: Update/improve documentation on how the scanner/generator work.
Closes: #29149
- Requests: Change make_session to use the TimedSession.
- CI: change to Ubuntu Xenial.
- docs: stop editing changelog on every bug/ticket. Closes ticket #28572.
- Change sbws scaling method to torflow. Closes: #28446.
- Round bandwidths to 2 significant digits by default.
Implements part of proposal 276. Implements 28451.
Added
~~~~~~
- Send scanner metadata as part of every HTTP request. Closes: #28741
- scanner: log backtrace when not progressing. Closes: 28932
v1.0.2 (2018-11-10)
-------------------
Fixed
~~~~~
- Update bandwidth file specification version in the ``generator``
(#28366).
- Use 5 "=" characters as terminator in the bandwidth files (#28379)
Changed
~~~~~~~
- Include the headers about eligible relays in all the bandwidth files,
not only in the ones that does not have enough eligible relays
(#28365).
v1.0.1 (2018-11-01)
-------------------
Changed
~~~~~~~
- Change default directories when sbws is run from a system service
(#28268).
v1.0.0 (2018-10-29)
-------------------
**Important changes**:
- ``generate`` includes extra statistics header lines when the number
of eligible relays to include is less than the 60% of the network. It
does not include the relays' lines.
- Speed up ``scanner`` by disabling RTT measurements and waiting for
measurement threads before prioritizing again the list of relays to
measure.
Fixed
~~~~~
- Update python minimal version in setup (#28043)
- Catch unhandled exception when we fail to resolve a domain name
(#28141)
- Bandwidth filtered is the maximum between the bandwidth measurements
and their mean, not the minimum (#28215)
- Stop measuring the same relay by two threads(#28061)
Changed
~~~~~~~
- Move ``examples/`` to ``docs/`` (#28040)
- Number of results comparison and number of results away from each
other are incorrect (#28041)
- Stop removing results that are not away from some other X secs
(#28103)
- Use secs-away when provided instead of data\_period (#28105)
- Disable measuring RTTs (#28159)
- Rename bandwidth file keyvalues (#28197)
Added
-----
- Write bw file only when the percentage of measured relays is bigger
than 60% (#28062)
- When the percentage of measured relays is less than the 60%, do not
include the relays in the bandwidth file and instead include some
statistics in the header (#28076)
- When the percentage of measured relays is less than the 60% and it
was more before, warn about it (#28155)
- When the difference between the total consensus bandwidth and the
total in the bandwidth lines is larger than 50%, warn (#28216)
- Add documentation about how the bandwidth measurements are selected
and scaled before writing them to the Bandwidth File (#27692)
v0.8.0 (2018-10-08)
-------------------
**Important changes**:
- Implement Torflow scaling/aggregation to be able to substitute
Torflow with sbws without affecting the bandwidth files results.
- Change stem dependency to 1.7.0, which removes the need for
``dependency_links``
- Update and cleanup documentation
Added
~~~~~
- Add system physical requirements section to INSTALL (#26937)
- Warn when there is not enough disk space (#26937)
- Implement Torflow scaling (#27108)
- Create methods to easy graph generation and obtain statistics to
compare with current torflow results.(#27688)
- Implement rounding bw in bandwidth files to 2 insignificant
digits(#27337)
- Filter results in order to include relays in the bandwidth file
that:(#27338)
- have at least two measured bandwidths
- the measured bandwidths are within 24 hours of each other
- have at least two descriptor observed bandwidths
- the descriptor observed bandwidths are within 24 hours of each other
Fixed
~~~~~
- Broken environment variable in default sbws config. To use envvar
$FOO, write $$FOO in the config.
- Stop using directory as argument in integration tests (#27342)
- Fix typo getting configuration option to allow logging to file
(#27960)
- Set int type to new arguments that otherwise would be string (#27918)
- Stop printing arguments default values, since they are printed by
default (#27916)
- Use dash instead of underscore in new cli argument names (#27917)
Changed
~~~~~~~
- sbws install doc is confusing (#27341)
- Include system and Python dependencies in ``INSTALL``.
- Include dependencies for docs and tests in ``INSTALL``.
- Point to ``DEPLOY`` to run sbws.
- Remove obsolete sections in ``INSTALL``
- Simplify ``DEPLOY``, reuse terms in the ``glossary``.
- Remove obsolete ``sbws init`` from ``DEPLOY``.
- Point to config documentation.
- Add, unify and reuse terms in ``glossary``.
- refactor v3bwfile (#27386): move scaling method inside class
- use custom ``install_command`` to test installation commands while
``dependency_links`` is needed until #26914 is fixed. (#27704)
- documentation cleanup (#27773)
- split, merge, simplify, extend, reorganize sections and files
- generate scales as Torflow by default (#27976)
- Replace stem ``dependency_links`` by stem 1.7.0 (#27705). This also
eliminates the need for custom ``install_command`` in tox.
v0.7.0 (2018-08-09)
-------------------
**Important changes**:
- ``cleanup/stale_days`` is renamed to
``cleanup/data_files_compress_after_days``
- ``cleanup/rotten_days`` is renamed to
``cleanup/data_files_delete_after_days``
- sbws now takes as an argument the path to a config file (which
contains ``sbws_home``) instead of ``sbws_home`` (which contains the
path to a config file)
Added
~~~~~
- Log line on start up with sbws version, platform info, and library
versions (trac#26751)
- Manual pages (#26926)
Fixed
~~~~~
- Stop deleting the latest.v3bw symlink. Instead, do an atomic rename.
(#26740)
- State file for storing the last time ``sbws scanner`` was started,
and able to be used for storing many other types of state in the
future. (GH#166)
- Log files weren't rotating. Now they are. (#26881)
Changed
~~~~~~~
- Remove test data v3bw file and generate it from the same test.
(#26736)
- Stop using food terms for cleanup-related config options
- cleanup command now cleans up old v3bw files too (#26701)
- Make sbws more compatible with system packages: (#26862)
- Allow a configuration file argument
- Remove directory argument
- Create minimal user configuration when running
- Do not require to run a command to initialize
- Initialize directories when running
- Do not require configuration file inside directories specified by the
configuration
v0.6.0 (2018-07-11)
-------------------
**Important changes**:
- The way users configure logging has changed. No longer are most users
expected to be familiar with how to configure python's standard
logging library with a config file. Instead we've abstracted out the
setting of log level, format, and destinations to make these settings
more accessible to users. Expert users familiar with `the logging
config file
format `__
can still make tweaks.
Summary of changes:
- Make logging configuration easier for the user.
- Add UML diagrams to documentation. They can be found in
docs/source/images/ and regenerated with ``make umlsvg`` in docs/.
Added
~~~~~
- UML diagrams to documentation. In docs/ run ``make umlsvg`` to
rebuild them. Requires graphviz to be installed.(GHPR#226)
- Add metadata to setup.py, useful for source/binary distributions.
- Add possibility to log to system log. (#26683)
- Add option to cleanup v3bw files. (#26701)
Fixed
~~~~~
- Measure relays that have both Exit and BadExit as non-exits, which is
how clients would use them. (GH#217)
- Could not init sbws because of a catch-22 related to logging
configuration. Overhaul how logging is configured. (GH#186 GHPR#224)
- Call write method of V3BWFile class from the object instance.
(#26671)
- Stop calculating median on empty list .(#26666)
Changed
~~~~~~~
- Remove is\_controller\_ok. Instead catch possible controller
exceptions and log them
Removed
~~~~~~~
- Two parsing/plotting scripts in scripts/tools/ that can now be found
at https://github.com/pastly/v3bw-tools
v0.5.0 (2018-06-26)
-------------------
**Important changes**:
- Result format changed, causing a version bump to 4. Updating sbws to
0.5.0 will cause it to ignore results with version less than 4.
Summary of changes:
- Keep previously-generated v3bw files
- Allow a relay to limit its weight based on
RelayBandwidthRate/MaxAdvertisedBandwidth
- 1 CPU usage optimization
- 1 memory usage optimization
Added
~~~~~
- Use a relay's {,Relay}BandwidthRate/MaxAdvertisedBandwidth as an
upper bound on the measurements we make for it. (GH#155)
- Ability to only consider results for a given relay valid if they came
from when that relay is using its most recent known IP address.
Thanks Juga. (GH#154 GHPR#199)
- Maintenance script to help us find functions that are (probably) no
longer being called.
- Integration test(s) for RelayPrioritizer (GHPR#206)
- Git/GitHub usage guidelines to CONTRIBUTING document (GH#208
GHPR#215)
Fixed
~~~~~
- Make relay priority calculations take only ~5% of the time they used
to (3s vs 60s) by using sets instead of lists when selecting
non-Authority relays. (GH#204)
- Make relay list refreshing take much less time by not allowing worker
threads to dogpile on the CPU. Before they would all start requesting
descriptors from Tor at roughly the same time, causing us to overload
our CPU core and make the process take unnecessarily long. Now we let
one thread do the work so it can peg the CPU on its own and get the
refresh done ASAP. (GH#205)
- Catch a JSON decode exception on malformed results so sbws can
continue gracefully (GH#210 GHPR#212)
Changed
~~~~~~~
- Change the path where the Bandwidth List files are generated: now
they are stored in ``v3bw`` directory, named ``YYmmdd_HHMMSS.v3bw``,
and previously generated ones are kept. A ``latest.v3bw`` symlink is
updated. (GH#179 GHPR#190)
- Code refactoring in the v3bw classes and generation area
- Replace v3bw-into-xy bash script with python script to handle a more
complex v3bw file format (GH#182)
v0.4.1 (2018-06-14)
-------------------
Changed
~~~~~~~
- If the relay to measure is an exit, put it in the exit position and
choose a non-exit to help. Previously the relay to measure would
always be the first hop. (GH#181)
- Try harder to find a relay to help measure the target relay with two
changes. Essentially: (1) Instead of only picking from relays that
are 1.25 - 2.00 times faster than it by consensus weight, try (in
order) to find a relay that is at least 2.00, 1.75, 1.50, 1.25, or
v1.00 times as fast. If that fails, instead of giving up, (2) pick the
fastest relay in the network instead of giving up. This compliments
the previous change about measuring target exits in the exit
position.
Fixed
~~~~~
- Exception that causes sbws to fall back to one measurement thread. We
first tried fixing something in this area with ``88fae60bc`` but
neglected to remember that ``.join()`` wants only string arguments
and can't handle a ``None``. So fix that.
- Exception when failing to get a relay's ``ed25519_master_key`` from
Tor and trying to do ``.rstrip()`` on a None.
- ``earliest_bandwidth`` being the newest bw not the oldest (thanks
juga0)
- ``node_id`` was missing the character "$" at the beginning
sbws-1.1.0/DEPLOY.rst 0000664 0000000 0000000 00000003750 13447420756 0014236 0 ustar 00root root 0000000 0000000 .. _deploy:
Deploying Simple Bandwidth Scanner
=====================================
To run sbws is needed:
- A machine to run the :term:`scanner`.
- One or more :term:`destination` (s) that serve a large file.
Both the ``scanner`` and your the ``destination`` (s) should be on fast,
well connected machines.
.. _destinations_requirements:
destination requirements
------------------------------------
- A Web server installed and running that supports HTTP GET, HEAD and
Range (:rfc:`7233`) requests.
``Apache`` HTTP Server and ``Nginx`` support them.
- TLS support to avoid HTTP content caches at the various exit nodes.
- Certificates can be self-signed.
- A large file; at the time of writing, at least 1 GiB in size
It can be created running::
head -c $((1024*1024*1024)) /dev/urandom > 1GiB
- A fixed IP address or a domain name.
- Bandwidth: at least 12.5MB/s (100 Mbit/s).
- Network traffic: around 12-15GB/day.
scanner setup
----------------------
Install sbws according to ``_ (in the local directory or GitHub)
or ``_ (local build or Read the Docs).
To run the ``scanner`` it is mandatory to create a configuration file with at
least one ``destination``.
It is recommended to set several ``destination``s so that the ``scanner`` can
continue if one fails.
If ``sbws`` is installed from the Debian package, then create a file in
``/etc/sbws/sbws.ini`` like in the following example:
.. literalinclude:: /examples/sbws.example.ini
:caption: Example sbws.example.ini
If ``sbws`` is installed from the sources as a non-root user then create the
file in ``~/.sbws.ini``.
More details about the configuration file can be found in
``./docs/source/man_sbws.ini.rst`` (in the local directory or GitHub) or
``_ (local build or Read the Docs) or
``man sbws.ini`` (system package).
See also ``./docs/source/man_sbws.rst`` (in the local directory or GitHub) or
``_ (local build or Read the Docs) or ``man sbws`` (system
package).
sbws-1.1.0/INSTALL.rst 0000664 0000000 0000000 00000005256 13447420756 0014353 0 ustar 00root root 0000000 0000000 .. _install:
Installing Simple Bandwidth Scanner
===================================
The recommended method is to install it from your system
distribution.
In Debian/Ubuntu systems::
sudo apt install sbws
To install also the documentation::
sudo apt install sbws-doc
Continue reading to install ``sbws`` in other ways.
System requirements
--------------------
- Tor (last stable version is recommended)
- Python 3 (>= 3.5)
- virtualenv_ (while there is not ``stem`` release > 1.6.0, it is
recommended to install the required python dependencies in a virtualenv)
In Debian::
sudo apt install tor python3 virtualenv
Python dependencies
--------------------
- Stem_ >= 1.7.0
- Requests_ (with socks_ support) >= 2.10.0
To install the Python dependencies, create a ``virtualenv`` first
::
virtualenv venv -p /usr/bin/python3
source venv/bin/activate
Clone ``sbws``::
git clone https://git.torproject.org/sbws.git
Install the python dependencies::
cd sbws && pip install .
``sbws`` needs :term:`destination` s to request files from.
Please, see ``_ (in the local directory or GitHub) or
``_ (local build or Read the Docs)
to configure, deploy and run ``sbws``.
System physical requirements
-----------------------------
- Bandwidth: at least 12.5MB/s (100 Mbit/s).
- Free RAM: at least 1.5GB
- Free disk: at least 3GB
``sbws`` and its dependencies need around 20MB of disk space.
After 90 days ``sbws`` data files use around 3GB.
If ``sbws`` is configured to log to files (by default will log to the
system log), it will need a maximum of 500MB.
It is recommended to set up an automatic disk space monitoring on ``sbws`` data
and log partitions.
Details about ``sbws`` data:
``sbws`` produces around 100MB of data a day.
By default raw results' files are compressed after 10 days and deleted after 90.
The bandwidth files are compressed after 7 days and deleted after 1.
After 90 days, the disk space used by the data will be aproximately 3GB.
It will not increase further.
If ``sbws`` is configured to log to files, logs will be rotated after they
are 10MB and it will keep 50 rotated log files.
.. _virtualenv: https://virtualenv.pypa.io/en/stable/installation/
.. _Stem: https://stem.torproject.org/
.. _socks: http://docs.python-requests.org/en/master/user/advanced/#socks
.. https://readthedocs.org/projects/requests/ redirect to this, but the
.. certificate of this signed by rtd
.. _Requests: http://docs.python-requests.org/
.. http://flake8.pycqa.org/ certificate is signed by rtf
.. _Flake8: https://flake8.readthedocs.org/
.. _pytest: https://docs.pytest.org/
.. _tox: https://tox.readthedocs.io
.. _Coverage: https://coverage.readthedocs.io/
sbws-1.1.0/LICENSE.md 0000664 0000000 0000000 00000014622 13447420756 0014114 0 ustar 00root root 0000000 0000000 CC0 1.0 Universal
==================
Statement of Purpose
---------------------
The laws of most jurisdictions throughout the world automatically confer exclusive Copyright and Related Rights (defined below) upon the creator and subsequent owner(s) (each and all, an "owner") of an original work of authorship and/or a database (each, a "Work").
Certain owners wish to permanently relinquish those rights to a Work for the purpose of contributing to a commons of creative, cultural and scientific works ("Commons") that the public can reliably and without fear of later claims of infringement build upon, modify, incorporate in other works, reuse and redistribute as freely as possible in any form whatsoever and for any purposes, including without limitation commercial purposes. These owners may contribute to the Commons to promote the ideal of a free culture and the further production of creative, cultural and scientific works, or to gain reputation or greater distribution for their Work in part through the use and efforts of others.
For these and/or other purposes and motivations, and without any expectation of additional consideration or compensation, the person associating CC0 with a Work (the "Affirmer"), to the extent that he or she is an owner of Copyright and Related Rights in the Work, voluntarily elects to apply CC0 to the Work and publicly distribute the Work under its terms, with knowledge of his or her Copyright and Related Rights in the Work and the meaning and intended legal effect of CC0 on those rights.
1. Copyright and Related Rights.
--------------------------------
A Work made available under CC0 may be protected by copyright and related or neighboring rights ("Copyright and Related Rights"). Copyright and Related Rights include, but are not limited to, the following:
i. the right to reproduce, adapt, distribute, perform, display, communicate, and translate a Work;
ii. moral rights retained by the original author(s) and/or performer(s);
iii. publicity and privacy rights pertaining to a person's image or likeness depicted in a Work;
iv. rights protecting against unfair competition in regards to a Work, subject to the limitations in paragraph 4(a), below;
v. rights protecting the extraction, dissemination, use and reuse of data in a Work;
vi. database rights (such as those arising under Directive 96/9/EC of the European Parliament and of the Council of 11 March 1996 on the legal protection of databases, and under any national implementation thereof, including any amended or successor version of such directive); and
vii. other similar, equivalent or corresponding rights throughout the world based on applicable law or treaty, and any national implementations thereof.
2. Waiver.
-----------
To the greatest extent permitted by, but not in contravention of, applicable law, Affirmer hereby overtly, fully, permanently, irrevocably and unconditionally waives, abandons, and surrenders all of Affirmer's Copyright and Related Rights and associated claims and causes of action, whether now known or unknown (including existing as well as future claims and causes of action), in the Work (i) in all territories worldwide, (ii) for the maximum duration provided by applicable law or treaty (including future time extensions), (iii) in any current or future medium and for any number of copies, and (iv) for any purpose whatsoever, including without limitation commercial, advertising or promotional purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each member of the public at large and to the detriment of Affirmer's heirs and successors, fully intending that such Waiver shall not be subject to revocation, rescission, cancellation, termination, or any other legal or equitable action to disrupt the quiet enjoyment of the Work by the public as contemplated by Affirmer's express Statement of Purpose.
3. Public License Fallback.
----------------------------
Should any part of the Waiver for any reason be judged legally invalid or ineffective under applicable law, then the Waiver shall be preserved to the maximum extent permitted taking into account Affirmer's express Statement of Purpose. In addition, to the extent the Waiver is so judged Affirmer hereby grants to each affected person a royalty-free, non transferable, non sublicensable, non exclusive, irrevocable and unconditional license to exercise Affirmer's Copyright and Related Rights in the Work (i) in all territories worldwide, (ii) for the maximum duration provided by applicable law or treaty (including future time extensions), (iii) in any current or future medium and for any number of copies, and (iv) for any purpose whatsoever, including without limitation commercial, advertising or promotional purposes (the "License"). The License shall be deemed effective as of the date CC0 was applied by Affirmer to the Work. Should any part of the License for any reason be judged legally invalid or ineffective under applicable law, such partial invalidity or ineffectiveness shall not invalidate the remainder of the License, and in such case Affirmer hereby affirms that he or she will not (i) exercise any of his or her remaining Copyright and Related Rights in the Work or (ii) assert any associated claims and causes of action with respect to the Work, in either case contrary to Affirmer's express Statement of Purpose.
4. Limitations and Disclaimers.
--------------------------------
a. No trademark or patent rights held by Affirmer are waived, abandoned, surrendered, licensed or otherwise affected by this document.
b. Affirmer offers the Work as-is and makes no representations or warranties of any kind concerning the Work, express, implied, statutory or otherwise, including without limitation warranties of title, merchantability, fitness for a particular purpose, non infringement, or the absence of latent or other defects, accuracy, or the present or absence of errors, whether or not discoverable, all to the greatest extent permissible under applicable law.
c. Affirmer disclaims responsibility for clearing rights of other persons that may apply to the Work or any use thereof, including without limitation any person's Copyright and Related Rights in the Work. Further, Affirmer disclaims responsibility for obtaining any necessary consents, permissions or other rights required for any use of the Work.
d. Affirmer understands and acknowledges that Creative Commons is not a party to this document and has no duty or obligation with respect to this CC0 or use of the Work.
sbws-1.1.0/MANIFEST.in 0000664 0000000 0000000 00000000220 13447420756 0014233 0 ustar 00root root 0000000 0000000 include *.md
include *.rst
include *.ini
recursive-include docs *
prune docs/build
recursive-include tests *
recursive-exclude **/__pycache__ *
sbws-1.1.0/README.md 0000664 0000000 0000000 00000005007 13447420756 0013764 0 ustar 00root root 0000000 0000000 # Readme
[](https://travis-ci.org/https://travis-ci.org/torproject/sbws)
Simple Bandwidth Scanner (called `sbws`) is a Tor bandwidth scanner that
generates bandwidth files to be used by Directory Authorities.
The scanner measures the bandwidth of each relay in the Tor network
(except the directory authorities) by creating a two hops circuit
with the relay. It then measures the bandwidth by downloading data
from a destination Web Server and stores the measurements.
The generator read the measurements, aggregates, filters and
scales them using torflow's scaling method.
Then it generates a bandwidth list file that is read
by a directory authority to report relays’ bandwidth in its vote.
**WARNING**: This software is intended to be run by researchers using a test
Tor network, such as chutney or shadow, or by the Tor bandwidth authorities
on the public Tor network.
Please do not run this software on the public Tor network unless you are one
of the Tor bandwidth authorities, to avoid creating unnecessary traffic.
**ADVICE**: It is recommended to read this documentation at
[Read the Docs](https://sbws.rtfd.io). In
[Github](https://github.com/torproject/sbws) some links won't be properly
rendered.
It can also be read after installing the Debian package ``sbws-doc`` in
``/usr/share/doc/sbws`` or after building it locally as explained in
``./docs/source/documenting.rst``.
Installing
------------
See [./INSTALL.rst](INSTALL.rst) (in local directory or GitHub) or
[INSTALL.html](INSTALL.html) (local build or Read the Docs).
Deploying and running
---------------------
See [./DEPLOY.rst](DEPLOY.rst) (in local directory or GitHub) or
[DEPLOY.html](DEPLOY.html) (local build or Read the Docs).
Changelog
--------------
See [./CHANGELOG.rst](CHANGELOG.rst) (in local directory or GitHub) or
[CHANGELOG.html](CHANGELOG.html) (local build or Read the Docs).
Documentation
--------------
More extensive documentation can be found in the ``./docs`` directory,
and online at [sbws.rtfd.io](https://sbws.readthedocs.io).
## License
This work is in the public domain within the United States.
We waive copyright and related rights in the work worldwide through the
[CC0-1.0 license](https://creativecommons.org/publicdomain/zero/1.0).
You can find a copy of the CC0 Public Domain Dedication along with this
software in ./LICENSE.md
## Authors
See [./AUTHORS.md](AUTHORS.md) (in local directory or GitHub) or
[AUTHORS.html](AUTHORS.html) (local build or Read the Docs). sbws-1.1.0/docs/ 0000775 0000000 0000000 00000000000 13447420756 0013433 5 ustar 00root root 0000000 0000000 sbws-1.1.0/docs/Makefile 0000664 0000000 0000000 00000002133 13447420756 0015072 0 ustar 00root root 0000000 0000000 # Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line.
SPHINXOPTS =
SPHINXBUILD = sphinx-build
SPHINXPROJ = simple-bw-scanner
SOURCEDIR = source
BUILDDIR = build
# generate SVG
IMAGEDIRS = $(SOURCEDIR)/images
BUILDDIRIMAGES = $(BUILDDIR)/html/_images
PYREVERSE = pyreverse
PYREVERSE_FLAGS = -o svg -p sbws ../sbws
UMLSVG := $(PYREVERSE) $(PYREVERSE_FLAGS);mv *.svg $(IMAGEDIRS);mkdir -p $(BUILDDIRIMAGES);cp $(IMAGEDIRS)/*.svg $(BUILDDIRIMAGES)
# Put it first so that "make" without argument is like "make help".
help:
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
umlsvg:
@echo "Generating UML SVG"
$(UMLSVG)
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
# commented because if system packages invoke make html, it'll automatically
# recreate the svg on every build, and it's not deterministic.
#$(UMLSVG)
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
sbws-1.1.0/docs/make.bat 0000664 0000000 0000000 00000001471 13447420756 0015043 0 ustar 00root root 0000000 0000000 @ECHO OFF
pushd %~dp0
REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=source
set BUILDDIR=build
set SPHINXPROJ=simple-bw-scanner
if "%1" == "" goto help
%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.http://sphinx-doc.org/
exit /b 1
)
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
goto end
:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
:end
popd
sbws-1.1.0/docs/source/ 0000775 0000000 0000000 00000000000 13447420756 0014733 5 ustar 00root root 0000000 0000000 sbws-1.1.0/docs/source/AUTHORS.md 0000777 0000000 0000000 00000000000 13447420756 0020472 2../../AUTHORS.md ustar 00root root 0000000 0000000 sbws-1.1.0/docs/source/CHANGELOG.rst 0000777 0000000 0000000 00000000000 13447420756 0021416 2../../CHANGELOG.rst ustar 00root root 0000000 0000000 sbws-1.1.0/docs/source/DEPLOY.rst 0000777 0000000 0000000 00000000000 13447420756 0020630 2../../DEPLOY.rst ustar 00root root 0000000 0000000 sbws-1.1.0/docs/source/INSTALL.rst 0000777 0000000 0000000 00000000000 13447420756 0021054 2../../INSTALL.rst ustar 00root root 0000000 0000000 sbws-1.1.0/docs/source/README.md 0000777 0000000 0000000 00000000000 13447420756 0020112 2../../README.md ustar 00root root 0000000 0000000 sbws-1.1.0/docs/source/activity_all.puml 0000664 0000000 0000000 00000000372 13447420756 0020320 0 ustar 00root root 0000000 0000000 @startuml
start
while (no SIGINT/SIGTERM?)
while (next relay to measure?)
:Select a destination;
:Select a second relay;
:Build a circuit;
:HTTP GET (Range-Bytes);
:Store measurement;
endwhile
endwhile
stop
@enduml sbws-1.1.0/docs/source/activity_second_relay.puml 0000664 0000000 0000000 00000001010 13447420756 0022205 0 ustar 00root root 0000000 0000000 @startuml
start
if (relay to measure is exit?) then (yes)
:obtain non-exits;
else (no)
:obtain an exits
without bad flag
that can exit
to port 443;
endif
:potential second relays;
:obtain a relay
from potential
sencond relays
randomly;
if (second relay has 2x bandwidth?) then (yes)
elseif (other second relay has 1.5x bandwidth?) then (yes)
elseif (other second relay has 1x bandwidth?) then (yes)
else (nothing)
stop
endif
:second relay selected!;
:Build a circuit
whith exit as
second hop;
stop
@enduml sbws-1.1.0/docs/source/bandwidth_authorities.rst 0000664 0000000 0000000 00000001370 13447420756 0022052 0 ustar 00root root 0000000 0000000 Bandwidth authorities in metrics
=================================
Current bandwidth authorities
-----------------------------
.. image:: images/bwauth.*
:alt: bandwidth authorities in metrics
https://metrics.torproject.org/rs.html
(flag:Authority)
Bandwidth Authorities - Measured Relays past 7 days
---------------------------------------------------
.. image:: images/bwauth_measured_7days.png
:alt: bandwidth measured in the past 7 days
https://consensus-health.torproject.org/graphs.html
Bandwidth Authorities - Measured Relays past 90 days
----------------------------------------------------
.. image:: images/bwauth_measured_90days.png
:alt: bandwidth measured in the past 90 days
https://consensus-health.torproject.org/graphs.html
sbws-1.1.0/docs/source/bandwidth_distribution.rst 0000664 0000000 0000000 00000002017 13447420756 0022230 0 ustar 00root root 0000000 0000000 Relays' bandwidth distribution
===================================
sbws raw measurements compared to Torflow measurements
------------------------------------------------------
.. image:: images/43710932-ac1eeea8-9960-11e8-9e7e-21fddff2f7a3.png
:alt: sbws and torflow raw measurements distribution
.. image:: images/43710933-ac95e0bc-9960-11e8-9aaf-0bb1f83b65e2.png
:alt: sbws and torflow raw measurements distribution 2
sbws linear scaling
--------------------
Multiply each relay bandwidth by ``7500/median``
See bandwidth_file_spec_ appendix B to know how about linear scaling.
Code: :func:`sbws.lib.v3bwfile.sbws_scale`
.. image:: images/20180901_163442.png
:alt: sbws linear scaling
sbws Torflow scaling
-----------------------
See bandwidth_file_spec_ appendix B to know how about torflow scaling.
Code: :func:`sbws.lib.v3bwfile.torflow_scale`
.. image:: images/20180901_164014.png
:alt: sbws torflow scaling
.. _bandwidth_file_spec: https://gitweb.torproject.org/torspec.git/tree/bandwidth-file-spec.txt
sbws-1.1.0/docs/source/classes_original.puml 0000664 0000000 0000000 00000011610 13447420756 0021152 0 ustar 00root root 0000000 0000000 @startuml
class RelayList {
stem.Controller _controller
Lock _refresh_lock
int _last_refresh
list @p relays
list @p bad_exits
list @p exits
list @p non_exits
list @p authorities
bool _need_refresh()
_init_relays()
_refresh()
list _relays_with_flag(int flag)
list _relays_without_flag(int flag)
list exits_not_bad_can_exit_to_port(int port)
}
RelayList *-- Relay
class Relay {
stem.RouterStatusEntryV3 _from_ns
stem.RelayDescriptor _from_desc
str @p nickname
str @p fingerprint
list @p flags
ExitPolicy @p exit_policy
str @p address
str @p master_key_ed25519
int @p observed_bandwidth
int @p average_bandwidth
int @p burst_bandwidth
int @p consensus_bandwidth
int @p consensus_bandwidth_is_unmeasured
obj _from_ns(attr)
obj _from_desc(attr)
bool can_exit_to_port(int port)
bool is_exit_not_bad_allowing_port(int port)
}
class RelayPrioritizer {
int fresh_seconds
ResultDump result_dump
RelayList relay_list
bool measure_authorities
generator best_priority()
}
RelayPrioritizer *-- RelayList
RelayPrioritizer *-- ResultDump
Result ^-- ResultError
Result ^-- ResultSuccess
Result -- Destination
class Result {
Result.Relay _relay
list @p circ
str @p dest_url
str @p scanner
int @p time
str @p type
int @p version
str @p nickname
str @p fingerprint
str @p address
str @p master_key_ed25519
int @p relay_observed_bandwidth
int @p relay_average_bandwidth
int @p relay_burst_bandwidth
int @p consensus_bandwidth
int @p consensus_bandwidth_is_unmeasured
dict to_dict()
Result from_dict(dict d)
}
Result -- Relay
Result *-- Result.Relay
class Result.Relay {
str nickname
str fingerprint
str address
str master_key_ed25519
int observed_bandwidth
int average_bandwidth
int burst_bandwidth
int consensus_bandwidth
int consensus_bandwidth_is_unmeasured
}
class ResultError {
str @p msg
}
ResultError ^-- ResultErrorCircuit
class ResultErrorCircuit {
}
ResultError ^-- ResultErrorStream
class ResultSuccess {
list @p rtts
list @p downloads
}
ResultDump *-- Result
ResultDump -- Relay
class ResultDump {
dict data
int fresh_days
str datadir
Lock data_lock
Thread thread
Queue queue
store_result(Result result)
handle_result(Result result)
enter()
list results_for_relay(Relay relay)
}
class DestinationList {
list _rl
Destination next()
DestinationList @sm from_config(...)
}
DestinationList *-- Destination
class Destination {
str @p hostname
int @p port
str @p url
bool @p verify
bool is_usable()
Destination @sm from_config(str conf_section,int max_dl)
}
V3BWHeader -- Result
class V3BWHeader {
int timestamp
str version
str file_created
str latest_bandwidth
int num_lines
str software
str software_version
str generator_started
int number_eligible_relays
int minimum_number_eligible_relays
int number_consensus_relays
int percent_eligible_relays
int minimum_percent_eligible_relays
int @p num_lines
V3BWHeader @cm from_results(dict results)
add_stats(**kwargs)
int @sm earliest_bandwidth_from_results(dict results)
str @sm generator_started_from_file(dict results)
int @sm latest_bandwidth_from_results(dict results)
}
V3BWLine -- Result
class V3BWLine {
int bw
str node_id
str master_key_ed25519
str nick
int rtt
str time
int success
int error_stream
int error_circ
int error_misc
int bw_median
int bw_mean
int desc_bw_avg
int desc_bw_bur
int desc_bw_obs_last
int desc_bw_obs_mean
consensus_bandwidth
consensus_bandwidth_is_unmeasured
int @sm bw_mean_from_results(list results)
int @sm bw_median_from_results(list results)
int @sm desc_bw_obs_last_from_results(list results)
int @sm desc_bw_obs_mean_from_results(list results)
V3BWLine @cm from_results(list results)
str @sm last_time_from_results(list results)
dict @sm result_types_from_results(list results)
list @sm results_away_each_other(list results)
list @sm results_recent_than(list results)
}
V3BWFile *-- V3BWHeader
V3BWFile *-- V3BWLine
V3BWFile -- Result
class V3BWFile {
V3BWHeader header
list bw_lines
@p info_stats
bool @p is_min_perc
int @p max_bw
int @p mean_bw
int @p median_bw
int @p min_bw
int @p num
int @p sum_bw
V3BWFile @cm from_results(dict results, ...)
list @sm bw_kb(bw_lines)
list @sm bw_sbws_scale(bw_lines)
list @sm bw_torflow_scale(bw_lines)
bool @sm is_max_bw_diff_perc_reached(bw_lines)
(dict, bool) @sm measured_progress_stats(bw_lines)
int @sm read_number_consensus_relays(str consensus_path)
(list, list, list) to_plt()
list update_progress(bw_lines, ...)
warn_if_not_accurate_enough(bw_lines, ...)
tuple to_plt(...)
write(str output)
}
CircuitBuilder *-- RelayList
CircuitBuilder -- Relay
class CircuitBuilder {
set built_circuits
RelayList relay_list
list relays
Controller controller
int build_circuit()
void close_circuit()
}
CircuitBuilder ^-- GapsCircuitBuilder
class State {
get()
}
@enduml sbws-1.1.0/docs/source/code_design.rst 0000664 0000000 0000000 00000002523 13447420756 0017732 0 ustar 00root root 0000000 0000000 Code design
=================
.. todo::
- Link to refactor proposal.
- Change this page when refactoring is implemented.
UML classes diagram
--------------------
.. image:: images/classes_original.*
:alt: UML classes diagram
`classes_original.svg <./_images/classes_original.svg>`_
Packages diagram
-----------------
.. image:: ./images/packages_sbws.*
:alt: packages diagram
`packages_sbws.svg <./_images/packages_sbws.svg>`_
scanner threads
----------------
- `TorEventListener`: the thread that runs Tor and listens for events.
- ResultDump: the thread that get the measurement results from a queue
every second.
- `multiprocessing.ThreadPool` starts 3 independent threads:
- workers_thread
- tasks_thread
- results_thread
- measurement threads: they execute :func:`sbws.core.scanner.measure_relay`
There'll be a maximum of 3 by default.
.. image:: images/threads.*
:alt: scanner threads
Critical sections
-----------------
Data types that are read or wrote from the threads.
.. image:: images/critical_sections.*
:alt: scanner critical sections
:height: 400px
:align: center
Call graph
--------------
Initialization calls to the moment where the measurement threads start.
.. image:: images/pycallgraph.png
:alt: call graph
:height: 400px
:align: center
`callgraph.png <./_images/pycallgraph.png>`_
sbws-1.1.0/docs/source/conf.py 0000664 0000000 0000000 00000014020 13447420756 0016227 0 ustar 00root root 0000000 0000000 # -*- coding: utf-8 -*-
#
# Configuration file for the Sphinx documentation builder.
#
# This file does only contain a selection of the most common options. For a
# full list see the documentation:
# http://www.sphinx-doc.org/en/stable/config
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
import os
def find_version():
with open(os.path.join("..", "..", "sbws", "__init__.py")) as fp:
for line in fp:
if "__version__" in line.strip():
version = line.split("=", 1)[1].strip().strip("'")
return version
# -- Project information -----------------------------------------------------
project = 'sbws'
copyright = 'CC0'
author = 'Matt Traudt [sirmatt at ksu dot edu], juga [juga at riseup dot net]'
# The short X.Y version
version = find_version()
# The full version, including alpha/beta/rc tags
release = version
# -- General configuration ---------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#
# needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.doctest',
'sphinx.ext.todo',
'sphinx.ext.coverage',
'sphinx.ext.githubpages',
'sphinx.ext.imgmath',
'sphinx.ext.intersphinx',
'sphinx.ext.viewcode'
]
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
#
source_suffix = ['.rst', '.md']
# source_suffix = '.rst'
# The master toctree document.
master_doc = 'index'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path .
exclude_patterns = []
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_theme = 'nature'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#
# html_theme_options = {}
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']
# Custom sidebar templates, must be a dictionary that maps document names
# to template names.
#
# The default sidebars (for documents that don't match any pattern) are
# defined by theme itself. Builtin themes are using these templates by
# default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
# 'searchbox.html']``.
#
# html_sidebars = {}
# html_last_updated_fmt = ''
# -- Options for HTMLHelp output ---------------------------------------------
# Output file base name for HTML help builder.
htmlhelp_basename = 'simple-bw-scannerdoc'
# -- Options for LaTeX output ------------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#
# 'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
#
# 'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
#
# 'preamble': '',
# Latex figure (float) alignment
#
# 'figure_align': 'htbp',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title,
# author, documentclass [howto, manual, or own class]).
latex_documents = [
(master_doc, 'simple-bw-scanner.tex', 'simple-bw-scanner Documentation',
[author], 'manual'),
]
# -- Options for manual page output ------------------------------------------
# One entry per manual page. List of tuples
# (source start file, name, description, authors, manual section).
man_pages = [
('man_sbws', 'sbws', 'Simple Bandwidth Scanner', [author], 1),
('man_sbws.ini', 'sbws.ini',
'Simple Bandwidth Scanner configuration', [author], 5)
]
# -- Options for Texinfo output ----------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
(master_doc, 'simple-bw-scanner', 'simple-bw-scanner Documentation',
author, 'simple-bw-scanner', 'One line description of project.',
'Miscellaneous'),
]
# -- Options for Epub output -------------------------------------------------
# Bibliographic Dublin Core info.
epub_title = project
epub_author = author
epub_publisher = author
epub_copyright = copyright
# The unique identifier of the text. This can be a ISBN number
# or the project homepage.
#
# epub_identifier = ''
# A unique identification for the text.
#
# epub_uid = ''
# A list of files that should not be packed into the epub file.
epub_exclude_files = ['search.html']
# -- Extension configuration -------------------------------------------------
# -- Options for todo extension ----------------------------------------------
# If true, `todo` and `todoList` produce output, else they produce nothing.
todo_include_todos = True
source_parsers = {
'.md': 'recommonmark.parser.CommonMarkParser',
}
numfig = True
sbws-1.1.0/docs/source/config.default.ini 0000777 0000000 0000000 00000000000 13447420756 0025314 2../../sbws/config.default.ini ustar 00root root 0000000 0000000 sbws-1.1.0/docs/source/config.example.ini 0000664 0000000 0000000 00000001077 13447420756 0020340 0 ustar 00root root 0000000 0000000 # Minimum configuration that needs to be customized
[scanner]
# A human-readable string with chars in a-zA-Z0-9 to identify your scanner
nickname = sbws_default
# ISO 3166-1 alpha-2 country code. To be edited.
# Default to a non existing country to detect it was not edited.
country = AA
[destinations]
foo = off
[destinations.foo]
url = https://example.com/does/not/exist.bin
# ISO 3166-1 alpha-2 country code. To be edited.
# Use ZZ if the destination URL is a domain name and it is in a CDN.
# Default to a non existing country to detect it was not edited.
country = AA
sbws-1.1.0/docs/source/config.log.default.ini 0000777 0000000 0000000 00000000000 13447420756 0026654 2../../sbws/config.log.default.ini ustar 00root root 0000000 0000000 sbws-1.1.0/docs/source/config.rst 0000664 0000000 0000000 00000003616 13447420756 0016740 0 ustar 00root root 0000000 0000000 .. _config_internal:
Internal code configuration files
==================================
Sbws has two default config files it reads: on general, and one specific to
logging.
They all get combined internally to the same ``conf`` structure.
It first reads the config file containing the default values for almost all
options. If you installed sbws in a virtual environment located at /tmp/venv,
then you will probably find the ``config.default.ini`` in a place such as
``/tmp/venv/lib/python3.5/site-packages/sbws/`` **You should never edit this
file**. The contents of this default config file can be found :ref:`at the
bottom of this page `.
Second, ``sbws`` will read ``config.log.default.ini``. It will be located in
the same place as the previous file, and **should not be edited** like the
previous file. The contents of this default log config file can be found
:ref:`at the bottom of this page `. Options set here
overwrite options set in the previous config file.
Sbws then reads your custom config file. By default, it will search for it
in ``~/.sbws.ini``. Options in this file overwrite options set in previously
read config files.
The user example config file provided by ``sbws`` might look like this.
.. _init-config:
.. literalinclude:: /examples/sbws.example.ini
:caption: Example sbws.example.ini
**No other configuration files are read.**
.. _default-config:
Default Configuration
----------------------
.. literalinclude:: config.default.ini
:caption: config.default.ini
.. _default-log-config:
If you know how to use
`Python's logging configuration file format`_,
then you can override or add to what is listed here by editing your config.ini.
.. literalinclude:: config.log.default.ini
:caption: config.log.default.ini
.. _Python's logging configuration file format: https://docs.python.org/3.5/library/logging.config.html#logging-config-fileformat
sbws-1.1.0/docs/source/config_tor.rst 0000664 0000000 0000000 00000002616 13447420756 0017623 0 ustar 00root root 0000000 0000000 .. _config_tor:
Internal Tor configuration for the scanner
------------------------------------------
The scanner needs an specific Tor configuration.
The following options are either set when launching Tor or required when
connection to an existing Tor daemon.
Default configuration:
- ``SocksPort auto``: To proxy requests over Tor.
- ``CookieAuthentication 1``: The easiest way to authenticate to Tor.
- ``UseEntryGuards 0``: To avoid path bias warnings.
- ``UseMicrodescriptors 0``: Because full server descriptors are needed.
- ``SafeLogging 0``: Useful for logging, since there's no need for anonymity.
- ``LogTimeGranularity 1``
- ``ProtocolWarnings 1``
- ``LearnCircuitBuildTimeout 0``: To keep circuit build timeouts static.
Configuration that depends on the user configuration file:
- ``CircuitBuildTimeout ...``: The timeout trying to build a circuit.
- ``DataDirectory ...``: The Tor data directory path.
- ``PidFile ...``: The Tor PID file path.
- ``ControlSocket ...``: The Tor control socket path.
- ``Log notice ...``: The Tor log level and path.
Configuration that needs to be set on runtime:
- ``__DisablePredictedCircuits 1``: To build custom circuits.
- ``__LeaveStreamsUnattached 1``
Currently most of the code that sets this configuration is in :func:`sbws.util.stem.launch_tor`
and the default configuration is ``sbws/globals.py``.
.. note:: the location of these code is being refactored. sbws-1.1.0/docs/source/contributing.rst 0000664 0000000 0000000 00000023512 13447420756 0020177 0 ustar 00root root 0000000 0000000 .. _contributing:
Contributing to Simple Bandwidth Scanner
=========================================
Thank you for your interest in Simple Bandwidth Scanner (``sbws``).
Examples of contributions include:
* Bug reports, feature requests
* Code/documentation patches
Bug reports or feature requests
---------------------------------
* Check that it has not been already reported.
.. _ticket-ref:
* Open a ticket in
`Tor Project Trac `_
and assign the component to ``Core Tor``/``sbws``.
Code/documentation patches
---------------------------
The sbws canonical repository is https://gitweb.torproject.org/sbws.git,
but we review patches using the Github canonical repository
(https://github.com/torproject/sbws) Pull Requests (PR).
To know more about ``sbws`` code,
.. seealso::
- :ref:`dev_doc`
- ``./docs/source/testing.rst`` (or `testing `_
or :ref:`testing`).
- ``./docs/source/documenting.rst`` (or `documenting `_
or :ref:`documenting`).
The following are guidelines we aim to follow.
Steps to create a PR
~~~~~~~~~~~~~~~~~~~~~
1. Create a ticket in Tor Project Trac (:ref:`Open ticket `)
2. Clone ``sbws`` via the Github web interface
https://github.com/torproject/sbws
3. Clone the repository locally
4. Install ``sbws`` as explained in ./INSTALL.rst and ./TESTING.rst
Use ``pip install -e <>``
5. If needed install the documentation and build it as explained in
./DOCUMENTATION.rst
6. Create a new branch, named ``ticketXXX``.
Optionally, name it with a string explaining what it does,
ie ``ticketXXX_contributing``
7. Write code (:ref:`codestyle-ref`), tests, documentation,
extra files (:ref:`extrafiles-ref`), commit (:ref:`commits-ref`), etc.
8. Ensure tests pass (./TESTING.rst).
9. Push your branch to your repository. If you have an account in Travis,
you can see whether it pass the tests in Github and in
https://travis-ci.org/youruser/sbws/
10. Create a PR from your branch to https://github.com/torproject/sbws
11. Change the Trac ticket status to ``needs_review``
.. _codestyle-ref:
Code style
~~~~~~~~~~
Follow the Zen of Python (:pep:`20`)
.. code-block:: pycon
>>> import this
The Zen of Python, by Tim Peters
Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Code should adhere to the :pep:`8` guidelines.
Before release 1.0.0, some guidelines have not been followed,
such as the ordering the inputs (:pep:`8#imports`).
External link: `Code Style `_
All functions, methods and classes should have :pep:`0257`
(except ``__repr__`` and ``__str__``).
Before release 1.0.0, some docstrigs do not have 3 double quotes ``"""``
(:pep:`0257#id15`).
External link: `Documentation `_
New features should add a corresponding documentation in /docs.
An editor compatible with `EditorConfig `_ will
help you to follow the general formatting code style.
Timestamps must be in UTC. It is prefered to use ``datetime`` objects or
Unix timestamps. Timestamps read by the user should be always formatted in
`ISO 8601 `_
Functional style is prefered:
- use list comprenhensions lambda, map, reduce
- avoid reasigigning variables, instead create new ones
- use ``deepcopy`` when passing list of objects to a function/method
- classes should change attributes only in one method (other than __init__?)
[FUNC]_
In general, do not reinvent the wheel, use Python native modules as ``logging``,
instead of implementing similar functionality.
Or use other packages when the new dependency can be extra, for instance
`vulture`_.
.. _`extrafiles-ref`:
Extra required files
~~~~~~~~~~~~~~~~~~~~~
Any non-trivial change should contain tests. See ./TESTING.rst.
When running tests, currently ``flake8`` informs on some PEP8 errors/warnings,
but not all.
.. _commits-ref:
Commits
~~~~~~~~~
Each commit should reference the Tor Project Trac ticket (example: ``#12345``)
and possibly the bugfix version.
Try to make each commit a logically separate changes.::
As a general rule, your messages should start with a single line that’s
o more than about 50 characters and that describes the changeset concisely,
followed by a blank line, followed by a more detailed explanation.
The Git project requires that the more detailed explanation include
your motivation for the change and contrast its implementation with
previous behavior — this is a good guideline to follow.
It’s also a good idea to use the imperative present tense in these messages.
In other words, use commands.
Instead of "I added tests for" or "Adding tests for," use "Add tests for."
[DIST]_
Template originally written by `Tim Pope`_: :ref:`example commit `
Code being reviewed workflow
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When a PR is being reviewed, new changes might be needed:
- If the change does not modify a previous change, create new commits and push.
- If the change modifies a previous change and it's small,
`git commit fixup `_
should be used. When it is agreed that the PR is ready, create a new branch
named ``mybranch_02`` and run:
.. code-block:: bash
rebase --autosquash
push, create new PR and close old PR mentioning the number of the new PR.
- If the review takes long and when it's ready code related to the PR has changed
in master, create a new branch named ``mybranch_02`` and run:
.. code-block:: bash
rebase master
push, create new PR and close old PR mentioning the number of the new PR.
[MERG]_
.. _review-ref:
Reviewing code
----------------
All code should be peer-reviewed. Two reasons for this are::
Because a developer cannot think of everything at once;
Because a fresh pair of eyes may spot an error, a corner-case in the code,
insufficient documentation, a missing consistency check, etc.
[REVI]_
Reviewers:
- Should let the contributor know what to improve/change.
- Should not push code to the contributor's branch.
- Should wait for contributor's changes or feedback after changes are requested,
before merging or closing a PR.
- Should merge (not rebase) the PR.
- If rebase is needed due to changes in master, the contributor should create
a new branch named `xxx_rebased` based on the reviewed branch, rebase and
create a new PR from it, as explained above.
- If new changes are needed when the contributor's branch is ready to merge,
the reviewer can create a new branch based on the contributor's branch,
push the changes and merge that PR.
The contributor should be notified about it.
- If the reviewer realize that new changes are needed after the PR has been
merged, the reviewer can push to master, notifying the contributor about the
changes.
- Because currently there are not many reviewers, reviewers can merge their own
PR if there was not any feedback after a week.
- Should not push directly to master, unless changes are trivial (typos,
extra spaces, etc.)
- Should not push to master new features while there are open PRs to review.
Currently, the reviewers are the persons that have contributed to the code:
pastly, teor, juga.
.. _releases-ref:
Releases
----------
Releases follow `semantic versioning`_.
Until release 1.0.0 is reached, this project is not considered production
ready.
Currently development happens in master, this might change from release 1.0.0
so that master has the last release changes, and development happens in the
next release branch.
Before major releases, ensure that:
- Installation from scratch, as specified in ./INSTALL.md, must success.
- All tests must pass.
- Tor must be able to parse the produced bw files
(current way is manual)
.. todo::
Test that run Tor as dirauth and parse the files
- Bandwidth files must produce graphs compatible with Torflow
(current way to test it is manual)
.. todo::
Implement something to compare error with current consensus.
- A dirauth should be able to understand the documentation, otherwise the
documentation should be clarified.
.. _changelog:
Create a ./CHANGELOG.rst file.
Each entry should reference the Tor Project Trac ticket (example: ``#12345``)
and possibly the bugfix version.
Until version 1.0.2 we have followed `keep a changelog`_ format.
.. _commit-msg:
Example commit message
-----------------------
::
Short (50 chars or less) summary of changes
More detailed explanatory text, if necessary. Wrap it to
about 72 characters or so. In some contexts, the first
line is treated as the subject of an email and the rest of
the text as the body. The blank line separating the
summary from the body is critical (unless you omit the body
entirely); tools like rebase can get confused if you run
the two together.
Further paragraphs come after blank lines.
- Bullet points are okay, too
- Typically a hyphen or asterisk is used for the bullet,
preceded by a single space, with blank lines in
between, but conventions vary here
.. rubric:: External eferences
.. [DIST] https://git-scm.com/book/en/v2/Distributed-Git-Contributing-to-a-Project
.. [MERG] https://www.atlassian.com/git/tutorials/merging-vs-rebasing
.. [REVI] https://doc.sagemath.org/html/en/developer/reviewer_checklist.html
.. [FUNC] https://medium.com/@rohanrony/functional-programming-in-python-1-lambda-map-filter-reduce-zip-8739ea144186
.. _tim pope: https://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html
.. _`keep a changelog`: https://keepachangelog.com/en/1.0.0/
.. _`semantic versioning`: https://semver.org/
.. _`vulture`: https://pypi.org/project/vulture/
sbws-1.1.0/docs/source/data/ 0000775 0000000 0000000 00000000000 13447420756 0015644 5 ustar 00root root 0000000 0000000 sbws-1.1.0/docs/source/data/critical_sections.dia 0000664 0000000 0000000 00000006724 13447420756 0022035 0 ustar 00root root 0000000 0000000 ‹ í]A“â6ľĎŻ KRŐ¶$Ë–CzR›¤¶ö0»©Úť\rˇ¸iÍô°‡ýí+Éî
Ʋ