debian/0000755000000000000000000000000012231276041007164 5ustar debian/control0000644000000000000000000000212512231274743010576 0ustar Source: python-stem Section: python Priority: optional Maintainer: Ulises Vitulli Build-Depends: debhelper (>= 9), python-all (>= 2.6.6-6), python3-all Standards-Version: 3.9.4 Homepage: https://gitweb.torproject.org/stem.git X-Python-Version: >= 2.7 Vcs-Git: git://git.debian.org/collab-maint/python-stem.git Vcs-Browser: http://anonscm.debian.org/gitweb/?p=collab-maint/python-stem.git Package: python-stem Architecture: all Depends: ${python:Depends}, ${misc:Depends} Description: Tor control library for Python Stem is a TorCtl successor, a Python Tor controller with extensions to support path building and various constraints on node and path selection, as well as statistics gathering. Package: python3-stem Architecture: all Depends: ${python3:Depends}, ${misc:Depends} Replaces: python-stem (<< 1.0.1-3) Breaks: python-stem (<< 1.0.1-3) Description: Tor control library for Python 3 series Stem is a TorCtl successor, a Python 3 Tor controller with extensions to support path building and various constraints on node and path selection, as well as statistics gathering. debian/rules0000755000000000000000000000115412231246432010246 0ustar #!/usr/bin/make -f # -*- makefile -*- # # # Uncomment this to turn on verbose mode. #export DH_VERBOSE=1 PYVERS := python2.7 PY3VERS := $(shell py3versions -s -d) %: dh $@ --with python2,python3 override_dh_installchangelogs: dh_installchangelogs debian/ChangeLog override_dh_auto_install: dh_auto_install set -ex; \ for py in $(PYVERS); do \ $$py setup.py install --root debian/python-stem \ --install-layout=deb; \ done; \ for py in $(PY3VERS); do \ $$py setup.py install --root debian/python3-stem \ --install-layout=deb; \ done; \ override_dh_auto_clean: dh_auto_clean -- --all rm -rf build debian/changelog0000644000000000000000000000272712231274743011055 0ustar python-stem (1.1.0-1) unstable; urgency=low * New upstream release. * debian/control: Update Homepage field. * debian/copyright: Update Source field. -- Ulises Vitulli Mon, 21 Oct 2013 13:00:01 -0300 python-stem (1.0.1-3) unstable; urgency=low [ Scott Kitterman ] * Move python3 files into python3-stem so they can have correct dependencies - Add python3-stem to debian/control - Adjust debian/rules * Adjust python related build-depends so clean works. [ Ulises Vitulli ] * Deploy separated packages for 2 and 3 python series, thanks ScottKitterman (Closes: #712743). * Replaces and Breaks on older python-stem for safely transitioning. * Minor fixes at copyright machine readable format. -- Ulises Vitulli Fri, 16 Aug 2013 12:11:14 +0200 python-stem (1.0.1-2) unstable; urgency=low * Fix FTBFS on unhandled py3.3 on Sid. Thanks JakubWilk (Closes: #707160). * Dropped unsed dep on pysocksipy. Thanks sramacher@ (Closes: #707563). * d/rules: Added python3.3 support. * d/compat: Fix debhelper compat version. Thanks lunar@. * d/control: Drop unused dep on python-support. * d/copyright: Add missing Copyright-Format header. Thanks lunar@. -- Ulises Vitulli Sat, 11 May 2013 10:27:37 -0300 python-stem (1.0.1-1) unstable; urgency=low * Initial release (Closes: #697880) (Fixes upstream Trac#7906). -- Ulises Vitulli Thu, 10 Jan 2013 16:06:33 -0300 debian/compat0000644000000000000000000000000212231246432010363 0ustar 9 debian/source/0000755000000000000000000000000012231246432010465 5ustar debian/source/format0000644000000000000000000000001412231246432011673 0ustar 3.0 (quilt) debian/copyright0000644000000000000000000000442112231274743011127 0ustar Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ Upstream-Contact: Damian Johnson Source: https://gitweb.torproject.org/stem.git Files: docs/_static/logo.* Copyright: 2010-2013 Jakub Jankiewicz License: public-domain This files were released under Public Domain from: http://www.wpclipart.com/plants/assorted/P/plant_stem.png.html Files: docs/_static/*/*debian*.png Copyright: 2012-2013 Alexandre Moore (http://sa-ki.deviantart.com/) License: GPL-2 This package is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. . This package is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. . You should have received a copy of the GNU General Public License along with this program. If not, see . On Debian systems, the complete text of the GNU General Public License version 2 can be found in "/usr/share/common-licenses/GPL-2". Files: * Copyright: 2011-2013 Damian Johnson , 2012 Sean Robinson License: GPL-3 Files: debian/* Copyright: (C) 2013 Ulises Vitulli License: GPL-3 License: GPL-3 This package is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version. . This package is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. . You should have received a copy of the GNU General Public License along with this package; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA . On Debian systems, the complete text of the GNU General Public License can be found in `/usr/share/common-licenses/GPL-3'. debian/ChangeLog0000644000000000000000000262365012231274743010763 0ustar commit 2ef492467d13a121a76ac8ebfdc2dfbaa4b20e1d Author: Damian Johnson Date: Sun Oct 13 18:15:41 2013 -0700 Stem release 1.1.0 commit 216937c0821e27301f60dd77a1addfb286b1b3ea Merge: 242569a 6c81b10 Author: Damian Johnson Date: Sun Oct 13 17:40:51 2013 -0700 Correcting test issues with python 2.6 and 3.x Generally I just work with python 2.7, so other interpretor versions just get love prior to releases. Most of this is issues with our tests, but @lru_cache introduced a lot of legitimate regressions for python 3.x. commit 6c81b101dd69cf9f4b9170aa089d98e38220f383 Author: Damian Johnson Date: Sun Oct 13 17:25:34 2013 -0700 Python 3 compatability for mapaddress test Quite a few byte vs unicode gotchas. Also, the behavior of indexing into byte strings changed in python 3 (it provides ints rather than characters). ====================================================================== ERROR: test_mapaddress ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/atagar/Desktop/stem/test/data/python3/test/integ/control/controller.py", line 813, in test_mapaddress test.network.negotiate_socks(s, '1.2.1.2', 80) File "/home/atagar/Desktop/stem/test/data/python3/test/network.py", line 321, in negotiate_socks request = "\x04\x01" + struct.pack("!H", port) + "\x00\x00\x00\x01" + "\x00" + host + "\x00" TypeError: Can't convert 'bytes' object to str implicitly commit 992c94bc6065cee6281266df6e482a6367dd56c6 Author: Damian Johnson Date: Sun Oct 13 16:39:28 2013 -0700 Fixing AttributeError in use_directory_mirrors() Correcting a couple issues, one in the tests and another in the remote descriptor module, causing integ failures. ====================================================================== ERROR: test_use_directory_mirrors ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/atagar/Desktop/stem/test/integ/descriptor/remote.py", line 62, in test_use_directory_mirrors downloader.use_directory_mirrors() File "/home/atagar/Desktop/stem/stem/descriptor/remote.py", line 419, in use_directory_mirrors if Flag.V2DIR in desc.flags: AttributeError: 'str' object has no attribute 'flags' ====================================================================== ERROR: test_using_authorities ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/atagar/Desktop/stem/test/integ/descriptor/remote.py", line 34, in test_using_authorities for authority, (address, dirport) in stem.descriptor.remote.DIRECTORY_AUTHORITIES.items(): commit bfd5a667f150d4fef77c6a730432494ee930610b Author: Damian Johnson Date: Sun Oct 13 13:20:08 2013 -0700 Fixing python 3 testing issue with key certs Test file wasn't being opened in bytes mode, causing... ====================================================================== ERROR: test_metrics_cert ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/atagar/Desktop/stem/test/data/python3/test/integ/descriptor/networkstatus.py", line 178, in test_metrics_cert cert = next(stem.descriptor.parse_file(cert_file)) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/__init__.py", line 200, in parse_file for desc in file_parser(descriptor_file): File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/__init__.py", line 241, in _parse_metrics_file for desc in stem.descriptor.networkstatus._parse_file_key_certs(descriptor_file, validate = validate, **kwargs): File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/networkstatus.py", line 252, in _parse_file_key_certs yield stem.descriptor.networkstatus.KeyCertificate(bytes.join(b"", keycert_content), validate = validate) TypeError: sequence item 0: expected bytes, str found commit a303b137a9ada7cc9a31a57ba2dcbeae89dd83a3 Author: Damian Johnson Date: Sun Oct 13 12:55:24 2013 -0700 Fixing python 3 compatability for remote descriptor unit tests ====================================================================== ERROR: test_query_with_timeout ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/python3.3/lib/python3.3/unittest/mock.py", line 1079, in patched arg = patching.__enter__() File "/opt/python3.3/lib/python3.3/unittest/mock.py", line 1132, in __enter__ self.target = self.getter() File "/opt/python3.3/lib/python3.3/unittest/mock.py", line 1294, in getter = lambda: _importer(target) File "/opt/python3.3/lib/python3.3/unittest/mock.py", line 988, in _importer thing = __import__(import_path) ImportError: No module named 'urllib2' commit ab86475d484c9c727ba17b9cf39e1661d0e72a8a Author: Damian Johnson Date: Sun Oct 13 12:49:22 2013 -0700 Fixing python 3 issue with orphaned annotation exceptions ====================================================================== ERROR: test_annotations ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/python3.3/lib/python3.3/unittest/mock.py", line 1087, in patched return func(*args, **keywargs) File "/home/atagar/Desktop/stem/test/data/python3/test/unit/descriptor/server_descriptor.py", line 224, in test_annotations self.assertRaises(ValueError, list, desc_iter) File "/opt/python3.3/lib/python3.3/unittest/case.py", line 570, in assertRaises return context.handle('assertRaises', callableObj, args, kwargs) File "/opt/python3.3/lib/python3.3/unittest/case.py", line 135, in handle callable_obj(*args, **kwargs) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/server_descriptor.py", line 157, in _parse_file raise ValueError('Content conform to being a server descriptor:\n%s' % '\n'.join(annotations)) TypeError: sequence item 0: expected str instance, bytes found ---------------------------------------------------------------------- Ran 31 tests in 0.028s FAILED (errors=1) commit 1ed6e2b7e553414736913b3cd440843d66ccea94 Author: Damian Johnson Date: Sun Oct 13 12:40:46 2013 -0700 Hash functions for descriptor classes Ok, finally found out what this python 2 vs 3 hashing oddity is. If you define an __eq__() method in python3 then it doesn't inherit its parent's __hash__(). This makes some sense, though it's unintuitive as hell. Oh well... http://stackoverflow.com/a/1608882/1067192 commit eaa89a67ebf6339a68972e705b76a8e01677d7b9 Author: Damian Johnson Date: Sun Oct 13 12:21:06 2013 -0700 Fixing python 3 compatablity for tordnsel test ====================================================================== ERROR: test_parse_file ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/atagar/Desktop/stem/test/data/python3/test/unit/descriptor/tordnsel.py", line 55, in test_parse_file descriptors = list(_parse_file(io.BytesIO(TEST_DESC))) TypeError: 'str' does not support the buffer interface ---------------------------------------------------------------------- Ran 1 test in 0.001s commit e1bf7cbacb8797845c1bc2a778baa93d9748f4c9 Author: Damian Johnson Date: Sun Oct 13 12:10:38 2013 -0700 Fixing a couple python 3 network status document TypeErrors ====================================================================== ERROR: test_duplicate_fields ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/atagar/Desktop/stem/test/data/python3/test/unit/descriptor/networkstatus/document_v3.py", line 247, in test_duplicate_fields if not is_consensus and lines[index].startswith('dir-source'): TypeError: startswith first arg must be bytes or a tuple of bytes, not str ====================================================================== ERROR: test_misordered_fields ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/atagar/Desktop/stem/test/data/python3/test/unit/descriptor/networkstatus/document_v3.py", line 225, in test_misordered_fields elif not is_consensus and lines[index].startswith('dir-source'): TypeError: startswith first arg must be bytes or a tuple of bytes, not str ---------------------------------------------------------------------- Ran 35 tests in 0.234s commit efe5fbdd56b28b3f76de144e71901c0c75ebe5ac Author: Damian Johnson Date: Sat Oct 12 20:14:59 2013 -0700 Caching hash values Well, even if we can't use @lru_cache we can still do it the old fashioned way. :P commit d32d4ab45d27776385036b1a57d9329f270354fc Author: Damian Johnson Date: Sat Oct 12 20:11:23 2013 -0700 Dropping @lru_cache for Version hashing The @lru_cache calls hash() so if ever called this would infinitely recurse. commit 208c03cded57507f835265d7e4d2b4677d0e30e5 Author: Damian Johnson Date: Sat Oct 12 20:07:52 2013 -0700 Fixing python 3 issue concerning exit policy compression Oops, forgot about the byte vs unicode differences when doing compression for the ExitPolicy. commit 4ef5ee1ee8fbecabeaee872f2681cdda1fb86584 Author: Damian Johnson Date: Sat Oct 12 20:18:12 2013 -0700 Adding hash functions to exit policy classes Huh, didn't expect that. To work the @lru_cache requires that all arguments (including the class in the ase of methods) are hashable. This makes perfectly sense, after all the cache is a 'argument => cached value' dictionary. Under python 2.x all seemed to be well, but under python 3.x the @lru_cache complained that our classes weren't hashable (which... well, they aren't). Why, then, did it work under python 2.x? Turns out there's a subtle difference where our object parent provides a __hash__ method for our id, but in python 3.x it's a little different (object still does, but for reasons I haven't been able to figure out it doesn't for our classes). The ExitPolicyRule can certainly be hashable. The ExitPolicy, however, is a lot trickier due to its lazy loading of rules. For now just opting for the same behavior as python 2.x and using the address. commit 677ee1570cc1378b8abadb4071627ff53510cb8a Author: Damian Johnson Date: Sat Oct 12 19:23:04 2013 -0700 Python 3 lru_cache isn't a collections annotation Oops, it's in functools, not collections. commit 1dfa353ecfbef39ee606231e745b9b6318fd3a31 Author: Damian Johnson Date: Sat Oct 12 17:35:55 2013 -0700 Using @lru_cache for exit policy's str() Oops, missed one of our instances of manual caching. commit e89c829fb75331fdc0d5aa03475c751371cb6c7e Author: Damian Johnson Date: Sat Oct 12 16:57:50 2013 -0700 Using python3's unittest.mock module when available Guess it's been a while since I last tried to test under python3. From python 3.3 on up the Mock library is built into the unittest module. Accounting for this so we can test under python3 again. commit d09bd8b365819f4ea0d0a7d6b782b1021dba91c1 Author: Damian Johnson Date: Sat Oct 12 16:37:02 2013 -0700 Fixing python3 issue related to unicode exit policies Our recent exit policy caching caused a regression under python3... Traceback (most recent call last): File "./test/data/python3/run_tests.py", line 25, in import test.runner File "/home/atagar/Desktop/stem/test/data/python3/test/runner.py", line 51, in import stem.connection File "/home/atagar/Desktop/stem/test/data/python3/stem/connection.py", line 109, in import stem.control File "/home/atagar/Desktop/stem/test/data/python3/stem/control.py", line 145, in import stem.descriptor.microdescriptor File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/__init__.py", line 546, in import stem.descriptor.server_descriptor File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/server_descriptor.py", line 91, in REJECT_ALL_POLICY = stem.exit_policy.ExitPolicy("reject *:*") File "/home/atagar/Desktop/stem/test/data/python3/stem/exit_policy.py", line 164, in __init__ self._input_rules = zlib.compress(','.join(rules)) TypeError: 'str' does not support the buffer interface commit c10eef255f0948bcc3cd86eab4221e2b335615eb Author: Damian Johnson Date: Sat Oct 12 17:13:56 2013 -0700 Fixing python 2.6 compatability issue with lru_cache The recepie is using set syntax only available in python 2.7 and above... Traceback (most recent call last): File "./run_tests.py", line 18, in import stem.prereq File "/home/atagar/Desktop/stem/stem/__init__.py", line 420, in import stem.util.enum File "/home/atagar/Desktop/stem/stem/util/enum.py", line 43, in import stem.util.str_tools File "/home/atagar/Desktop/stem/stem/util/str_tools.py", line 21, in import stem.prereq File "/home/atagar/Desktop/stem/stem/prereq.py", line 29, in from stem.util.lru_cache import lru_cache File "/home/atagar/Desktop/stem/stem/util/lru_cache.py", line 37 fasttypes = {int, str, frozenset, type(None)}, ^ SyntaxError: invalid syntax commit 242569af36ebb1f30a5d2629254b55320cfd5298 Author: Damian Johnson Date: Sat Oct 12 13:11:11 2013 -0700 Replacing manual caching with @lru_cache Replacing instances where we do... def get_stuff(): if self._stuff is None: self._stuff = ... calculated stuff... return self._stuff ... with a @lru_cache(). commit 94b4924a3970fe3877b276ad2d07ccd149ed8d3a Author: Damian Johnson Date: Thu Oct 10 08:51:59 2013 -0700 Revert "Chaching version comparison and hash function" Wow, shame on me for not even running unit tests. This causes version comparisons to infinitely recurse - hot. commit e28032aebb14043c2149fe949ef49ddb82083c21 Author: Damian Johnson Date: Wed Oct 9 08:55:00 2013 -0700 Mocking pydocs referenced functionality we've removed commit 281db911a5912836ceb63247b55d8f8300b639ec Author: Damian Johnson Date: Tue Oct 8 09:11:48 2013 -0700 Chaching version comparison and hash function Versions are intended to be read-only objects, so we might as well add caching for these methods. commit ba9799e20e8046ef1316616a3c665f8e4558665a Merge: d1218e0 1920baf Author: Damian Johnson Date: Tue Oct 8 00:42:46 2013 -0700 Reducing memory usage of descriptor content Handful of improvements to reduce the memory usage of stem's descriptor content (with a special focus on server descriptors). This drops memory usage by roughly 20% without having a sizable impact on runtime. Script used for testing: import resource import time from stem.descriptor import reader print "memory (initial): %s" % resource.getrusage(resource.RUSAGE_SELF).ru_maxrss start_time = time.time() with reader.DescriptorReader(['/home/atagar/.tor/cached-descriptors']) as descriptor_reader: server_descriptors = list(descriptor_reader) print "memory (after read): %s" % resource.getrusage(resource.RUSAGE_SELF).ru_maxrss print "read time: %0.2f seconds" % (time.time() - start_time) Before: % python scratch.py memory (initial): 7468 memory (after read): 50572 read time: 6.40 seconds After: % python scratch.py memory (initial): 7532 memory (after read): 40816 read time: 6.31 seconds commit 1920baff1ad557a2a556dcb0cb46609f50f3cde1 Author: Damian Johnson Date: Tue Oct 8 00:37:11 2013 -0700 Compressing unparsed exit policy content Exit policies constitute a rather substantial chunch of server descriptors. We already optimize our runtime by lazily parsing policies on demand. On reflection though, we can improve our memory usage quite a bit too by compressing the unparsed content. This drops our memory usage by roughly 6% without impacting runtime. commit 2ce003ef33c8693234f7dce562fdd3cff4e37e07 Author: Damian Johnson Date: Tue Oct 8 00:15:51 2013 -0700 Caching parsed tor versions Reusing parsed tor versions saves us an impressive 3.7% memory usage without any noticable impact on parsing times. commit dbee38d1b95d943435cbb6bc989a9a5024b4e8e8 Author: Damian Johnson Date: Mon Oct 7 23:09:04 2013 -0700 Referencing a single reject-all policy Oddly, the only exit policy I can find with greater than five occurances is the reject-all policy. Maybe a scripting bug. Reguardless, referencing a single reject-all instance saves us a little memory (1.3 MB with the present consensus, which is 1.6%). commit a0ce8664bc5758030c912456fcf2ff33a619373f Author: Damian Johnson Date: Mon Oct 7 22:11:44 2013 -0700 Storing a single IPv6 default exit policy Each server descriptor constructed its own default value for the IPv6 exit policy, which was foolish. ExitPolicy instances are read-only instances, so all descriptors might as well use the same default. This saves roughly 2.3 MB of memory when downloading all of the current server descriptors (or roughly 3%). commit 19cca3a04b9b02e94d22ebb877af3ca2ca9ad5e4 Author: Damian Johnson Date: Mon Oct 7 21:14:00 2013 -0700 Using @lru_cache for exit policies The ExitPolicy and ExitPolicyRule classes are a perfect fit for @lru_cache. They're read-only classes that already do a fair bit of caching. The annotation lets us avoid doing this ourselves. commit d1218e00feeb1d4afc93a105b9a99d38a4ffa841 Author: Damian Johnson Date: Sun Oct 6 23:15:05 2013 -0700 Switching to the https endpoint of www.atagar.com Now that we have SSL support there's little reason to point to the http endpoint. commit d9fbdc968479b0d413d1ff875ca19c855e7ce2ae Author: Damian Johnson Date: Sun Oct 6 17:32:10 2013 -0700 Adding python 2.x backport of @lru_cache Python 3.2 added a memoization annotation to python's functools module. This is very, very handy, allowing us to avoid the caching boilerplate I do way too often... def get_foo(): if self._foo is None: ... stuff to calculate self._foo... return self._foo With a memoization function this becomes... @lru_cache() def get_foo(): ... stuff to calculate self._foo... This is a MIT licensed backport from... http://code.activestate.com/recipes/578078-py26-and-py30-backport-of-python-33s-lru-cache/ Looking forward to when we require python 3.2 so we can use the builtin! commit e60a033941c63885cb349d3bc181bbbf66227acf Author: Damian Johnson Date: Wed Oct 2 09:19:38 2013 -0700 Tests can fail with TypeError when unable to start tor process Traceback (most recent call last): File "./run_tests.py", line 404, in main() File "./run_tests.py", line 245, in main println(exc, ERROR) File "/home/atagar/Desktop/stem/test/output.py", line 59, in println msg = term.format(msg, *attr) File "/home/atagar/Desktop/stem/stem/util/term.py", line 96, in format return (CSI % ";".join(encodings)) + msg + RESET TypeError: cannot concatenate 'str' and 'UnsupportedOperation' objects commit bcd849f0cbf3e5a823049dcea51f9158baf897b9 Author: Damian Johnson Date: Wed Oct 2 07:32:11 2013 -0700 Adding a hash method to our Version class Our Version class lacked a __hash__() method, causing it to behave unintuitively in dictionaries and sets... >>> from stem.version import Version >>> foo = set([Version('0.2.4.9-alpha')]) >>> bar = set([Version('0.2.4.9-alpha')]) >>> foo.difference(bar) set([]) This is because python opts for the object's identity when a hash method does not exist. commit a5f9e9cd5b4662bc5d34e19621fa9667cf71a07d Author: Damian Johnson Date: Mon Sep 30 14:39:29 2013 -0700 Halo for windrose icon Adding a blue halo for the center of the windrose. This doesn't address the main thing I dislike about the icon, which is the pixelation of the black lines, but still it makes it look a bit nicer. I've tried several things to address the black line pixilation (blur, thickening, etc) but this is the best improvement I've yet found. commit 4866bd336cd70eedcd00a685df26737fd7c37939 Author: Damian Johnson Date: Sun Sep 29 20:07:33 2013 -0700 Adding utilities tutorial to the menu Oops, forgot one of the spots we need to add new pages... commit b0be16b7e93f56e6189cea6b0b991cded1db6a02 Author: Damian Johnson Date: Sun Sep 29 20:01:16 2013 -0700 Adding connection resolution tutorial Introducing a new tutorial section called 'East of the Sun & West of the Moon' (... I love that fairytale) that introduces users to our utility modules. Presently this just has connection resolution, a spiffy new feature I added last weekend. commit ec24345427ac804640a425bbb900bf41aae3be83 Author: Damian Johnson Date: Sat Sep 28 16:49:25 2013 -0700 Linking to the DocTor repository Oops, our example page was still linking to my tor-utils repository rather than DocTor's. commit 276fb65ef4dbf0536be91e3fcb1912363d5ada73 Author: Damian Johnson Date: Mon Sep 23 12:03:25 2013 -0700 Only report connection resolvers that are in our PATH Having get_system_resolvers() check that resolvers are in our PATH before returning them. commit 7c0002d40db176d02c6a57e400ec53b5fb7a63a4 Author: Damian Johnson Date: Mon Sep 23 00:28:01 2013 -0700 Proc unit tests didn't reflect protocol addition Oops, our unit tests didn't account for proc now providing the protocol. Caught by our jenkins tests. commit 1082349b59eb6ecd89eca1a1f2aa2ab1df007315 Author: Damian Johnson Date: Sun Sep 22 23:38:17 2013 -0700 Connection resolution support Adding long overdue support for process connection resolution... https://trac.torproject.org/7910 This is a highly popular capability of arm, and stem's counterpart for it is quite a bit cleaner (with tests!). That said, this still doesn't get around tor's annoying DisableDebuggerAttachment feature which screws up proc permissions. That's something I'll need to figure out before our next arm or stem release... commit 87b0d512e2b83a2dcced709e33128eaed10cc439 Author: Damian Johnson Date: Sun Sep 22 12:25:00 2013 -0700 Adding function for getting the system's resolvers Adding a get_system_resolvers() function that provides the connection resolvers that are likely to work on a given platform. commit aee7de80db712ab1003eb6d8ac52e78d5f0f18cc Author: Damian Johnson Date: Sat Sep 21 17:46:02 2013 -0700 Adding a is_private_address() function to stem.util.connection Helper function for determining if an IPv4 address belongs to a private range or not. commit f40c912ecb8fdb542a7bd12e7a440e5301ea9798 Author: Damian Johnson Date: Sat Sep 21 17:41:13 2013 -0700 Adding a get_vote() method to the DescriptorDownloader Stem is frequently being used for answering questions concerning directory authorities, so adding a method to more intuitively get their votes. commit 1d2fe8a218768eb5a1713cd2daa6d54df0d6c7ca Author: Damian Johnson Date: Sat Sep 21 12:05:25 2013 -0700 Disable caching of HiddenServiceOptions Stem's controller has pretty straight forward caching for tor's configuration options. If you call 'GETCONF foo' then the value of foo is cached until you call either 'SETCONF foo' or 'RESETCONF foo'. In general this is all well and good, but HiddenServiceOptions behave in a different fashion. With those the caller might call 'SETCONF HiddenServiceDir' and expect 'GETCONF HiddenServiceOptions' to change. As such, opting out of caching hidden service options. This was a bug uncovered by wayzard on... http://stackoverflow.com/questions/18777837/tors-stem-module-python-loading-but-not-unloading-config https://trac.torproject.org/projects/tor/ticket/9792 Prior to this fix... >>> from stem.control import Controller >>> controller = Controller.from_port() >>> controller.authenticate() >>> controller.get_conf_map('HiddenServiceOptions') {'HiddenServiceOptions': []} >>> controller.set_options([('HiddenServiceDir','/tmp/my_hidden_service'), ('HiddenServicePort', '1236 127.0.0.1:1236')]) >>> controller.get_conf_map('HiddenServiceOptions') {'HiddenServiceOptions': []} Now it behaves as expected... >>> from stem.control import Controller >>> controller = Controller.from_port() >>> controller.authenticate() >>> controller.get_conf_map('HiddenServiceOptions') {'HiddenServiceOptions': []} >>> controller.set_options([('HiddenServiceDir','/tmp/my_hidden_service'), ('HiddenServicePort', '1236 127.0.0.1:1236')]) >>> controller.get_conf_map('HiddenServiceOptions') {'HiddenServicePort': ['1236 127.0.0.1:1236'], 'HiddenServiceDir': ['/tmp/my_hidden_service']} commit cfec958ddec6c1b104def1fdb40cacbb4e73b79d Author: Damian Johnson Date: Sat Sep 21 11:32:03 2013 -0700 Adding default and constraints for Support022HiddenServices Reflecting constraints for a new param in network status documents... https://gitweb.torproject.org/torspec.git/commitdiff/6c974c5 commit e2cff0be96640b584fa1e7044878891ff4bb69b0 Author: Damian Johnson Date: Sat Sep 14 17:25:09 2013 -0700 Dropping config's _raw_contents reference On reflection we never do anything with the _raw_contents. In fact, we don't even have getter methods for it, and never use it within the Config class itself. Dropping it to both save a little memory and make it easier to purge values. commit fb3907913b09770ed6de5ff8914553e424fa8f47 Author: Damian Johnson Date: Sat Sep 14 13:53:33 2013 -0700 Logging tor's output during integ test runs Saving tor's logging output to disk to help with troubleshooting issues. commit 00b97f1d711d0d713d03a9ace72ba42cf0e3003e Author: Damian Johnson Date: Mon Sep 9 12:19:04 2013 -0700 Changing 'DocTor' spelling Karsten has been pretty consistent about calling the project 'DocTor' rather than 'Doctor' so I should do the same. commit 7679276642ddf3a64aa8473f2c6de68549aea6df Author: Damian Johnson Date: Mon Sep 9 12:16:21 2013 -0700 Adding TorPS to examples page commit 032df08a5591afaf810dabdbca9177fcc528a7ee Author: Damian Johnson Date: Mon Sep 2 13:16:00 2013 -0700 Adding stem.util.system.get_name_by_pid() Function for getting the command running under a given pid. This is the inverse of get_pid_by_name(). commit 26dadd28e70efc5fac826b62bf4f4b35b7acadeb Author: Damian Johnson Date: Mon Sep 2 09:30:34 2013 -0700 Replacing DIRECTORY_AUTHORITIES constant with get_authorities() On reflection a DIRECTORY_AUTHORITIES constant is not something I want to vend in our next stem release. That would make it tricky to be flexible about where we get that information. For instance, we might want to get this from tor itself (if that ever becomes an option), or possibly a config file rather than hardcoded dict. commit e5bfee6dc5ec66f46560e2b9bae68340569e1d77 Author: Damian Johnson Date: Sun Sep 1 16:24:48 2013 -0700 Typo correction in tutorial Caught and fixed by irregulator. commit ec0c7eb0c3b96fc50ddc117bd83f4f6477c13f4c Author: Damian Johnson Date: Sun Sep 1 11:52:21 2013 -0700 Minor improvements for connection examples Changes include... * more conventional example using the 'with' keyword * disconnecting the controller afterward * corrected indentation commit 53068b2f3066df1a01be5f13a3578263e21fac54 Author: Damian Johnson Date: Sun Sep 1 10:23:49 2013 -0700 FAQ entry for 'How do I connect to Tor?' While code reviewing ra's RTT Prober work I realized that our site didn't explain the various ways a user can get a Controller. Adding a FAQ entry with examples that explains it. commit 3288f61da61932664c677619fe303d1d076dbe83 Author: Damian Johnson Date: Fri Aug 30 14:14:22 2013 -0700 Fixing case for new faq entry Oops, FAQ entry titles shouldn't be all caps. commit fdd84baa1edd30cf11d47ad7b5d1c490024c2990 Author: Damian Johnson Date: Fri Aug 30 10:34:00 2013 -0700 FAQ entry for alternate tor controllers Entry for our FAQ linking to other controller libraries. This came out of the following tor-dev@ thread... https://lists.torproject.org/pipermail/tor-dev/2013-August/005340.html commit e34b8c943a7e59ac8d8cbe9a77424b44e60e35d5 Author: Damian Johnson Date: Fri Aug 30 08:58:36 2013 -0700 Minor typo correction Little typo I spotted while answering... http://stackoverflow.com/questions/18372615/tors-stem-module-python-failing commit a7ec700c1b366b5cd96ba894916ad0a6e6fa0b4c Author: Damian Johnson Date: Thu Aug 29 10:13:48 2013 -0700 Explaining python's 'with' keyword We've had a couple stem newcomers that saw the 'with' keyword in our examples then misused it in their scripts. Adding a little tutorial on the keyword and the common mistake they've been making. commit 303b4dfdac10ecb6c348dafbb08f27cae9799347 Author: Damian Johnson Date: Tue Aug 27 22:37:56 2013 -0700 Adding tor directory authority information To work our remote descriptor fetching module needed the addresses and dirports of voting authorities. However, on reflection it's useful if we reflect the rest of the authority information from tor's config.c as well. For instance, the v3ident is handy if our users want to query authority key certificates. commit 847d2377846bf1a64881934e74c10193ec3fb130 Author: Damian Johnson Date: Mon Aug 26 21:27:50 2013 -0700 Using remote module for last descriptor tutorial Using the remote descriptor fetching module for the last descriptor tutorial rather than the controller. Also updating our unit tests for the tutorials. commit d5c02be90b3a9d200773e4e128303bd4cee062a8 Author: Damian Johnson Date: Mon Aug 26 09:22:50 2013 -0700 Adding remote descriptor fetching to our tutorials Replacing the "Where can I get the current descriptors?" subsection with an example using the remote descriptor fetching module. This is the simplest way for our users to get descriptor data without the headache of piggybacking on a tor instance. commit 0416b071d4f7b8518ed4e8d36c0afeb8b01e327f Author: Damian Johnson Date: Sun Aug 25 18:12:45 2013 -0700 Adding doctor to examples page Swapping our 'Descriptor Monitors' sub-entry with an entry with Doctor. commit f1e771a37828f85e3bd8631ad7234f50b0efac6e Author: Damian Johnson Date: Sun Aug 25 11:45:08 2013 -0700 Support for TRANSPORT_LAUNCHED events Adding support for George's new TRANSPORT_LAUNCHED events... https://gitweb.torproject.org/torspec.git/commitdiff/48f6dd0 commit 4a84b47b6ebbd3a740884fd8e24f499b79582028 Author: Damian Johnson Date: Sat Aug 24 10:57:59 2013 -0700 Adding support for extrainfo descriptor's 'bridge-ip-transports' Support for a new bridge-specific extrainfo descriptor field... https://gitweb.torproject.org/torspec.git/commitdiff/a01bb8e8e These are pretty much identical to bridge-ip-versions lines. Like those this is part of the ExtraInfoDescriptor class rather than BridgeExtraInfoDescriptor because the spec doesn't prohibit its appearance in public relays (though it doesn't really make sense there). commit c076038e82fcd5f5188ec7f889c9c9dc5d5eafd3 Author: Damian Johnson Date: Sat Aug 24 10:45:14 2013 -0700 Handling bridge-ip-versions as a general extrainfo field In commit 0d01ad6 I added support for the 'bridge-ip-versions' lines. These are only relevant to bridge descriptors, and ineed don't presently appear in the extrainfo consensus. However, that once could be said about the 'bridge-ips' field too. It's wrong for bridge attributes to appear on public relays, but there's nothing in the spec prohibiting it. Hence moving this attribute's handler to the common ExtraInfoDescriptor parent class. commit 003bbd7e8f0ad01300d4b0365302b31a921da37c Author: Damian Johnson Date: Fri Aug 23 09:08:18 2013 -0700 Assigning stem's copyright to the Tor Project Revising the copyright notices throughout stem so both the Tor Project and I have copyright. This has a couple advantages... 1. If I get nailed by the bus or otherwise somehow disappear then I want the Tor Project to have copyright. This will make life easier for any future stem maintainer by allowing them to share code with our other (mostly BSD licensed) projects. 2. Contributors are understandably antsy about giving copyright to an individual rather than an organization. Thus far I've been able to largely sidestep those concerns by asking for contributions to be in the public domain, but there was one occasion where that was a no-go... https://trac.torproject.org/projects/tor/ticket/7566 Hopefully assigning copyright to the Tor Project is something that'll make everyone happy. I've contacted Sean and he's fine with assigning the copyright for his stem contrutions to the Tor Project... Yes, I am still interested in and willing to assign my existing and future contributions (unless I explicitly say otherwise) to Stem to the Tor Project. If the previous sentence is not legally binding enough, let me know. Tweaked the FAQ entry a bit to reflect this change. commit d092810bb9810f4dfeb244b71ae10bd6031b52b3 Author: Damian Johnson Date: Thu Aug 22 09:30:22 2013 -0700 Adding TorDNSEL docs to the site commit d7833f31c392623f8dad5a30e7b16713cff510aa Author: Damian Johnson Date: Thu Aug 22 09:20:20 2013 -0700 Minor formatting changes for DNSEL addition Nothing functional, just rearranging a few things to satisfy my OCD. :) commit 4631228400e0cca43d8c7ba514c40cbcbf2bda34 Author: Arlo Breault Date: Mon Aug 19 22:49:40 2013 -0700 Parse exit list entry. Published by DNSEL or TorBEL to indicate what ip address exit relay X had at timestamp Y. See #8255 commit 4863c2281c26660a249378e65354a00e406fffbc Author: Damian Johnson Date: Mon Aug 19 18:15:47 2013 -0700 Validating that votes only have a single authority entry Vote documents should only contain an entry from the authority that issued it. I was checking this in my consensus-checker monitor but Karsten made the good point that the check belongs in stem. commit bba4069fdf70d422e4d6f4cbd8a50c0be6747732 Author: Damian Johnson Date: Sat Aug 17 12:48:49 2013 -0700 Don't fail tests upon lingering thread The chief cause of our present jenkins failures are lingering threads... https://trac.torproject.org/projects/tor/ticket/8896 This is certainly an problem. It indicates either a bug with stem closing threads, or that we aren't closing a controller in the tests. However, I'm tired of the false alarms from the jenkins tests, besides which this is a very minor issue. commit 5be9904711cc7f15f438cad9456dfcc31309c73d Author: Damian Johnson Date: Fri Aug 16 08:41:45 2013 -0700 Propagating additional descriptor keyword arguments Adding a 'kwargs' argument to all methods for fetching descriptors. This argument is passed along to the descriptor contructor. I just ran into a need for this because the NetworkStatusDocumentV3 class has a 'default_params' argument that I couldn't access through parse_file(). We'll likely be adding more constructor args in the future so this will help to make those... well, useful. commit e9f23a1996356bef3edf22c143898c1f78498486 Author: Arlo Breault Date: Mon Aug 12 23:36:59 2013 -0700 Typo in FAQ. commit ced3c4883056c23fa43d3975850880841e1677c7 Author: Arlo Breault Date: Mon Aug 12 22:13:05 2013 -0700 FetchDirInfoExtraEarly requires FetchDirInfoEarly. Tor complains, [warn] Failed to parse/validate config: FetchDirInfoExtraEarly requires that you also set FetchDirInfoEarly commit bb3de90a82ff927c5005fbadce3b2ea975a15aae Author: Damian Johnson Date: Sun Aug 11 13:49:30 2013 -0700 Dropping document_handler from get_consensus() Oops, the document_handler is already a query parameter. There's no benefit to having a default value for it in get_consensus(), and that had the drawback of preventing us from asserting a default value. commit 4fc65393374ed32a9c67c11758b0c5b94ed6f5f5 Author: Damian Johnson Date: Wed Aug 7 09:49:25 2013 -0700 Issuing DNS lookups via tor in client usage example Suggestion from Julian to issue DNS lookups via tor. This is based on... http://www.talesofacoldadmin.com/2012/11/13/proxying-dns-with-python.html commit e0956015a1589dba607dc41ffa486e20c00f863b Author: Damian Johnson Date: Wed Aug 7 09:09:14 2013 -0700 Correcting definition of a 'dirty circuit' A circuit becomes 'dirty' once it has serviced traffic, not when it expires. Thanks to Roger for the terminology correction. commit 0a79385a89fabd14721ce3fd0e4a49c63c1c13da Author: Damian Johnson Date: Wed Aug 7 09:00:21 2013 -0700 Dropping routers attribute from get_network_status_document_v2() We only have a single test using this mocking module function. The 'routers' attribute was both unused and unimplemented so simply dropping it. We can implement it later if we add a test that needs it. commit ccf03a04fcc160a2b87fd0361e7b4c31c562f1b1 Author: Damian Johnson Date: Sun Aug 4 16:49:14 2013 -0700 Additions for our examples page Adding Robert's RTT Prober and updating the entry for my descriptor monitors. commit e78f1b74efac02ed09f963c65a652b87ea80f5f2 Author: Damian Johnson Date: Sun Aug 4 16:25:09 2013 -0700 Having get_server_descriptor() warn when using microdescriptors Tor presently gets microdescriptors rather than server descriptors by default. More than one person has been confused by this, so adding a warning to our Controller when server descriptors are unavailable... >>> controller.get_server_descriptor('9695DFC35FFEB861329B9F1AB04C46397020CE31') Traceback (most recent call last): File "", line 1, in File "stem/control.py", line 1207, in get_server_descriptor raise ValueError(SERVER_DESCRIPTORS_UNSUPPORTED) ValueError: Tor is presently not configured to retrieve server descriptors. As of Tor version 0.2.3.25 it downloads microdescriptors instead unless you set 'UseMicrodescriptors 0' in your torrc. This was suggested by Robert in... https://lists.torproject.org/pipermail/tor-dev/2013-July/005196.html commit e7c165ca7122a12fd5baade33221887176de3fe7 Author: Damian Johnson Date: Sun Aug 4 12:04:22 2013 -0700 Adding a 'content' attribute to the Query class Including a Query attribute so the caller can get our raw descriptor content. I'm actually doing this to simplify the run() method's usage. It had provided an iterator so... query = Query(my_resource) print list(query) # this would print the ist of descriptors print list(query) # this would be an empty list The reason the second call is an empty list is because we already iterated over the query. This is confusing, especially since... query = Query(my_resource) query.run(True) print list(query) # also an empty list ... due to run() returning a list under the covers. By making run() provide a fresh iterator each time I also now had the downloaded content handy so making it public made sense (it'll commonly be wanted by our callers). commit 5c5c59f1e7b16ed1f7a5131ec072e5646d086e44 Author: Damian Johnson Date: Sun Aug 4 11:38:28 2013 -0700 Adding a 'block' argument to the Query class I'm finding it to be pretty common to run... query = Query(my_resource) query.run(True) if not query.error: # do something else: # report error Adding an argument to the Query constructor so we can drop the following run(). commit faae224eb833d7612e2779e34d0f1010f8cb2d08 Author: Damian Johnson Date: Sun Aug 4 11:31:05 2013 -0700 Query's run() method didn't block Well, that was confusing as hell. Say you wrote the following script... query = stem.descriptor.remote.Query( '/tor/server/all.z', timeout = 60, ) query.run(True) if not query.error: print "Results downloaded: %i" % len(list(query)) else: print "Got an error: %s" % query.error You'd expect run() to block, right? You should - that's what it was documented as doing. Instead it returned an iterator and we don't block until the list(query) call. This means that we would *never* report an error. This has been causing stacktraces for my monitors every time it encountered an error. commit cf0a4b2f7cff17ff6f819c91e35b2a86ffcb6a81 Author: Damian Johnson Date: Sun Aug 4 11:21:22 2013 -0700 Defaulting Query's fall_back_to_authority to False When a user opts to use the Query class direcly they're doing so to use specific endpoints. We'll probably cause more confusion then help by falling back to the authorities here. We'll still fall back by default in the downloader, though. This also includes a tweak so users can override the downloader's endpoints. commit 539cd19919a3f5fe0e5ab39259441f736c1a8579 Author: Damian Johnson Date: Sun Jul 28 13:34:23 2013 -0700 Style correction for event listing tutorial Oops, minor PEP8 correction for one of our tutorial examples. Not sure why I did camel case for this function... commit b7ecd5951f1b3c1f47f4249c4d4822c00aa041d0 Author: Damian Johnson Date: Sun Jul 28 13:15:24 2013 -0700 Adding a couple missing changelog entries Entries for the remote descriptor fetching module and frontpage UI overhaul. commit 7e786104dc1704d8a378918de08474c23c1d47e0 Author: Damian Johnson Date: Sun Jul 28 13:05:07 2013 -0700 Using a red download icon I opted for ColorCons' 'smoke' colored download icon because I thought it would be more subtle. However, all the other section icons are pretty bright primary colors so that just made it stand out more. Opting for red so it'll match the bug tracker icon. Also very minor alignment adjustments for the changelog and tutorial icons. commit 8d39e338bedc1e6c67338feff5176c697e4e08bd Author: Damian Johnson Date: Sat Jul 27 19:39:44 2013 -0700 Replacing buttons for our frontpage As a few people have noted our present blue buttons on stem's frontpage looks pretty aweful. It clashes with the site's color scheme and is pretty in-your-face. Swapping to another color scheme that's hopefully less offensive... http://www.atagar.com/transfer/stem_frontpage/before.png http://www.atagar.com/transfer/stem_frontpage/after.png commit 5eda329ed6a288575f61f26ac2a457299c2f9d04 Author: Damian Johnson Date: Fri Jul 26 09:17:07 2013 -0700 Inverted conditional for reporting bugs for password exceptions Issue caught by str4d while porting stem to I2P. commit d7b68d01aabc6fdcd544a479cfa2cac377eaddeb Author: Damian Johnson Date: Fri Jul 26 08:13:58 2013 -0700 Having use_directory_mirrors() return the consensus Originally I was planning an on-disk cache so calling get_consensus() after use_directory_mirrors() would result in an on-disk read. However, that has been put on hold for now so having use_directory_mirrors() return the consensus it fetches in order to get the mirrors. We unfortunately already need to read the full consensus into memory before processing it due to how urllib2 works, so this shouldn't cause a memory usage hit. We might someday want to change this though if we get a constant time download/parsing scheme. commit 7f050ebfa475aeb39f6b1b4a0f7ad53620931646 Merge: b6c23b0 713b046 Author: Damian Johnson Date: Sun Jul 21 18:39:17 2013 -0700 Remote descriptor fetching module What started as a tor-dev@ thread has finally accumulated in a surprisingly simple module. This provides the ability to download tor descriptors from directory mirrors or authorities, and is made of up of two classes... * Query The core of the module. This is a request to download a given descriptor resource from a pool of authorities and/or mirrors. This retries failed requests and can be iterated over to get the descriptor results. * DescriptorDownloader A configurable class for issuing Queries. This provides only two things: simple methods to get common descriptor types, and an easy way of specifying default Query parameters. https://lists.torproject.org/pipermail/tor-dev/2013-May/004924.html https://trac.torproject.org/8257 commit 713b04632e6ca1120c6d5c2661206fd84c01904b Author: Damian Johnson Date: Sun Jul 21 18:37:48 2013 -0700 Adding remote descriptor sphinx docs to site Tidying up our pydocs and including it in our site. commit 4d122b1a417fe318f0bb3d169688d25f3e749754 Author: Damian Johnson Date: Sun Jul 21 17:10:26 2013 -0700 Implementing a get_key_certificates() method Method for fetching key certificates for the authorities. This included a little work so parse_file() could return multiple certificates when they're concatenated together. commit e95e00dd6431001000a79e2bfc009a7f3abbde80 Author: Damian Johnson Date: Sun Jul 21 16:31:00 2013 -0700 Adding a query_args parameter to all downloader methods Simple tweak in case callers want to do something dynamic (such as varying the timeout with each request). commit 6f12acc9e54842371f52203550a6ef2f9cbfb835 Author: Damian Johnson Date: Sun Jul 21 16:25:58 2013 -0700 Simplifying DescriptorDownloader class The DescriptorDownloader is a very, very simple class that simply issues Queries on your behalf. Why should it pretend to be anything else? Dropping most of its attributes (and with it a lot of copy-and-pasted pydocs). Also making the query() method public. commit 90ec648ed075245f6ed2bae42dd4c6968e15fab1 Author: Damian Johnson Date: Sun Jul 21 15:57:47 2013 -0700 Support for compressed descriptors Ummm... well, that was easy. When I was initially looking into tor's directory responses I had trouble getting valid compressed results. Kostas suspected that the responses might have been truncated. Oh well, maybe a wget thing. In either case adding compression support to our module was trivial. commit d5b3ec93f44de01b21b27264e761fe8f09ec8012 Author: Damian Johnson Date: Sun Jul 21 10:45:56 2013 -0700 Fixing "cannot unmarshal code" errors Damnit python, your import scheme is stupidly confusing. The descriptor's __init__ module has a circular dependency with its contents. This is because the parse_file() function calls the constituent modules, while those modules need the Descriptor class from __init__. So far so good. Only trouble is that python's support for circular dependencies sucks. To address this I did lazy imports in __init__, so we imported within the parse_file() function. On the surface this seemed to work. All the tests certainly passed. The trouble is that this style of python import is buggy as hell. Turns out that lazy imports leave the module in question in a unexecutable state so this *only* works if you've also imported the module another time during the interpretor execution. Our tests did this, hence passing tests. I first encuntered "cannot unmarshal code" while writing the remote descritpor tests (both unit and integ). I was content to hack around this with superfluous import statements while this only manifested within the tests, but now I'm seeing it during general usage too... >>> from stem.descriptor.remote import DescriptorDownloader >>> list(DescriptorDownloader().get_microdescriptors('jzcx+1fHsi47Tu+vQIcyItgn4lKs6aKnFshQ0lZ2JTg')) Traceback (most recent call last): File "", line 1, in File "stem/descriptor/remote.py", line 311, in __iter__ for desc in self.run(True): File "stem/descriptor/remote.py", line 300, in run for desc in self._results: File "stem/descriptor/__init__.py", line 154, in parse_file import stem.descriptor.server_descriptor RuntimeError: cannot unmarshal code objects in restricted execution mode Joy. After much head scratching and forum reading it sounds like there's something magical about 'from' imports so switching the descriptor modules to that, and moving the __init__ imports to the end. I'm not entirely clear on the magic going on here, but its elmiminated the errors. commit 60ad230d88b383a259c531649fdba78d0acd6493 Author: Damian Johnson Date: Sun Jul 21 09:03:39 2013 -0700 Expanding remote descriptor pydocs Adding documentation and examples for the remote descriptor API. This includes some minor API tweaks and a missing import statement. commit 25951d4f12a16a1fa6e42ea6b7479959d636139b Author: Damian Johnson Date: Sat Jul 20 10:00:04 2013 -0700 Making descriptor validation optional Adding a 'validate' argument that can be passed thorough when parsing descriptors. commit 749c8d37fd54ea6e41dfca04495c35b47a755f6a Author: Damian Johnson Date: Sat Jul 20 09:52:07 2013 -0700 Adding a broken get_microdescriptors() method Huh. Not really sure why this isn't working. Adding a get_microdescriptors() for how the spec seems to indicate that the method works but tor gives an empty reply. % curl http://217.13.197.5:9030/tor/micro/d/6dCl6ab8CLo0LeMjxi/MZgVJiZgWN8WKTesWPBMtyTo Checking with karsten on irc t see if he knows what's up, and moving on to another part. commit 4005a88becb68858ec4742455ea4d80e1ff9f827 Author: Damian Johnson Date: Fri Jul 19 10:12:39 2013 -0700 Implementing a use_directory_mirrors() method Method so we can balance load against directory mirrors rather than hammering the authorities. We can either request this during construction (which fails silently) or call this method explicitly (to get the exception). commit 5514b2cfd7d313f429bfe1bb8a13c975f7c3413f Author: Damian Johnson Date: Thu Jul 18 09:59:30 2013 -0700 Implementing a get_consensus() method Originally this was gonna be a get_network_status(fingerprint) method but evedently we can't request individual router status entries. Understandable since signatures are for the whole document but still a bit of a pity from an API perspective. Oh well. commit a014b9b967e8b825a9a4c99514ce2098ab6ba614 Author: Damian Johnson Date: Wed Jul 17 09:58:35 2013 -0700 Implementing get_extrainfo_descriptors() method Very similar to the get_server_descriptors() counterpart. commit 871a957f23097807f67e502fd6e2c9a9b9a6c456 Author: Damian Johnson Date: Tue Jul 16 09:28:18 2013 -0700 Rejecting requests for more than 96 descriptors In our tor-dev@ discussion Karsten mentioned that we could request at most 96 descriptors at a time when polling by their fingerprints... https://lists.torproject.org/pipermail/tor-dev/2013-June/005005.html I've emailed him to clarify where this limitation comes from (the url length or something within tor?), but in the meantime adding a check for this to our get_sever_descriptors() method. commit ac694d3a98ce1e56c674893f8006efe785eb41c9 Author: Damian Johnson Date: Tue Jul 16 09:16:06 2013 -0700 Implementing get_server_descriptors() method Implementing our first downloader method and added an ONLINE integ test. Initially I did a get_microdescriptors() method but turns out that tor didn't implement what we need... https://trac.torproject.org/9271 commit 7d04653d908f0c62e197bafa27a5cd94634cbb53 Author: Damian Johnson Date: Mon Jul 15 09:16:54 2013 -0700 Moving retry functionality to Query class It's far better if the Query class handles retries rather than the DescriptorDownloader. Advantages include... * The DescriptorDownloader no longer has a reason to support blocking queries. There's no advantage to them since the Query knows if/when the request fails (and hence can retry it on our behalf). * The Query class is easier to test. The more functionality we can push down into it the simpler the DescriptorDownloader will be. * More advanced use cases will be using Query instances rather than DescriptorDownloader. By having retry functionality there it'll be easily available to them. commit 0b1c13553f842088fd5f5ae21600d28e25e6b633 Author: Damian Johnson Date: Mon Jul 15 08:43:41 2013 -0700 Initial pass at the DescriptorDownloader Implementation of the DescriptorDownloader's _query() helper method which will be the basis of its other methods. On reflection I should move retry fruntionality and probably caching down to the Query class. commit d87f35c7c0f99d52a530cdf12d03ef6abd9cadb0 Author: Damian Johnson Date: Sun Jul 14 14:06:12 2013 -0700 Performing integ test queries in parallel What good is designing an asynchronous descriptor fetching API if you poll things serially? Trivial tweak to reduce this test's runtime from 2.4s to 0.4s. commit dc6632b5afba37111b9b2d6fb0c316cca1d9e046 Author: Damian Johnson Date: Sun Jul 14 11:44:57 2013 -0700 Integ test to exercise each of the authorities Adding an integration test that fetches moria1's descriptor from each of the directory authorities. This obviously requires network activity so it's run via the ONLINE target. This uncovered a couple interesting finds... * This is our first time actually using urllib2 (it was mocked previously), and its 'file like object' is incomplete. It lacks tell() and seek() which means that our parsers don't have random access. As such we need to read this all into memory and use a BytesIO.... damn. * The moria1 authority has an extra newline in its descriptor content. Why? Haven't a clue. Maybe it's running a different version of tor or maybe this is an artifact of fetching its own descriptor. Whatever it is, it's the only authority to do so. Simply stripping the content to avoid having our parser complain about this. commit 61f00819dc3f18b17af0d8d9d3f4f90791364e39 Author: Damian Johnson Date: Sun Jul 14 11:07:40 2013 -0700 Dropping Tonga from remote descriptor fetching Karsten reports that we shouldn't use authorities that lack 'v3ident'. At present that's just Tonga. commit 762b7239e4866ba1b98212259e9751723e0865df Author: Damian Johnson Date: Sat Jul 6 10:54:58 2013 -0700 Workaround for screwy RuntimeError related to mock causing imports The following isn't used by this directly, but we're still importing it due to a screwy aspect of how mock works. If patched() results in an import that we haven't done before then we can fail with... ====================================================================== ERROR: test_query_download ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/mock.py", line 1201, in patched return func(*args, **keywargs) ... File "/home/atagar/Desktop/stem/stem/descriptor/__init__.py", line 156, in parse_file import stem.descriptor.networkstatus RuntimeError: cannot unmarshal code objects in restricted execution mode I've only seen this arise when running this test directly (without the rest of the unit tests, some of which import networkstatus on their own). commit 762f0ad396dd752f0f807871e0715c142cdbf475 Author: Damian Johnson Date: Fri Jul 5 19:18:02 2013 -0700 Class for issuing remote descriptor queries Adding a Query class which does most of the heavy lifting for downloading descriptors. commit 65924aff93ef70c879175510e9cd2307e8507a9e Author: Damian Johnson Date: Thu Jun 20 08:57:36 2013 -0700 Stub for a remote descriptor fetching module Starting a module for remote descriptor fetching. Presently this just includes a no-op DescriptorDownloader class. Guts of this sucker are coming soon! commit b6c23b0c2934b6c1b1c51530754cc48f6ef4e38e Author: Damian Johnson Date: Sun Jul 7 13:35:50 2013 -0700 Typo in descriptor reader pydocs Correction from Roger. Honestly I had to look this one up. Both 'run' and 'ran' sound right when I read them to myself. Oh well, grammar is most certainly not my strong suit. commit 5cd89263f0acc5a7b6fcdd4021efa09a4f1634ab Author: Damian Johnson Date: Thu Jul 4 13:41:22 2013 -0700 Pyflakes and PEP8 corrections After upgrading both they each complained about a few new things. Mostly pyflakes getting confused by pydoc comments, but there were a couple fair catches. commit d0ce002d568627876c59c3ff23d6d86fbb049aa9 Author: Damian Johnson Date: Thu Jul 4 13:34:56 2013 -0700 Checking for testing prereq of mock 0.8.0 We had a requirement on mock 0.7.0 but it turns out that we're using a 0.8.0 feature too. Spotted this while trying to use ubuntu jaunty's apt-get version... Traceback (most recent call last): File "./run_tests.py", line 405, in main() File "./run_tests.py", line 184, in main for test_class in test.util.get_unit_tests(args.test_prefix): File "/home/atagar/Desktop/stem/test/util.py", line 130, in _get_tests module = __import__(module_name) File "/home/atagar/Desktop/stem/test/unit/tutorial.py", line 23, in class TestTutorial(unittest.TestCase): File "/home/atagar/Desktop/stem/test/unit/tutorial.py", line 24, in TestTutorial @patch('sys.stdout', new_callable = StringIO.StringIO) TypeError: patch() got an unexpected keyword argument 'new_callable' commit 8662332c844827b2be4ab7ab1c77027738e5723d Author: David Triendl Date: Sat Jun 22 09:03:50 2013 +0200 Fix parsing of IPv6 address:port pairs in stream events (fixes #9181) commit 00d78f244bda322f30ecb5d31f3278eeeaa96056 Author: Damian Johnson Date: Sun Jun 30 10:46:11 2013 -0700 Adding Slackware to our download page Markus just added stem to slackbuilds (hazaa!)... http://slackbuilds.org/repository/14.0/python/stem/ commit 003fa8e9e4b38304ccbb28e75a3f646ab6ba1ab2 Author: Damian Johnson Date: Tue Jun 25 08:57:42 2013 -0700 Switching get_network_status() to provide v3 entries Turns out 'GETINFO ns/*' provides v3 directory information rather than the v2 documented in the spec. Now that the spec is fixed swapping us over to take advantage of it... https://trac.torproject.org/7953 https://gitweb.torproject.org/torspec.git/commitdiff/d2b7ebb RouterStatusEntryV3 provides a superset of the attributes of RouterStatusEntryV2 so backward compatability shouldn't be a concern. commit 5f65f786dee22089323febc899bf2c85a53c99aa Author: Damian Johnson Date: Mon Jun 24 09:23:16 2013 -0700 Clarifying why test's load_conf() expects an exception For a long while I've been puzzled why this load_conf() call asserted an exception. When I ran it in a cusom prompt it ran fine - turned out that it was relying on us having a custom DataDir. Tor balks (rightfully) when we'd shift our DataDir out from a running tor instance. Doing a better job explaining what's up in the comment. commit 042fcfce009e5abc86f066c8ef0788cef871998f Author: Damian Johnson Date: Fri Jun 21 09:27:45 2013 -0700 Test runner tried to use non-existant print_error() function In commit e751397 we removed the test.output.print_error() function in favor of something less verbose. Somehow we missed its usage in test.runner - weird that pyflakes didn't catch this... Traceback (most recent call last): File "./run_tests.py", line 405, in main() File "./run_tests.py", line 212, in main integ_runner.start(target, args.attribute_targets, args.tor_path, extra_torrc_opts = test.util.get_torrc_entries(target)) File "/home/atagar/Desktop/stem/test/runner.py", line 310, in start self._start_tor(tor_cmd) File "/home/atagar/Desktop/stem/test/runner.py", line 718, in _start_tor test.output.print_error(" failed to start tor: %s\n" % exc) AttributeError: 'module' object has no attribute 'print_error' Also, the runner had a couple calls of... test.runner.get_runner() Those should obviously just be 'get_runner()', and only worked because of our test.output import (damn python's transitive imports are stupidly confusing). commit 456bc64f2fbd41918dd2db711eaf259a90f200e3 Author: Damian Johnson Date: Sun Jun 16 14:44:30 2013 -0700 FAQ entry about how to reload your torrc commit d921dfdfe2f085cd7de986ed16956b2a7361e32a Author: Damian Johnson Date: Sun Jun 16 12:50:50 2013 -0700 Renaming FAQ categories Trivial change to rename 'Usage' to 'General Information', and 'Tasks' to 'Usage'. commit e8a278cfed828be68d83486deefbff86fb548b0d Author: Damian Johnson Date: Sat Jun 15 20:40:01 2013 -0700 FAQ entry about how to get information about your present exits Answer for a question I like on Stack Overflow... http://stackoverflow.com/questions/9777192/how-do-i-get-the-tor-exit-node-ip-address-over-the-control-port commit 69ff2a6eec5b99a7b13a57f09de2e2abbd87b91d Author: Damian Johnson Date: Sat Jun 15 19:26:15 2013 -0700 FAQ entry for how to request a new tor identity A common question on Stack Overflow reguarding tor goes along the lines of 'how do I get a new identity?'. For instance... http://stackoverflow.com/questions/17013707/how-can-i-change-my-tor-process-endpoint-in-stem http://stackoverflow.com/questions/16987518/how-to-request-new-tor-identity-in-terminal Tor doesn't support cycling your IP, so adding a FAQ entry explaining why and telling them how NEWNYM works. commit 792557ccdb475cdd95be09b71a69d38f7770b3c7 Author: Damian Johnson Date: Sat Jun 15 15:45:02 2013 -0700 Requiring mock version 0.7.0 or later Our jenkins tests presently have mock version 0.6.0, but we're using mock's patch.dict which was introduced in 0.7.0. This causes the tests to fail with... Traceback (most recent call last): File "./run_tests.py", line 394, in main() File "./run_tests.py", line 173, in main for test_class in test.util.get_unit_tests(args.test_prefix): File "/srv/jenkins-workspace/workspace/stem-tor-ci/test/util.py", line 130, in _get_tests module = __import__(module_name) File "/srv/jenkins-workspace/workspace/stem-tor-ci/test/unit/version.py", line 20, in class TestVersion(unittest.TestCase): File "/srv/jenkins-workspace/workspace/stem-tor-ci/test/unit/version.py", line 22, in TestVersion @patch.dict(stem.version.VERSION_CACHE) Making run_tests.py balk if we're using a version of mock prior to 0.7.0. commit a5dc68b06d049df62f29f308f2029bdbd4521507 Author: Damian Johnson Date: Sat Jun 15 15:16:28 2013 -0700 Warning that the twitter example is broken Twitter has deprecated the 1.0 version of their API, which we used on... https://stem.torproject.org/tutorials/to_russia_with_love.html#reading-twitter Warning our users about this. The issue is being tracked on... https://trac.torproject.org/9003 commit 101cf0b71bab55d72a831eff0236dcef5edbff0f Merge: 65846e8 2b634a9 Author: Damian Johnson Date: Thu Jun 13 09:48:30 2013 -0700 Migrating to the mock library Our homemade mocking framework has served us well, but over time it's taught me one very important lesson: writing a mocking framework is hard. On the surface it seems pretty simple: apply and revert a set of monkey patches. But how do you monkey patch class methods? What about alias imports like the os module? And god forbid you want to mock python's open() function. I'm finally taking a lesson from one of my coworkers and using a library for this. Python has several options but the most common is PyPI's mock module, which became part of the standard library in Python 3.3... http://www.voidspace.org.uk/python/mock/ commit 2b634a98e16c3b2437cab27c7198acd6a738fd13 Author: Damian Johnson Date: Thu Jun 13 09:30:07 2013 -0700 Accounting for GetInfoResponse entries having bytes Oops. I forgot to update this test when we switched the entries attribute to bytes in commit a94f1f0. commit c7ac81546dcdf1fb9639fdbf0e4997e8ee39ba38 Author: Damian Johnson Date: Wed Jun 12 21:59:41 2013 -0700 Python3 mock compatability Ahhh, more unicode vs bytes joy. commit 35b44e3c7d5090806922cd0775775bb2985f8a9b Author: Damian Johnson Date: Wed Jun 12 21:48:44 2013 -0700 Dropping homemade mocking functions Done replacing our mocking methods with the mock module. It's nice to finally be able to drop these... commit da365bc6ea5312d467c33f14942695c66c6f4e10 Author: Damian Johnson Date: Wed Jun 12 21:39:25 2013 -0700 Using mock for system integ tests Unfortunately I'm not spotting a nice method via the mock module to get the original function. The patch() function has a get_original() method which looks promissing, but I didn't spot a method for accessing it. Oh well... commit f38959c05a5dbedcf4a90ed5fc6c1f9809ca5d62 Author: Damian Johnson Date: Wed Jun 12 19:53:35 2013 -0700 Using mock for protocolinfo integ tests commit 98fd433b09c2449e5ea6dc6d381e729ba32a6099 Author: Damian Johnson Date: Wed Jun 12 19:44:34 2013 -0700 Dropping mock from server descriptor integ test Well, that's odd. We revert mocking in these integ tests but never use it. Oh well. commit 21cd260745a1b824c908e3eacac4083fe16750a6 Author: Damian Johnson Date: Wed Jun 12 19:42:48 2013 -0700 Dropping mock from descriptor reader integ test I was gonna migrate this call over but it looks to be unneeded. Test passes so guess we probably replaced the problematic descriptor content with a real descriptor. commit 625472df56748195c5b0474209278dd57a66c338 Author: Damian Johnson Date: Wed Jun 12 19:38:11 2013 -0700 Using mock for process integ tests commit 752a957698550afc112ded236e3a7cceea500245 Author: Damian Johnson Date: Wed Jun 12 19:29:23 2013 -0700 Using mock for descriptor export unit tests commit 320d3ba383ab32ec4cecfc4158e9c7336df11c1e Author: Damian Johnson Date: Wed Jun 12 19:26:06 2013 -0700 Using mock for server descriptor unit tests commit 6c0af9db6169b7edb4eeb83f3555b9e5d6cc2d6e Author: Damian Johnson Date: Wed Jun 12 09:32:37 2013 -0700 Using mock for descriptor reader unit tests We get to replace some icky open() mocks. Yay! commit 33eeb9bbcc4a877d02f227d9f9b76a4631f3053e Author: Damian Johnson Date: Wed Jun 12 09:14:43 2013 -0700 Using mock for system unit tests Pretty much a drop-in replacement for these. commit 298a08c4b5488bf756c4dbce75ee0ebf4eeb2b0b Author: Damian Johnson Date: Tue Jun 11 09:27:08 2013 -0700 Dropping tempfile usage from controller unit test Now that I have a better understanding of the magic behind mocking open, there's no need for us to make a temporary file in our unit tests. commit 8e90c1029e650536cf07ee42ddeb06300f0aa002 Author: Damian Johnson Date: Tue Jun 11 09:09:05 2013 -0700 Using mock for proc unit tests Now that I have a rough understanding of the open mock it's quite a bit nicer than our prior version (which got especially icky for python 3 compatability). commit 8303c8d99ac6728367dabec3f6b2c9e43bb68c69 Author: Damian Johnson Date: Tue Jun 11 08:43:05 2013 -0700 Using mock for controller unit tests Another heavy mock user. Nice improvements, I'm really liking this library! commit 5089ded80aaa4fd5bc260c2b7fb5bf03e69b2133 Author: Damian Johnson Date: Mon Jun 10 08:50:44 2013 -0700 Using mock for protocolinfo unit tests Oooh, I like this change. commit f79278eec35de5520158923563256c45aa00f066 Author: Damian Johnson Date: Mon Jun 10 08:42:16 2013 -0700 Using mock for event unit tests We don't really need a mock in this test, guess we do so to be more faithful to the example. commit 1007a0a2d1963f350444c3b762cab6be8bda6eda Author: Damian Johnson Date: Mon Jun 10 08:30:16 2013 -0700 Using mock for authentication unit tests Mostly sorting out another thorny test that exhaustively exercises all authentication use cases. commit 9c0f1d37c0febbaacd2b6b73a2d094d3583869ba Author: Damian Johnson Date: Sun Jun 9 18:20:49 2013 -0700 Using mock for tutorial unit tests The unit tests for our tutorials are some of the ugliest we have since they make heavy use of mocks. Hence a good next step for mock. The mock module makes the controller mocking considerably nicer. That said, mock's open() mocking is crap. Their mock_open() docs... http://www.voidspace.org.uk/python/mock/helpers.html#mock.mock_open Tripped me up for well over an hour because their examples are hardcoded for the __main__ namespace (so it works when I do it in an interpretor, but not the tests). Even with that fixed the mock object it returns is buggy as hell (it's documented as supporting readlines() but that's a lie). They'd get a lot more mileage if they used an io.BytesIO instead. commit e50a62fd56aec4f5b9f96bf0828338356cc903e9 Author: Damian Johnson Date: Sun Jun 9 14:28:43 2013 -0700 Using mock for tutorial unit test stdout Mock's patch() docs include an example for mocking stdout. Using it for our tutorial unit tests... http://www.voidspace.org.uk/python/mock/patch.html#mock.patch commit 9109b6c078496c56b8ba1b85b0932b405be7d953 Author: Damian Johnson Date: Sun Jun 9 14:10:01 2013 -0700 Using mock for version unit tests First replacement of our mocking module with mock. This is actually both quite a bit cleaner, and also checks that our caching is working. commit 606ccfbb5001d5a13bce8b7a51cea283c074f3c1 Author: Damian Johnson Date: Sun Jun 9 12:14:56 2013 -0700 Adding the mock module as a test dependency Mocking is hard. We've gotten a lot of mileage by writing our own mocking helpers, but they're clunky and complicated. It's about time for us to use a more standard, mature option. The pypi mock module is not only well documented, but looks to be the direction the python world is going. As of python 3.3 it's now part of the standard library. As the first step of swapping us over I'm adding it as a dependency for run_tests.py. This warns the user if it's unavailable... atagar@morrigan:~/Desktop/stem$ ./run_tests.py --unit To run stem's tests you'll need mock... https://pypi.python.org/pypi/mock/ You can get it by running 'sudo pip install mock'. commit 65846e81628040779e4d28041d1400f53719468c Author: Damian Johnson Date: Sat Jun 8 13:22:12 2013 -0700 Correcting Controller pydoc links Handful of mistakes in the stem.control module that prevented sphinx's internal linking. Pity sphinx doesn't provide warnings about this... commit 2257f7954a0d9dba46905a4813b828fcc0b52086 Author: Damian Johnson Date: Thu May 30 08:43:10 2013 -0700 Minor fixes for ONLINE testing fixes Just some minor tweaks to fix the style checker and s/range()/xrange(). commit b5273658f6470a643b70b7ef96654248788eef2f Author: Jeremy Date: Wed May 29 23:53:11 2013 -0400 Made attachstream and mapaddress tests more robust Modified the attachstream and mapaddress integ tests so that they don't fail as easily in the face of minor network failures unrelated to stem (like temporary issues building a circuit, or an exit node having a different IP from the one Tor reports). This is to address... https://trac.torproject.org/8692 commit d7d14a82e7d28ff863c71b4b027c9922e17a6661 Author: Damian Johnson Date: Wed May 29 20:44:23 2013 -0700 Methods to query user tor is running as Couple additions to make it easier to determine the user tor is running as: * stem.util.system.get_user(pid) => determines the user a process is running as * Controller.get_user() => provides the user that tor is running as commit f5bdd944df426410fe6b21088280cf3779709ed1 Author: Damian Johnson Date: Wed May 29 08:53:12 2013 -0700 Checking is_caching_enabled() when using cache Adding a is_caching_enabled() call to our getter/setter for the Controller's cache. This has a couple advantages... * It prevents us from accidently caching stuff when the cache should be disabled. In at least one point we were using the cache without checking is_caching_enabled(), which was probably an error. * Lets us simplify our code a bit by making is_caching_enabled() optional for the cache getter/setter callers. commit 341091492edf7c060609b73b8619b18eda322ebd Author: Damian Johnson Date: Wed May 29 08:44:44 2013 -0700 Supporting get_pid() remotely via GETINFO One of get_pid()'s resolution methods (GETINFO process/pid) works remotely, so attempting it before checking that tor is running locally. commit 31456707624847f8b0404b2e84ebcc2dc7fe22fd Author: Damian Johnson Date: Wed May 29 08:19:54 2013 -0700 System module's call() function ignored exit statuses Well... oops. Our call() method was documented as raising an OSError (or returning the default value) when the call fails but it completely ignored the exit status. In practice the function only behaved as documented when Popen failed - otherwise it returned an empty list. Fixing this behavior and updating most of our callers since they expected call() to not raise any exceptions. commit 18aa77f4d6074e54db84f93e630aaacd5e3fbcb3 Author: Damian Johnson Date: Tue May 28 17:32:11 2013 -0700 System utility for getting FreeBSD jail paths Adding a util to better support FreeBSD jails (a rather common way to run tor on that platform). commit 24d9377cdc2d1ee3531238b825621934053bae9f Author: Damian Johnson Date: Tue May 28 09:32:13 2013 -0700 System utility for getting process start times Adding a get_start_time() to the system utils to provide the unix timestamp for when a given pid started. commit 292ef5c36e38dd2820ed00e34ab270f2bb28ecb6 Author: Damian Johnson Date: Mon May 27 21:28:55 2013 -0700 Performing pid resolion via getinfo when able Oops, forgot that I had added a 'GETINFO process/pid' option to tor for just this case. Using it when it's available. commit 0f7d5d118d83db000dd57b646ba91d4152c44e91 Author: Damian Johnson Date: Mon May 27 21:20:42 2013 -0700 Controller method to query tor's pid Adding a get_pid() method to the controller to make it simpler to figure out its pid. This attempts resolution via the PidFile, process name, control port, and control socket file. commit 0efbdf80f94574805b0fa080965bb509d535a4db Author: Damian Johnson Date: Sun May 26 21:27:43 2013 -0700 Allowing ControlMessage.from_str() to include conversion For txtorcon or anything else using our ControlMessage.from_str() function the following is very, very common... msg = stem.response.ControlMessage.from_str(getinfo_content) stem.response.convert("GETINFO", msg) Adding a 'msg_type' argument to from_str() so the two calls can be combined. This addresses... https://trac.torproject.org/8605 commit 7b396bdcebe7165df6ea6aaa911707e74974fa38 Author: Damian Johnson Date: Sun May 26 18:14:15 2013 -0700 Dropping validation on bandwidth-weights contents Our validation of network status documents included checking for the bandwidth-weights specified in the spec. This is because, according to the spec, they were all mandatory and new weights could not be added. Tor has loosened this restriction... https://gitweb.torproject.org/torspec.git/commitdiff/ec3eaca https://trac.torproject.org/6872 commit 08c4d273c67e1bb4d273ba5de188703d27a0568c Author: Damian Johnson Date: Sat May 25 20:57:12 2013 -0700 Noting additions in the change log Adding items from our commit history I forgot to include with the change log earlier. commit 03f4313b04285745f0c46cd83ac26fcc57bf0a6f Author: Damian Johnson Date: Sat May 25 20:26:11 2013 -0700 Adding Ubuntu to the download page Stem is now on Launchpad, evidently pulled from Debian for the Saucy Salamander release of Ubuntu. commit ad82c905347f3e5849930fdb3e63721b294d38b4 Author: Damian Johnson Date: Tue May 21 21:30:49 2013 -0700 Processing controller descriptors as bytes When descriptor content was fetched by the Controller it was first converted to unicode, then back into bytes. As a result descriptors with content that couldn't be converted (usually on the contact line) got mangled, and this in turn caused our check of its signature to fail. Caught thanks to aj00200 on... https://trac.torproject.org/8755 commit a94f1f0a51faacf1fda5350cdfd489d45b41adc4 Author: Damian Johnson Date: Mon May 20 09:35:45 2013 -0700 Support for getting byte content from get_info Adding an optional flag to get byte content from get_info rather than a str. This only effects python 3.x. commit 0362564cc8e1d48ebf7b61c75550fa9e1c4fd321 Author: Damian Johnson Date: Sun May 19 17:29:43 2013 -0700 Normalize ControlMessage content to be bytes The socket module's recv_message function converted read content to be a str (bytes in python 2.x and unicode in python 3.x). In 99.9% of the cases this is exactly what we want, but in a few edge cases we actually want byte content under python 3.x. For instance, in #8755 descriptors with non-unicode contact lines are being altered by the socket module, causing our validation of their signature to fail. Functionally this commit should have no effect besides adding the option to get byte content from the ControlMessage. The tests seem happy so fingers crossed! commit 525f2f9cee559fb8b77cb8daf46c68737487c63b Author: Damian Johnson Date: Sun May 19 14:59:41 2013 -0700 Using a real descriptor for test_non_ascii_descriptor Our test_non_ascii_descriptor test used an altered descriptor with katakana (Japanese characters) in its contact line. When we use doctored descriptors we can no longer exercise _validate_content since the signature won't match. Replacing the example descriptor with the one that's causing issues in #8755. commit ba50df27f2b69e05ce3470c4c1127974420a93bc Author: Damian Johnson Date: Sun May 19 14:11:39 2013 -0700 Mocking's sign_descriptor_content() incompatible with python 3.x I didn't have a python 3.x copy of pycrypto installed, so there were some code paths I didn't previously encounter. Unit tests failed due to sign_descriptor_content() not explicitly using bytes at several points. ====================================================================== ERROR: test_mirror_mirror_on_the_wall_4 ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/unit/tutorial.py", line 169, in test_mirror_mirror_on_the_wall_4 exit_descriptor = mocking.sign_descriptor_content(exit_descriptor) File "/home/atagar/Desktop/stem/test/data/python3/test/mocking.py", line 984, in sign_descriptor_content public_key_string = public_key_string[:64] + "\n" + public_key_string[64:128] + "\n" + public_key_string[128:] TypeError: can't concat bytes to str ---------------------------------------------------------------------- commit b816137fc07d139808dd80a410eba04fd662fdba Author: Damian Johnson Date: Sun May 19 12:52:30 2013 -0700 Minor simplifications for dependency module When we dropped python 2.5 support our _check_version() only had a single caller, making it not a very helpful helper. We were also missing pydocs for is_crypto_available(). commit b626f0c5156e8f5d2a89558ce54890b931648758 Author: Jeremy Date: Sat May 18 22:08:57 2013 -0400 Revise approach to owning Tor in integ tests The integ tests now only try to own the Tor process via control connection if controllers have a way to access Tor. This is needed to successfully run under the RUN_NONE target. commit 29086a71cd508614d247f70888df884028d51185 Author: Jeremy Date: Thu May 16 23:23:35 2013 -0400 Asserting ownership over the integ tests Ticket #8634. Integ tests now assert ownership over the tor process, but avoid accidentally terminating tor mid-test. commit 13dc7861ae7bfa280fda8536ca39fcf6135d7b56 Author: Damian Johnson Date: Thu May 16 09:20:11 2013 -0700 Integ tests resulted in temporary directories Our integ test runner created temporary directories for our control socket but failed to clean them up. On reflection it's simpler for us to use a UUID rather than tempfile.mkdtemp() here so our start() and stop() methods can perform the creation/cleanup. Tested by running our integ tests with a control socket... ./run_tests.py --integ --target RUN_SOCKET ... and confirming both that a temporary resource is made with our control socket and that it's cleaned up afterward. Issue caught thanks to help from Ashish on... https://trac.torproject.org/8622 commit ffd7274c8bc97557d4027257b958b388911cb429 Author: Damian Johnson Date: Tue May 14 08:30:54 2013 -0700 Re-enabling test_event_handling Oops, I confused this test with another. I bumped the timeouts on test_reattaching_listeners rather than this... https://gitweb.torproject.org/stem.git/commitdiff/a5f8168 Re-enabling the test with similar timeouts to see if jenkins is still happy. However, this is definitely still code stink indicating that we need to do something smarter for these tests. commit 73b5ed39da299109c8515bdbd6c18397a18a9634 Author: Damian Johnson Date: Mon May 13 09:59:42 2013 -0700 Client usage example for polling twitter Adapting a script from Ashish for polling a twitter feed as another tutorial example... https://pastee.org/ghbhp commit 1b965a4c81d0734d7f94a0283dbdd4b8793c4fa3 Author: Damian Johnson Date: Mon May 13 09:31:33 2013 -0700 Making SocksiPy work with official version Ashish's SocksiPy was based on a fork of the main project... https://code.google.com/p/socksipy-branch/ This caused my attempts to run it to fail with... Traceback (most recent call last): File "twitter_script", line 12, in socks.wrapmodule(urllib) AttributeError: 'module' object has no attribute 'wrapmodule' Making this work with the official upstream project is trivial so making the tweak. commit a72e22512e86653d86ec2015bbf3416a3de52aae Author: Damian Johnson Date: Mon May 13 08:45:51 2013 -0700 Using SocksiPy for client usage tutorial Exemplifying using tor with both SocksiPy and PycURL. It's quite a bit simpler with SocksiPy so I'll probably use this for most other examples. Change is thanks to Ashish. commit c4b47fc89582dc96b75c90d8eb7629d88304b06d Merge: 76f82d7 e96440e Author: Damian Johnson Date: Sat May 11 19:37:48 2013 -0700 Controller cache thread safety Making our caching operations thread safe. Patch thanks to Akshit... https://trac.torproject.org/8607 commit e96440eda20e1bbd89ed542654b4f080b8ecb064 Author: Damian Johnson Date: Sat May 11 18:36:44 2013 -0700 Revisions for cache thread safety Handful of changes for the prior couple commits. Most are stylistic, but there's a couple funcitonal ones: * _set_cache() could accidently remove 'exit_policy' * _confchanged_listener didn't use the new thread safe operations commit f208da6d4ca07ee4d241bd67085ba331bcc07d4b Author: Akshit Khurana Date: Sun May 12 03:03:50 2013 +0530 Add function for single param cache query Separate functions for single param cache query and multiple params. * get_conf : takes single param, returns value * get_conf_map: takes list of params, return dict commit 2c51c6a10cef7b68d9acc7703014bea2e4bd3101 Author: Akshit Khurana Date: Thu Apr 4 18:09:06 2013 +0530 Make controller cache thread safe reads and writes take place under cache lock https://trac.torproject.org/8607 commit 76f82d74960df9bccececff6792aa7dab88cd77b Author: Damian Johnson Date: Sat May 11 17:30:29 2013 -0700 Allowing for instantanious closure in test_status_notifications Well, that's a little odd. Our jenkins instance reported a failure of test_status_notifications... ====================================================================== FAIL: test_status_notifications ---------------------------------------------------------------------- Traceback: File "/srv/jenkins-workspace/workspace/stem-tor-ci/test/integ/control/base_controller.py", line 224, in test_status_notifications self.assertTrue(state_observer.timestamp < time.time()) AssertionError ---------------------------------------------------------------------- In the past we've seen test failures due to dixie being far, far faster than my netbook. I'm a little dubious that state_observer.timestamp truely equaled time.time() (that would imply that the above statements took zero nanoseconds) but oh well. I've only seen this happen once and there's little reason not to pass when they're equal. commit 2f69e4de1e5d9982dfe94a90c62da3c71edf96a2 Author: Damian Johnson Date: Sat May 11 17:21:13 2013 -0700 Disabling test_event_handling Well damn. Even with expanded sleeps test_event_handling is occasionally failing on the jenkins host. No test is better than a flaky test so disabling this for the time being. Ticket for fixing this test... https://trac.torproject.org/8865 commit 7c0b3ca25596123b5cfbc9251bdfbe630d23ffed Author: Damian Johnson Date: Sat May 11 17:09:52 2013 -0700 Support for pulling multiple pids with get_pid_by_name() A leading cause of test failures in our jenkins environment is test_get_pid_by_name_lsof. This is because the test assumes that there is only a single tor process, but that's often not the case on the dixie host. This not only fixes the test, but expands get_pid_by_name() to include a 'multiple' flag to pull all pids for a given process name. This includes unit test coverage. commit bbd736d71e25db09af6b6be05a4a33ef2ff17465 Author: Damian Johnson Date: Thu May 9 08:47:24 2013 -0700 Download page entry for Fedora Juan's package is now live so dropping our Red Hat entry in favor of Fedora (where the package is actually available). commit 76909729f3169af14d194a2f57e78e1604af7132 Author: Tomasz Kunikowski Date: Mon May 6 19:36:34 2013 +0200 Fixing DescriptorReader links in 'Mirror Mirror...' tutorial Links to DescriptorReader class were pointing to a wrong location. commit de576cad4bbdd4901ee1857334a7420593ef2458 Author: Sreenatha Bhatlapenumarthi Date: Thu Apr 25 06:36:27 2013 +0530 Updating GETCONF cache after CONF_CHANGED events https://trac.torproject.org/7713 commit f7d3a232a40ec93fec8c64f46fefba71a7fe9648 Author: Damian Johnson Date: Sun May 5 18:22:56 2013 -0700 Handle relative targets in the DescriptorReader The _handle_file() funciton expects to receive absolute paths so expanding relative inputs. This is a slight modification of wfn's patch from... https://trac.torproject.org/8815 commit a5f81683e1c29ba37cc4e97f0581d20c3949eb8b Author: Damian Johnson Date: Mon Apr 29 08:11:52 2013 -0700 Bumping sleep for test_reattaching_listeners On our jenkins instance we're seeing this test fail roughly 1/24 of the time (breaking one target in 1/3 of the test runs). We *should* be doing some kind of activity guarenteed to trigger an event rather than rely on a BW event being emitted each second. For now trying to simply bump its sleeps though to see if that relieve the symptoms. This adds around eight seconds to our integ runtime so definitely something we should look into doing right later. commit 3d047cb483f0692ef94b5b0cdac478da448957b0 Author: Sean Robinson Date: Fri May 3 17:28:48 2013 -0700 Use the more idiomatic staticmethod decorator This is just a coding style change and not a functional change. Funnily enough, there was already one of these decorators added in commit 5da6b9790da266f9. So, this standardizes on the more Pythonic form. Signed-off-by: Sean Robinson commit f5d690e5e241c8b53f6c4b0efa9f2519819a024e Merge: 1773eba 411f8cc Author: Damian Johnson Date: Sun May 5 18:05:43 2013 -0700 Using new syntax for exception variables Python has two styles for exception variables... * Python 2.7 and earlier (discontinued on 3.x) except ValueError, exc: * Python 2.6 and later (started in 2.5) except ValueError as exc: Change is thanks to Sean on... https://trac.torproject.org/8823 commit 411f8ccb508ae5eb941f5b8008909e4f1254374c Author: Damian Johnson Date: Sun May 5 18:00:11 2013 -0700 Expanding comments for exception style check Just adding a bit more of an explanation to the new style check. I'm a little tempted to make the second conditional "',' in content" so it'll catch exceptions by other names, but that might be too loose. commit 113f8c14723db3df5dacc02c5efaf54aeca6e688 Author: Sean Robinson Date: Thu May 2 16:29:18 2013 -0700 Convert all except clauses to new grammar Use the try..except...as grammar available in Python 2.6/2.7 and required in Python 3. Signed-off-by: Sean Robinson commit e65930b402cf4edee8ae18a812af4b3ea1aeeb21 Author: Sean Robinson Date: Thu May 2 16:15:26 2013 -0700 Check that except clauses are not using old grammar This tests that code matches PEP3110 and is more Python 3 compatible out of the box. This is possible because we no longer support Python 2.5. Signed-off-by: Sean Robinson commit 1773ebaab470206653ce6d84c3ef1276f81c5d0a Author: Damian Johnson Date: Sun Apr 28 18:13:12 2013 -0700 Updating download page entry for debian Debian was having some issues getting new packages to be reflected on 'packages.debian.org'... https://lists.debian.org/debian-infrastructure-announce/2013/04/msg00000.html Now that it's been resolved we can link to the package Derek made for us. Yay! commit f13c439c6b5e3702dce29d90b3daf24a4232487d Author: Damian Johnson Date: Thu Apr 25 09:28:28 2013 -0700 Typo in FAQ entry for 'how to run tests' Oops, s/static/style in "but static checking is run separately...". commit a1149f5489065cf7445f83137f3747a4866351e1 Author: Damian Johnson Date: Wed Apr 24 21:01:06 2013 -0700 Standardizing on digest() providing unicode Our digest() method provided bytes with python 2 and unicode with python 3. I'm trying to avoid these sort of version-dependent differences since they're a headache when we're support both serieses. The comparison for digest validation was also still failing in python 3 due to types. Caught thanks to aj00200. commit 92af5b8d50e3dac52e2380cd44ff706c88d9332e Author: Damian Johnson Date: Wed Apr 24 09:35:52 2013 -0700 Server descriptor digest validation failed for python 3 When validating the digest for server descriptors via python 3 we forgot to explicitly use byte values (rather than unicode). This is an adaptation of a patch from aj00200... https://trac.torproject.org/8755 commit c7ed4c84b30c8555ba8e2e63c158b74876d08db2 Author: Damian Johnson Date: Wed Apr 24 09:27:07 2013 -0700 Helper for getting torrc of a target Breaking a chunk of run_tests.py into a get_torrc_entries() helper function to improve readability. The runner could use an overhaul, and might later take on this functionality. commit 2a205415ed9bb1601a1ac65ec7c48f056561b3c8 Author: Damian Johnson Date: Mon Apr 22 07:40:35 2013 -0700 Checking pycrypto version when running tests Adding a check similar to python, pyflakes, and pep8 for the version when we run our tests. Bundling in various other small testing tweaks. commit 8adc55c0139ff9f0925541be1254bea4767716ed Author: Tomasz Kunikowski Date: Thu Apr 18 23:54:25 2013 +0200 Extra spaces were inserted into process names via set_process_name() After running set_process_name() ps would report spaces within our process name. For instance, when running our integ tests our name would become... p y t h o n . / r u n _ t e s Fixed thanks to ragwater, who figured out that ctypes.memmove() wanted unicode rather than byte content... https://trac.torproject.org/8631 commit 3d2f54327d64937016542381816a3529dcbfaa84 Author: Damian Johnson Date: Wed Apr 17 10:27:57 2013 -0700 Client usage tutorial can be disrupted if tor is running If you already have a tor instance running then the 'To Russia With Love' tutorial will fail to bootstrap due to a conflicting data directory. Thanks to Sreenatha for reporting this! commit 3ba0e6571bd0593c547bf27dd702679e12f802ef Author: Damian Johnson Date: Wed Apr 17 09:26:32 2013 -0700 Adding a :spec: role Generalising our roles to also support a :spec: alias for torspec commits. commit 1ad1f71be638fa21b8268d2f9cebec2c82ff17dc Author: Damian Johnson Date: Tue Apr 16 09:02:27 2013 -0700 Dropping '-dev' suffix for stem version on the site Like arm we're appending a '-dev' suffix for our git repository (to differentiate it from releases). However, for our site we just want the version itself. commit 5b58537814dcc4c2c2e474adb49b899add2fd0c4 Author: Damian Johnson Date: Mon Apr 15 20:40:41 2013 -0700 Shrinking FAQ icon Reducing the size of the FAQ icon a little bit to more closely match the download icon. Ain't perfect, but closer. commit f5aacc8bff0056748e8bc16e15f98d13e4dad024 Author: Damian Johnson Date: Mon Apr 15 20:30:14 2013 -0700 Raising FAQ category types Changing the category types so... 'subcategories' => 'categories' 'subsubcategories' => 'subcategories' This better emphasizes the questions and gives them a slight horizontal ruler. I'd like to get a more emphasized divider to better split up the questions but the only way I know of to do that is a table, which would be gross. commit 6b7b50c5c7834f76ddba0a78f337a5f1e4998599 Author: Damian Johnson Date: Mon Apr 15 09:29:32 2013 -0700 Adding a FAQ page Introducing a page with FAQ items for the most common questions that I've seen so far. Suggested by gsathya on... https://trac.torproject.org/8577 commit f17e370bb75c28c36e5aae1fe942b5bc97ca5438 Merge: fae5b1d 4bdcba0 Author: Damian Johnson Date: Sun Apr 14 16:07:18 2013 -0700 Custom sphinx role for :trac: Thanks to ragwater we now have a :trac: role for our docs. With this we no longer need to hardcode ticket urls, and gives a nice template we can use for other customizations. https://trac.torproject.org/8671 http://doughellmann.com/2010/05/defining-custom-roles-in-sphinx.html commit 4bdcba090b5a29e2a64c229a5ba9246ddbb59a69 Author: Damian Johnson Date: Sun Apr 14 16:05:03 2013 -0700 Style revisions Our trac.py is based on an external tutorial... http://doughellmann.com/2010/05/defining-custom-roles-in-sphinx.html As such it naturally had different stylistic conventions from stem. There were also some minor points that had some room for simplification. commit 5ab172051665101ce3c8797f3aac24a6d79459e7 Author: Damian Johnson Date: Sun Apr 14 15:37:17 2013 -0700 Using the :trac: role Replacing our hardcoded trac links with our fancy new :trac: role. commit d5ae78b127d1103927a6febb44587f62d1ed03d1 Author: Damian Johnson Date: Sun Apr 14 15:26:24 2013 -0700 Truncating generated trac urls Trac expands short urls. For instance, when visited... https://trac.torproject.org/8596 ... is expanded to... https://trac.torproject.org/projects/tor/ticket/8596 I kinda prefer using the shortened version in our docs so copied links are more readable. commit 02fa064867d186a3c9bbcc984f93ac0c20669ba9 Author: Damian Johnson Date: Sun Apr 14 15:17:39 2013 -0700 Including docs in our pyflakes and pep8 checks Including source from our docs directory in static checks. commit 0567974e7c3e0610dfa62e54b0a96ef54fd59866 Author: Tomasz Kunikowski Date: Sun Apr 14 22:29:06 2013 +0200 Implemented custom sphinx role :trac: for trac tickets Modified: docs/conf.py Added: docs/trac.py In accodrance to ticket #8671 Using :trac:`1234` will display 'ticket #1234' commit fae5b1db09ff925ac41d8bfab36621e29dbf0e69 Author: Damian Johnson Date: Sun Apr 14 14:47:35 2013 -0700 Checking for unused tests Well... shame on me. We've always had a hardcoded list of our tests, but somehow when I moved it to our settings.cfg I missed a few. I'm not so much troubled about that mistake, but more that this mistake went undetected. Adding a check at the start of our tests for this kind of misconfiguration. commit b36a9e3dc44adfda40390527fbb667a174a4736d Author: Damian Johnson Date: Sun Apr 14 14:20:10 2013 -0700 Revising ATTACHSTREAM 555 response handling Thanks to Roger we now know that a 555 response means that a stream's in a state where it can't be attached. Swapping the exception we raise to UnsatisfiableRequest and adding a little test for this. commit 5268d28c885509de174e5b7c1a657bc0278fd797 Author: Damian Johnson Date: Sun Apr 14 13:22:54 2013 -0700 Catching 555 response from ATTACHSTREAM ATTACHSTREAM should only respond with 250, 552, or 551 according to the spec but someone came across a 555 response in the wild... https://trac.torproject.org/8701 Tentatively handling this with OperationFailed until the spec is updated with more information. commit ae5193f93ec7f3c9db74e1b0452939a7cb1196c4 Merge: 314876a 62413a2 Author: Damian Johnson Date: Sat Apr 13 21:27:29 2013 -0700 Testing framework overhaul Over time we've accumulated quite a bit of cruft. Rewriting run_tests.py and related modules. commit 62413a29dbc73377ef3dd2231da1e9f35e4f30a9 Author: Damian Johnson Date: Sat Apr 13 21:22:40 2013 -0700 Rewriting run_tests.py Now that the building blocks are in place giving run_tests.py a long overdue rewrite. This pushes a great deal of the work to the test utils in the form of Tasks, units of work we can do in groups. commit 8a09c4951afb3b806a75fed00e70f280287a76f2 Author: Damian Johnson Date: Sat Apr 13 12:54:51 2013 -0700 Dropping direct term usage in run_tests.py Our use of the output module is trending more toward printing message categories rather than specific terminal attributes. This will let us customize how things are rendered through the output module rather than chasing down all of the individual println() calls. commit 24e4f0fca0fefa865aee1be527e5bb491f32aab8 Author: Damian Johnson Date: Sat Apr 13 12:47:35 2013 -0700 Dropping direct use of print Replacing our print calls with println(). This is both to standardize how we print our output as well as make later customization easier (for instance, if we want a --quiet option). commit e7513978c0a39fb815eecaff2f8d9a8c958236df Author: Damian Johnson Date: Sat Apr 13 12:41:22 2013 -0700 Less verbose usage of the test.output module We use the print_line() function throughout the runner and run_tests.py. Shortening it to 'println()' (pity we can't call it print()) and making its usage a little nicer by flattening the input attributes. commit b1f3504ac7fac93b5a26fd02525cc962228aaea2 Author: Damian Johnson Date: Sat Apr 13 10:05:57 2013 -0700 Dropping stem.prereq.is_python_26() The is_python_26() check was used to support python 2.5 hacks. Dropping this and the hacks that it was supporting. commit 00edd58211c9f67a0499aa6c04d5c4fa97580708 Author: Damian Johnson Date: Sat Apr 13 10:00:37 2013 -0700 Making check_requirements() require python 2.6 Oops, the prereq module was checking for python 2.5. We're now only compatable with 2.6 and above. commit 3d7bcee1aeba0f2ec2b8d808a1b57ba45ef731ae Author: Damian Johnson Date: Fri Apr 12 23:14:06 2013 -0700 Merging static_checks.py with the test utils Revising the static check functions and merging them with the test util module. commit 6076f4cf8375565acdde8dfbfffe2be4a2327480 Author: Damian Johnson Date: Fri Apr 12 09:13:30 2013 -0700 Moving clean_orphaned_pyc() to test utils Moving the functional bit of clean_orphaned_pyc() to the test utilities (the module specifically will *not* be used for generating output). commit 49c55af78b8adf699f934bcd48248539acc83d6b Author: Damian Johnson Date: Fri Apr 12 08:57:50 2013 -0700 Moving funcions for getting tests into util Now that we've freed up the util namespace I'm gonna gobble it up again for testing framework helpers. Hopefully this will be able to replace quite a few of our functions. commit e28971e61ec41ec76be74d9b703cd728d7ec0d0f Author: Damian Johnson Date: Fri Apr 12 08:40:16 2013 -0700 Merging test.network and test.util Both modules are to support tests for the ONLINE target. The main difference is that one module was made by Ravi and the other by Sean. At some point we should truely unify their functionality, but for now sticking them in the same place. commit 02fb92e92892f58c18c4f2de1bcf831ee43f4ff6 Author: Damian Johnson Date: Fri Apr 12 08:33:34 2013 -0700 Dropping runability of static_checks.py Once upon a time static_checks.py had a single function that did a fair bit more validation. At that time it made sense for this to be runable as a standalone script. However, nowadays it's mostly just helpers for pyflakes and pep8. commit f141a007655ae4d667942745b1acafa77391e0d0 Author: Damian Johnson Date: Fri Apr 12 08:31:49 2013 -0700 Supporting list of paths in static checker functions Reudcing a bit of redundancy by having the static_checks module support list of paths. commit 90cb6e347fc02dd60e19011a13e9947e0eed7381 Author: Damian Johnson Date: Fri Apr 12 08:10:56 2013 -0700 Moving ERROR_ATTR to output module The ERROR_ATTR was defined in both run_tests.py and the runner. Making both modules a little cleaner by moving it to the output module. commit 10d0bd80985c6dbdf8423890d0cd5f7f346ae559 Author: Damian Johnson Date: Fri Apr 12 07:56:32 2013 -0700 Untangling integration targets from configuration Even I was starting to get confused by 'em. We've always implicitely had two kinds of targets: run targets which constitute an integraion test run and attribute targets which have a non-torrc change on all test runs (such as being in a chroot). Making this division more explicit and dropping the confusing configuration dance we were doing. commit d46ecdfafd6883ac39bf24841af988d45d2fd64e Author: Damian Johnson Date: Thu Apr 11 09:28:18 2013 -0700 Dropping the argument.* config options Global state is icky, and in essence all stem.util.conf usage is global state. As mentioned in earlier commits the argument.* options were never used. Happily they were only consumed within run_tests.py so narrowing their scope is trivial. commit a8d1eb06768335ac2d95af78195608ba0f091f7b Author: Damian Johnson Date: Thu Apr 11 08:58:05 2013 -0700 Dropping load_user_configuration() helper One step closer to simplifying our arg parsing. The load_user_configuration() was mostly there to unify our configuration and commandline arguments. Now that the new-world-order will only have the later we should drop the helper to simplify the code. commit 5ccfcfcc1827e38901754dd3fab20c3f9dcba42b Author: Damian Johnson Date: Thu Apr 11 08:50:20 2013 -0700 Dropping support for a --config argument Lets face it, no one ever uses this. I doubt anyone besides me even knows *what* the hell it even does. Dropping this will let us simplify the test runner quite a bit. commit 930e3608cfef3678357f9e4ad4168f24da19ddf6 Author: Damian Johnson Date: Thu Apr 11 08:44:23 2013 -0700 Dropping argument.no_color Disabling output colorization stopped being a runner argument a long time ago. Like git, stem's test runner makes a pretty good guess about if colored output is supported or not. If it's wrong then we can fix the check. commit 347d2859073672e9c0354727901769245fb3b758 Author: Damian Johnson Date: Thu Apr 11 08:35:31 2013 -0700 Moving test filter to helper function Shifting support for the '--test' argument to the helper. Oh, and the integ helper was running the unit tests instead. ;) commit 76e751f9c8fd338c73752a8547cedc92a95363de Author: Damian Johnson Date: Wed Apr 10 10:16:35 2013 -0700 Moving test imports to the runner module Ideally the run_tests.py module should be as simple as possible, simply parsing the user input and calling helper functions to run the tests. To this end I'm moving the code for enumerating unit and integration tests into the runner module. commit 314876a6a7ad2067a5e630faae2b88de180cfd62 Author: Damian Johnson Date: Mon Apr 8 09:10:10 2013 -0700 Adding a link to our jenkins setup Link at the bottom of our index for our automated testing. commit 8774d00808a33dc6265d994d05d61a8823d24c75 Author: Damian Johnson Date: Mon Apr 8 08:13:39 2013 -0700 Missing runner import for stem.connection One thing I dislike about python imports is that they can sometimes happen transtively. If modoule A imports module B which imports module C then module A seems to have a reference to module C. This gets confusing as hell because when you see that the 'import B' is unused and remove it you get completely unrelated errors. Anywho, I got a slew of stacktraces while running the integ tests about a missing import for stem.connection in the runner module... ====================================================================== ERROR: test_get_pid_by_port_netstat ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/integ/util/system.py", line 250, in test_get_pid_by_port_netstat elif not runner.is_ptraceable(): File "/home/atagar/Desktop/stem/test/runner.py", line 415, in is_ptraceable tor_version = self.get_tor_version() File "/home/atagar/Desktop/stem/test/runner.py", line 569, in get_tor_version control_socket = self.get_tor_socket() File "/home/atagar/Desktop/stem/test/runner.py", line 536, in get_tor_socket stem.connection.authenticate(control_socket, CONTROL_PASSWORD, self.get_chroot()) AttributeError: 'module' object has no attribute 'connection' I'm not really sure when this was introduced, but fixing it. commit 744941e781c0597b05166001bae00b81ec8aaab1 Author: Damian Johnson Date: Sat Apr 6 13:25:38 2013 -0700 Moving list of test modules to settings.cfg Further cleaning up our overly-verbose run_tests.py by moving the list of test modules to our settings. commit 76a7a12ae041cf9bab120264acd52a08a318d8d7 Author: Damian Johnson Date: Sat Apr 6 12:57:01 2013 -0700 Dynamically importing test modules Whenever we add a new test we need to add it to both the imports and UNIT_TESTS or INTEG_TESTS tuple. This is both a little clunky and, now that we have a crap ton of tests, it makes run_tests.py really long. Using __import__() to dynamically import the tests as we run them. This might have a slight runtime advantage too since we're just importing what we need, though it's not noticeable. This trick comes from... http://effbot.org/zone/import-string.htm commit 29e7169d9a9947b86fe7af557a7230932fcc49f8 Merge: b7d8cad c719b1e Author: Damian Johnson Date: Sun Apr 7 17:25:34 2013 -0700 Site navbar menu Replacing the normal sphinx 'previous/next' navigation with a navbar menu. Hopefully this will make site navigation far, far easier... https://trac.torproject.org/8431 commit c719b1e5680898e9d1f6ede953b3169d12580f64 Author: Damian Johnson Date: Sun Apr 7 17:22:17 2013 -0700 Simplifying site navigation via a navbar menu Using a CSS style navbar to make site navigation easier. This is based on the one from... http://aext.net/2009/09/elegant-drop-menu-with-css-only/ commit 174d39e9e5b6825da3ebb0490b3ebffb7d858fff Author: Damian Johnson Date: Sun Apr 7 11:33:49 2013 -0700 Replacing paging nav bar with categories Sphinx wants the layout of sites to be like a book, with a 'previous' and 'next' page. I'm at a loss for how that could possibly be a good idea. Sites like stem's have categories (tutorial, api, download, etc) which our navbar should provide a quick method to access. commit bc8ec1da82bd45e2ec7b694ea824fedc49ae6395 Author: Damian Johnson Date: Sun Apr 7 11:32:17 2013 -0700 Forking the haiku theme The haiku theme has limited customizability. In particular we can't... * Overwrite the normal nav elements (previous, contents, next) with something else since they aren't within a block. The only way to overwrite them it to clobber the contents block, which kinda defeats the point of using their layout. * The css uses text layout right but we want text layout left. Hence I'm copying in the haiku layout and css so I can customize them. This is the first step toward a proper navbar for our site... commit b7d8cadd12ff630bfe204e3f575bd624e9fae224 Author: Damian Johnson Date: Sat Apr 6 22:14:45 2013 -0700 Favicon for our site Mmmm, 32x32 favicon. I tried to get an ico as the sphinx config suggests but that didn't work. Oh well, just about every browser except maybe IE can handle pngs so I'd prefer to go with that anyway. commit ad6973d05a0b25dc80768264a41cb717326c75f3 Author: Damian Johnson Date: Sat Apr 6 21:47:44 2013 -0700 Adding a change log page As changes accumulate we need a page to convey them to our users. Adding a change log that both has this, and information about what our versioning means. commit 64fef2d9ee6ab5aefc597eb4f68c910df8f0a197 Author: Damian Johnson Date: Sat Apr 6 18:13:35 2013 -0700 Settling on a logo for stem Using Jakub Jankiewicz's WPClipart image as the logo for stem. This image is under the public domain, and a very nice fit for our library. :) commit 9c8208474758862b7ae451c5c7eb93795df1015a Author: Damian Johnson Date: Sat Apr 6 17:43:17 2013 -0700 Noting alternate onion images Citing some other more artistic (but less iconic) images that I came across while looking for a logo for stem. Might come in handy in the future and are under Creative Commons (thanks Andrea!). commit 46d39df76cc51132c8e38c4fcbaabd45844e0c1c Author: Damian Johnson Date: Sat Apr 6 16:52:09 2013 -0700 Support for the ADDRMAP event's CACHED flag Adding support for the new CACHED flag added by Desoxy... https://trac.torproject.org/8596 https://gitweb.torproject.org/torspec.git/commitdiff/25b0d43a9ebe89541625e9f1a090e4a701748c76 commit 5879995bf9118792a9c27376a93361e779ec4fc2 Author: Damian Johnson Date: Fri Apr 5 09:19:48 2013 -0700 Skipping test_get_pid_by_port when connection resolvers are unavailable Our jenkins setup revealed that when netstat, sockstat, and lsof are all unavailable test_get_pid_by_port fails. Skipping instead since those connection resolvers are dependencies for the test. commit 95e9d663796a49fdc2427c7b70fe5f557b0e2cb3 Author: Damian Johnson Date: Fri Apr 5 09:02:59 2013 -0700 Improving error message when get_system_tor_version() can't find tor When tor is unavailable the error output from get_system_tor_version() is pretty sucky... Traceback (most recent call last): File "./run_tests.py", line 525, in our_version = stem.version.get_system_tor_version(CONFIG["argument.tor"]) File "/home/atagar/stem/stem/version.py", line 84, in get_system_tor_version raise IOError(exc) IOError: [Errno 2] No such file or directory Making it more developer friendly. commit f5971e1da626425062d8922a35b7b87447faa238 Author: Damian Johnson Date: Fri Apr 5 08:47:34 2013 -0700 Still passing when tor's stime is zero The stime proc field is an integer, rounded down to the number of seconds that a process has been running. On my piddly little netbook tor takes long enough to start that this is always non-zero, but on our jenkins systems it's so quick that the test fails. Thanks to weasel for all of the help! commit a920e5cfc2e3d0283bdb467a9a78bd281b8ee2a2 Author: Damian Johnson Date: Wed Apr 3 08:33:40 2013 -0700 Python 2.6 doesn't support assertIsNotNone() The controller integ tests included a couple assertIsNotNone() calls, which wasn't added until python 2.7. Caught and fixed by Desoxy - thanks! https://trac.torproject.org/8629 commit 5c9efa028b3e61d038d6fbf617e3f7807a75a1f3 Author: Damian Johnson Date: Tue Apr 2 10:15:20 2013 -0700 Adding resource for event tutorial logo Including the full sized image used for the 'Tortoise and the Hare' tutorial. commit ec40da3a8a5959b92897c17d284d5883082567bd Merge: d3f1d80 c713715 Author: Damian Johnson Date: Sat Mar 30 18:50:19 2013 -0700 Revising the download page Expanding our download page to include packaging done for our 1.0.1 release. Many thanks to everyone involved! commit c71371521ef0b8a294755f00a7689a935ab67f32 Author: Damian Johnson Date: Sat Mar 30 18:43:19 2013 -0700 Emphasising installation commands Using a preformatted block for installation commands since this is what visitors are likely most interested in. commit c9122fbd24f32b288eebfa60dc7c77e802e58935 Author: Damian Johnson Date: Sat Mar 30 18:24:15 2013 -0700 Adding Red Hat to the download page Linking to the ticket Juan filed to add our stem release to redhat/fedora. commit 170e44d3380b4cac140dc0ee52c280692d6aec0f Author: Damian Johnson Date: Sat Mar 30 18:07:49 2013 -0700 Resizing FreeBSD logo Shrinking the logo a little bit to match the other entries. commit 302ad96b7eaacc40784dfef43d4a3cdbcba4986e Author: Damian Johnson Date: Sat Mar 30 18:02:59 2013 -0700 Using the official git logo I'm a bit torn between our custom git logo and the official one. Opting for the later in large part because it's now next to FreeBSD which has a red logo, so this looks a little better. commit ee0089ba76d3c4357113fe11d3302682afca50de Author: Damian Johnson Date: Sat Mar 30 17:10:19 2013 -0700 Instructions links for ArchLinux and FreeBSD Adding the instruction links for installing via AUR and ports. commit c3b37286675617930f7bd3aad2d8581bab2752d3 Author: Damian Johnson Date: Sat Mar 30 16:20:15 2013 -0700 Adding FreeBSD to our download page Adding the stem port Carlo made for our release. commit 74dc0846a647b4a7c55b51e39fd9b3c6d8a24e0d Author: Damian Johnson Date: Sat Mar 30 16:06:50 2013 -0700 Citing ITP request in debian download page entry Pointing the request Dererk filed for adding our release to debian. commit 3db5fd85b275a9413c518760e8e78dd35272974e Author: Damian Johnson Date: Sat Mar 30 16:01:59 2013 -0700 Updating ArchLinux download url Sjon has uploaded a new AUR package for our 1.0.1 release. Pointing our download page at that instead. commit 626f8247b74e8500a4557b4da55f94f533f3ed3c Author: Damian Johnson Date: Sat Mar 30 15:58:15 2013 -0700 Revising download page entry for the source repository Placing the source repository at the bottom of the download page and renaming it. commit 9e4d1b9adde0414efaec23bb1547f055b8db0f92 Author: Damian Johnson Date: Sat Mar 30 15:48:45 2013 -0700 Adding PyPI to download page Stem is now available on the python package index... https://pypi.python.org/pypi/stem/ Thanks to refnode for providing us with the namespace! commit d3f1d80483d88deed04871aac4403377458fb406 Author: Damian Johnson Date: Wed Mar 27 09:41:58 2013 -0700 Stem release 1.0.1 commit 4ee02d05f266a0fd60f470d28992026f6a6d49d6 Author: Damian Johnson Date: Wed Mar 27 09:37:58 2013 -0700 Avoiding stem imports in setup.py Our setup.py may be ran under python 2.x or 3.x. Because of this the file and anything it imports needs to run under both python serieses *without* a 2to3 conversion. We were importing basic module information from __init__ which in turn brought in the enum and str_tools utils... atagar@morrigan:~/Desktop/stem$ python3 setup.py install Traceback (most recent call last): File "setup.py", line 5, in from stem import __version__, \ File "/home/atagar/Desktop/stem/stem/__init__.py", line 504, in "ERR", File "/home/atagar/Desktop/stem/stem/util/enum.py", line 64, in UppercaseEnum return Enum(*[(v, v) for v in args]) File "/home/atagar/Desktop/stem/stem/util/enum.py", line 77, in __init__ if isinstance(entry, (bytes, unicode)): NameError: global name 'unicode' is not defined Bug caught by Dererk. commit 07553f7b4cb3e21b74b5582f035a2f22477183c4 Author: Damian Johnson Date: Tue Mar 26 09:21:06 2013 -0700 Stem release 1.0.0 commit 8bc11004558a1362f61f75d967a221183777672d Author: Damian Johnson Date: Tue Mar 26 08:29:13 2013 -0700 Another python3 authentication issue with mixed str types Fixing another failure in our python3 integ tests with cookie authentication. The hmac function provides us with bytes but we need to be careful to use unicode in the formatted string to avoid a b'' wrapper. commit c75ab2812b90a730e135b5970ff35d431fa2c4a4 Author: Damian Johnson Date: Mon Mar 25 08:33:57 2013 -0700 Safecookie authentication broken when using python 3 The os.urandom() provides bytes so there's no need to convert that, but we needed to be more careful about using the converted hex in the formatted string. It too was bytes, causing an extra embedded b''... >>> "hello %s" % b"damian" "hello b'damian'" Tor in turn rejected the nonce as being invalid. I don't get why python3 does it. With python2 bytes and unicode could both be used in a reasonable fashion for formatted strings - these extra characters are almost always undesirable. Oh well... commit 83b99dbe6579af04f18553740b175d6de44fa27d Author: Damian Johnson Date: Mon Mar 25 08:12:30 2013 -0700 Integ tests failed to create a control socket The RUN_SOCKET test was failing while trying to make the control socket because it tried to change the permissions of /tmp to 700 (which, of course, normal users can't). Tor rejects to start if the control socket is in a globally readable directory, so making a temporary directory we can set the permissions on. commit 89974d0f67f09f44c0fb8b3f2344a78b27280dc4 Author: Damian Johnson Date: Mon Mar 25 08:10:36 2013 -0700 Only running RUN_OPEN by default or if requested We were configured to always run the RUN_OPEN target, even if something else had been requested. This bug lasted quite a while because we usually either run the default or RUN_ALL (which includes RUN_OPEN). commit 5e6b5407f087b6b31d28ed5047e44b07f082d197 Author: Damian Johnson Date: Sun Mar 24 21:05:14 2013 -0700 Controller provided unicode microdescriptors with python 3 When using python 3 the get_info() method provides unicode content. We were passing this content to descriptor constructors, causing its raw_content to be unicode rather than bytes. This in turn caused a stacktrace when calling methods like hashlib.sha256(). This doesn't effect python 2.x, where str is bytes anyway. commit fa0fbd195d0531980a6e250ee2e32c99a0708259 Author: Damian Johnson Date: Sun Mar 24 20:52:07 2013 -0700 Allowing stem.descriptor.parse_file() to handle paths Several times now I've passed a string path to parse_file() rather than a file object and been confused for a sec by the stacktrace. On reflection there's no reason *not* to accept paths - that's commonly what we want it for after all and eliminating the extra open() lets our callers have much nicer code. commit e10b6124e22fd9febd7bd23abcc4b820353fbf8b Author: Damian Johnson Date: Sun Mar 24 20:35:22 2013 -0700 Simplifying use of digest values for microdescriptors To be useful a microdescriptor often needs to be matched against its corresponding router status entry. The design of microdescriptors makes this suck - the only way to do so is to iterate over all router status entries looking for one with the corresponding digest. Adding examples for doing this and making the following changes to make the matching easier... * changing the digest value of RouterStatusEntryMicroV3 from base64 to hex * adding an attribute with the hex digest to Microdescriptors commit b05bf7a7eeb9807bf003518b39677f8c6abc39c8 Author: Damian Johnson Date: Sun Mar 24 19:40:23 2013 -0700 Having ExitPolicyRules report /0 as a wildcard ExitPolicyRules can apply to both IPv4 and IPv6. Because of this I treated * as a wildcard, but *not* /0 (since 0.0.0.0/0, though it applies to all IPv4 addresses, doesn't cover IPv6). However, in most cases (everywhere except the torrc?) Tor's exit policies are limited to IPv4. As such, /0 should usually be treated as a wildcard. Change suggested by Aaron Johnson. commit caee7d6c968fd6cd9912dc4a554d66e303316b61 Author: Damian Johnson Date: Sat Mar 23 19:50:52 2013 -0700 Switching exit policy's can_exit_to() and is_match() be permissive Switching how the ExitPolicy's can_exit_to() and ExitPolicyRule's is_match() handle an undefined address or port. Previously we reported if we matched or allowed exiting to *all* instance of the defined destination. Now, however, rather if we match against *any*. Change was requested by Aaron Johnson. You can still get the old behavior by setting the 'strict' argument. commit cefe584cd57c45a77379428a4c15fb225c0afdcc Author: Damian Johnson Date: Sat Mar 23 19:17:31 2013 -0700 Testing errors when descriptors are unavailable Fixing a couple testing errors spotted by Dererk. These arise when we're either unable to fetch descriptors (disconnected with a fresh test data directory) or possably if descriptors are still in the process of being fetched. ====================================================================== ERROR: test_get_microdescriptors ---------------------------------------------------------------------- Traceback: File "/home/dererk/stem/stem/test/integ/control/controller.py", line 843, in test_get_microdescriptors for desc in controller.get_microdescriptors(): File "/home/dererk/stem/stem/stem/control.py", line 1050, in get_microdescriptors raise exc OperationFailed: Data directory doens't contain cached microescriptors ====================================================================== FAIL: test_get_microdescriptor ---------------------------------------------------------------------- Traceback: File "/home/dererk/stem/stem/test/integ/control/controller.py", line 823, in test_get_microdescriptor test_relay = self._get_router_status_entry(controller) File "/home/dererk/stem/stem/test/integ/control/controller.py", line 1018, in _get_router_status_entry self.fail("Unable to find any relays without a nickname of 'Unnamed'") AssertionError: Unable to find any relays without a nickname of 'Unnamed' He also got a third error that mystifies me a bit. Fingers crossed that earlier changes have addressed it, if not then I'll need repro steps to dig in further... ====================================================================== ERROR: test_get_network_status ---------------------------------------------------------------------- Traceback: File "/home/dererk/stem/stem/test/integ/control/controller.py", line 922, in test_get_network_status self.assertRaises(stem.ControllerError, controller.get_network_status, "blargg") File "/usr/lib/python2.7/unittest/case.py", line 476, in assertRaises callableObj(*args, **kwargs) File "/home/dererk/stem/stem/stem/control.py", line 1166, in get_network_status raise exc ValueError: Router status entries must have a 'r' line: commit 73ba530be8d7df7600e1b9a29df904de29a94831 Author: Damian Johnson Date: Sat Mar 23 19:03:46 2013 -0700 Switching network status document routers to a dict Changing the 'routers' attribute of NetworkStatusDocuments to a dict mapping fingerprints to the router status entry. This is generally far more convenient to work with - thanks to gsathya for the idea! commit ea060119690650b4997c5f4c25f036b3ddeda3ac Author: Damian Johnson Date: Sat Mar 23 16:24:47 2013 -0700 Presently no quoted GETCONF responses exist As per ticket #6172 the GETCONF response currently returns each result on a separate line, so quotes are unnecessary. If tor does add quoted values then we'll need to see at that time how it should be handled. commit b3c00c4a6d0565939745e6eba2064b17aaff491a Author: Damian Johnson Date: Sat Mar 23 15:55:13 2013 -0700 Dropping python 2.5 compatibility hacks As discussed on tor-dev@ there's a lot of arguments against maintaining support for python 2.5 and precious few for... https://lists.torproject.org/pipermail/tor-dev/2013-March/004551.html Dropping all the hacks we adopted to support it. commit 88c3b1a6ae4152f1724fcc7267b7a5b69aae5a74 Author: Damian Johnson Date: Sat Mar 23 15:06:08 2013 -0700 Changing isinstance() checks to look for both bytes and unicode Commit b12e9c0 broke a couple of our integration tests because it caused us to call controller methods with unicode rather than str arguments... https://trac.torproject.org/8576 This should be fine - python 2.x can usually use bytes and unicode inputs interchangeably. The problem was that we routinely use isinstance() checks to see if an input is a string and, if it isn't, assume it's a collection. As a result we were calling... GETINFO m d / n a m e / f o o ... rather than... GETINFO md/name/foo Tor in turn errored saying that those single character getinfo options didn't exist. To fix this I'm changing most of our isinstance() checks to look for both bytes and unicode. commit cba36249020b0f322ce0ca6438d31dd5d4281905 Author: Damian Johnson Date: Sat Mar 23 12:33:56 2013 -0700 Issuing exit status 1 if there's any testing failures We explicitely set an exit status at several points but we don't set one for the most important situation: testing success/failure at the very end. Caught by weasel. commit bee3c2b53653976a36b16a7cefb6e11bb56e4045 Author: Damian Johnson Date: Fri Mar 22 20:32:07 2013 -0700 Running python 3 tests errors when done with python 2.6 When running tests with '--python3' we were using the named tuple sys.version_info. This was just a normal tuple prior to python 2.7, causing an error when run with python 2.6. Caught and fixed by Desoxy... https://trac.torproject.org/8565 commit c7f34c929a4f091b0527bec30d340472b55eacab Author: Damian Johnson Date: Fri Mar 22 19:49:06 2013 -0700 Swapping contact and platform attributes to bytes The 'contact' and 'platform' lines in a server descriptor can have any arbitrary byte content. This isn't necessarily unicode, so we can't provide that to our callers without potentially mangling the data. Non-unicode contact lines are surprisingly common (fourteen instance in one cached-descriptors, around 0.4%). commit b12e9c01c318863f4e4cddacf341abfd197ff412 Author: Damian Johnson Date: Wed Mar 20 18:28:55 2013 -0700 Storing raw descriptor content as bytes Tor descriptors are byte strings. While most content is ASCII (and hence maps to UTF8) there are a couple exceptions: the server descriptor's contact and platform lines which are arbitrary byte content. Usually this is unicode but on occasion it isn't (14 of 3044 in my old consensus cache). In those cases we converted the content to unicode and replaced the un-decodeable characters. This in turn caused our _verify_digest() check to report the descriptor content as being malformed if the user had pycrypto installed. Changing the raw contents the Descriptor class tracks to be bytes instead of unicode. This actually leads to far more elegant code, letting us drop the UnicodeReader which was always a pretty gross hack. This breaks python 2.5 compatability by using the io.BytesIO class. Presently I'm checking with tor-dev@ to see if anyone objects with dropping python 2.5 support entirely... https://lists.torproject.org/pipermail/tor-dev/2013-March/004551.html If the answer is... * yes, drop support: we'll replace test.mocking.BytesBuffer with io.BytesIO (they both do the exact same thing) * no, keep support: we'll need to move test.mocking.BytesBuffer to a util and use it instead of io.BytesIO commit 98cb69522254ddfd7bdd3c8df3b0dc7020c8e0a3 Author: Damian Johnson Date: Wed Mar 20 17:34:07 2013 -0700 Avoiding use of sign_descriptor_content() The test.mocking.sign_descriptor_content() function is slow. When the user has pycrypto the runtime of our unit tests jump from 17 to 83 seconds. This is a no-go since it discourages the use of our tests (our unit tests were already taking too long imho before this jump). By dropping sign_descriptor_content() the digest validation checks fail, so mocking those out. commit 617661308bbc70c08b2d45767965969c690ac9f9 Author: Damian Johnson Date: Wed Mar 20 09:50:12 2013 -0700 Test still used old arg name for socket connections Oops. When running other targets we were sometimes hitting an instance that still used 'socket_path' rather than 'path'. Caught by gsathya. commit dd220882e6c73863cc574ec4ba3833ee478314cb Author: Damian Johnson Date: Tue Mar 19 13:40:10 2013 -0700 Dropping support for port lists in descriptors The 'or-address' and 'a' descriptors supported comma separated lists of ports. However, as per spec commit 99a9587 tor has never supported this, and the spec no longer includes this capability. commit cbd83abf6e91955d84a87d3dada8548ad13ab99f Author: Damian Johnson Date: Tue Mar 19 08:25:06 2013 -0700 Commenting tutorial use of curses.wrapper() Suggestion from Lunar for a rough spot in the tutorials. commit 32244cdc3e8fbfff0cb9ca17f848ce8ede8ade3b Author: Damian Johnson Date: Tue Mar 19 08:15:19 2013 -0700 Tutorial wording revisions Addressing various suggestions from Sean for improving the tutorials. commit 796ea199b44fbd0fe4d2b55faad1a1bdcd64bef0 Author: Damian Johnson Date: Tue Mar 19 07:09:50 2013 -0700 Switching the server descriptor's family to a set Suggestion from gsathya. Duplicate family entries are moot and order doesn't matter, so might as well make this a set. commit decc35249d2605644aec5a3ef5bb337cd0d41175 Author: Damian Johnson Date: Mon Mar 18 18:40:13 2013 -0700 Test accidently left around a file in /tmp The test_save_processed_files_malformed checks that save_processed_files() fails when presented with malformed data. However, this function still writes out a blank file. I noticed this because I just tried running the tests as a different user from prior test runs, causing... ====================================================================== ERROR: test_save_processed_files_malformed ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/integ/descriptor/reader.py", line 172, in test_save_processed_files_malformed self.assertRaises(TypeError, stem.descriptor.reader.save_processed_files, "/tmp/foo", listing) File "/usr/lib/python2.7/unittest/case.py", line 465, in assertRaises callableObj(*args, **kwargs) File "/home/atagar/Desktop/stem/stem/descriptor/reader.py", line 229, in save_processed_files with open(path, "w") as output_file: IOError: [Errno 13] Permission denied: '/tmp/foo' ---------------------------------------------------------------------- Ran 21 tests in 1.558s commit 8099efdecf3d64abef787d1b19b537c4b9c30088 Author: Damian Johnson Date: Mon Mar 18 18:32:28 2013 -0700 Only query by name for relays with the Named flag Our integ tests pick a relay at random from its present consensus to test the 'GETINFO desc/name/*', 'md/name/*', and 'ns/name/*' functions. However, this could fail if we're querying a relay without the Named flag. ====================================================================== ERROR: test_get_microdescriptor ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/integ/control/controller.py", line 826, in test_get_microdescriptor md_by_nickname = controller.get_microdescriptor(test_relay.nickname) File "/home/atagar/Desktop/stem/stem/control.py", line 1001, in get_microdescriptor raise exc InvalidArguments: GETINFO request contained unrecognized keywords: md/name/MukiMukiAmaguri ====================================================================== ERROR: test_get_network_status ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/integ/control/controller.py", line 928, in test_get_network_status desc_by_nickname = controller.get_network_status(test_relay.nickname) File "/home/atagar/Desktop/stem/stem/control.py", line 1165, in get_network_status raise exc InvalidArguments: GETINFO request contained unrecognized keywords: ns/name/MukiMukiAmaguri ---------------------------------------------------------------------- Ran 31 tests in 11.307s commit 6862dc4994996e78ebf4516094c29aad66ceb524 Author: Damian Johnson Date: Mon Mar 18 18:20:35 2013 -0700 Skipping tests that fail when run as root Some of our integ tests rely on permission failures for files we lack access to, or path expansions. Running as root causes these to fail. Caught by Dererk... https://trac.torproject.org/7925 commit 98fd0a7ba4787340d05d0baf341594555da87d00 Author: Damian Johnson Date: Mon Mar 18 16:36:11 2013 -0700 The 'name' attribute of TarInfo classes can't be set in python 3 Workaround for the 'name' attribute being unwriteable in python 3. On reflection this is a little more elegant than our prior hack. :) Exception in thread Descriptor Reader: Traceback (most recent call last): File "/home/atagar/Python-3.3.0/Lib/threading.py", line 639, in _bootstrap_inner self.run() File "/home/atagar/Python-3.3.0/Lib/threading.py", line 596, in run self._target(*self._args, **self._kwargs) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/reader.py", line 435, in _read_descriptor_files self._handle_file(target, new_processed_files) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/reader.py", line 514, in _handle_file self._handle_archive(target) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/reader.py", line 557, in _handle_archive entry.name = target AttributeError: attribute 'name' of '_io.BufferedReader' objects is not writable commit c40ea0fb2f2188d540655cfc4c6975bffc9585b2 Author: Damian Johnson Date: Mon Mar 18 16:19:39 2013 -0700 Fixing open() mock for python 3 Our unit tests were broken under python 3 due to needing a slightly different mock for the open() function. ====================================================================== ERROR: test_mirror_mirror_on_the_wall_2 ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/unit/tutorial.py", line 107, in test_mirror_mirror_on_the_wall_2 tutorial_example() File "/home/atagar/Desktop/stem/test/data/python3/test/unit/tutorial.py", line 95, in tutorial_example for desc in parse_file(open("/home/atagar/.tor/cached-consensus")): FileNotFoundError: [Errno 2] No such file or directory: '/home/atagar/.tor/cached-consensus' commit bc155d6ee800c1741fb7b1b7780cf696217115f4 Author: Damian Johnson Date: Mon Mar 18 14:00:45 2013 -0700 Dropping the memory usage assertion when reading descriptors Our integ test for reading descriptors tries to check that we aren't consuming gobs of memory. Unfortunately this assertion seems to be platform dependent (it fails on OSX) so dropping the assertion. Yea, yea, I should be fixing it but we haven't had this issue for a very long while and it isn't likely to reoccure. commit c614080d911cec936a90b1f106e777f962ad8249 Author: Damian Johnson Date: Mon Mar 18 13:53:17 2013 -0700 Tor switched from server to microdescriptors in 0.2.3.3 commit bf5bac34abf95f1e12f0ce4030c305c364939c1f Author: Damian Johnson Date: Mon Mar 18 13:34:41 2013 -0700 Digest validation fails for 'non-ascii_descriptor' The 'non-ascii_descriptor' test descriptor fails digest validation due to being slightly modified. Finding that descriptor was a PITA so simply mocking out the digest validation rather than trying to dig up the original. commit b8d79e24b4599b59df1d68bd99273cebb11a8038 Author: Damian Johnson Date: Sat Mar 16 11:30:36 2013 -0700 Unescaped characters shouldn't be considered in further unescaping When we unescape responses we should process the string from start to end, and when something is unescaped it needs to be removed from further consideration. For instance, '\\t' should be unescaped to '\t', not a tab. Caught on... https://trac.torproject.org/8471 commit a124b62f26c8ae1cee01f780a0383389275ee873 Author: Damian Johnson Date: Sat Mar 16 10:12:57 2013 -0700 Supporting 'Unmeasured' in router status entries Spec commit c3860e3 added a new 'Unmeasured' key/value entry to the 'w' lines of v3 router status entries. commit cb7915bb4734742f487519dad6ce0ec2a84b5bb7 Author: Damian Johnson Date: Sat Mar 16 09:42:41 2013 -0700 Including param constraints for UseOptimisticData We were missing the bounds for the UseOptimisticData parameter. commit 2f90285dd88db3fdac5dc14441b7ffe19ce8da38 Author: Damian Johnson Date: Sat Mar 16 09:05:22 2013 -0700 Chaning GuardEvent to match spec revisions Nick has clarified GUARD events in... https://trac.torproject.org/7619 Updating our enum docs to reflect the revisions and changing our 'name' attribute to reflect that it could contain either a fingerprint or fingerprint/nickname pair. This is a similar pattern to what we use for ORConnEvent. commit 0ce65a43d68291388a5a0f61be2427d78ce7167f Author: Damian Johnson Date: Thu Mar 14 09:23:21 2013 -0700 Providing default ipv6-policy entries The server descriptor's newish 'ipv6-policy' lines are now documented as being reject-all if it's missing... A missing "ipv6-policy" line is equivalent to "ipv6-policy reject 1-65535". https://gitweb.torproject.org/user/nickm/torspec.git/commitdiff/ef5513b1ad3671153f20b3c3929e1acba1b873bc commit 8474971d99da327169e5827db660f8f1f91bbd59 Author: Damian Johnson Date: Wed Mar 13 09:38:19 2013 -0700 Dropping the V3Dir flag Nick just pushed a change noting that the V3Dir flag has never existed, so updating our enum too. https://gitweb.torproject.org/torspec.git/commitdiff/67768b16c247f937a4345af94beb58a4238b0730 commit 89e8fb8f37ebaad8d2009fedba766d1d42831dd8 Author: Damian Johnson Date: Wed Mar 13 09:14:33 2013 -0700 More uniformly renaming to 'tutorials' Several of our resources, including individual tutorial pages, were still under 'tutorial'. I'm also dropping the old 'tutorial.rst' (I'm not getting any replies to the tor-dev@ email and having a stale copy is confusing). commit c1abfa73fc24528713be233347cbd0f8796c537e Author: Damian Johnson Date: Wed Mar 13 09:09:54 2013 -0700 'Tortoise and the Hare' tutorial Adding a tutorial for event listening. This has the most substantial example yet, making a curses graph for tor's bandwidth usage. commit 92d395c8515e72e9c9b9b89d9e84512419f51aa4 Author: Damian Johnson Date: Tue Mar 12 06:55:22 2013 -0700 Tutorials still referenced a control_port arg Oops. When updating the tutorial tests I guess that I forgot to update the examples. :( commit fb35fc7a9dd9d0217bedbc38bda51f58579d5f77 Author: Damian Johnson Date: Mon Mar 11 19:01:16 2013 -0700 Filling in a description for MEASURE_TIMEOUT Nick added a description to address... https://trac.torproject.org/7626 I'm not quite sure what it means (does "see how long it takes" mean measure the time to make the circuit? or the time it takes before being timed out like the name suggests?), so just copying what he put in the spec. commit 98f878087dfb5eeac3fd2720b47b4ffe1fe1bc75 Author: Damian Johnson Date: Mon Mar 11 18:28:24 2013 -0700 Dropping 'aka' from tutorial labels The labels look a little nicer this way. Suggested by Josh Barr. commit 2c9710060e6194e30d6971377ddc80fa681eeaa9 Author: Damian Johnson Date: Mon Mar 11 18:07:54 2013 -0700 More changes for tutorial rename Drats, I pushed the prior commit before I had intended to. Fixing the buttons too to reflect the rename. commit 48938b1d95d25ef255c19cfdd7a8a30714a80b76 Author: Damian Johnson Date: Mon Mar 11 00:22:43 2013 -0700 Pluralizing the tutorials page Renaming the 'tutorial' page to 'tutorials'. I can't remove the old page yet since I just sent it to tor-dev@, but now that its only link is swapped over we should be able to move it in a few weeks. commit 74b80d61fe3279ca006ecae86271557523e33967 Author: Damian Johnson Date: Sun Mar 10 23:07:48 2013 -0700 Making all labels links Among the tutorials the labels (ie. the text summary for a section) was a link but on the download and example pages they weren't. Standardizing on them being links. commit 7d13f767c08e35d55324186832deb7c6df6de62d Author: Damian Johnson Date: Sun Mar 10 22:59:23 2013 -0700 More clearly stating tutorial topics While our cutesy titles are fun, they shouldn't get in the way of our tutorial page making sense. Dropping them to be a subheading, in favour of a clearer summary of what the tutorial's about. commit 4d89032ba6f7d1c2babd54c593668c8e74b23f98 Author: Damian Johnson Date: Sun Mar 10 22:40:11 2013 -0700 Adding an examples section to our site Example sections containing projects and scripts using stem. Presently this is pretty small - hopefully it'll grow soon! commit 99dbc604fb6781163d2cdf5262b759cbdc62c71a Author: Damian Johnson Date: Sun Mar 10 17:41:58 2013 -0700 Making stem.util.term public Now that we're using it for a tutorial we should vend this module. I'm a little unsure about the name of the format() function since there's a builtin by that name, but I'm having a hard time thinking of something better. commit ecc5f7bcd7d9669521e3b3b9275b18dff98c4270 Author: Damian Johnson Date: Sun Mar 10 11:28:58 2013 -0700 'To Russia With Love' tutorial Soviet themed tutorial for programmatically starting tor and reading a site through it. I've been wanting to write this for quite a long time... https://trac.torproject.org/7505 commit 860e89bed7a4d6769d9d4376a770e9af19de684b Author: Damian Johnson Date: Sat Mar 9 23:13:25 2013 -0800 Dropping 'last updated' timestamp from site While I'd kinda like to have this on our site, it's stupidly broken. This is probably due to sphinx caching (it only makes static files for things that have changed). Presently it says that our site was last updated on December 31st which is... very wrong. No date is better than the wrong date so dropping it. https://trac.torproject.org/7780 commit a7fae56f9dab6cf766699c229edb2fd0531b7bd9 Merge: d45f89d aab3bf5 Author: Damian Johnson Date: Sat Mar 9 23:10:26 2013 -0800 Marking stem.util modules and functions as being private The stem.util space has several things that I'd rather not vend. Things that are either likely to change in the future or simply don't make sense outside of stem's internal use. Marking functions and some entire modules as being private. commit aab3bf57a87bb09d4d572195e568e73da489f38a Author: Damian Johnson Date: Sat Mar 9 23:07:23 2013 -0800 Marking stem.util.str_tools functions as being private Swapping several of the functions that we don't want to vend as being private. The to_bytes() and to_unicode() functions in particular are simply there for python 3.x support, so they'll disappear if/when we drop python 2.x support. commit 7f4b7a93d2ff55388a8566919fc19baf2f1a48df Author: Damian Johnson Date: Sat Mar 9 22:47:05 2013 -0800 Marking the log, proc, term, and tor_tools utils as being private More modules that I'd rather not yet vend to stem users. They're more than welcome to use them, but I can't yet promise a stable API (and in many of these cases there's little reason they ever should be used outside of stem). commit ec0b5538a95a39f7e2b3a3fbcf47a03739f16a6b Author: Damian Johnson Date: Sat Mar 9 22:33:07 2013 -0800 Notice to stem users not to use stem.util.ordereddict The ordereddict module is to allow us to support python 2.5 and 2.6. Adding a notice that it may be dropped in the future. commit f8b538b00bd16fece4d368c892a7977dd0f8260b Author: Damian Johnson Date: Sat Mar 9 22:29:37 2013 -0800 Making several stem.util.connection functions private The connection module has several functions that I'd rather not vend. Marking them as private to discourage their use. commit 26c854e7c032d2f7df822316f56b0d62075d29d2 Author: Damian Johnson Date: Sat Mar 9 22:18:22 2013 -0800 Renaming stem.util.connection.get_mask() Renaming the function to get_mask_ipv4() to match the module conventions now that we have ipv6 support. commit 08c5f3d6020c860077125a976a336f076f9b9e7c Author: Damian Johnson Date: Fri Mar 8 09:37:35 2013 -0800 Renaming stem.util.connection.is_valid_ip_address() Changing is_valid_ip_address() to is_valid_ipv4_address() so it'll be in better keeping with the modules conventions now that it also supports IPv6. We might add is_valid_ip_address() later to check if it's an IPv4 *or* IPv6 address. commit d45f89dbce487aec2a3988138966387da55adaf5 Author: Damian Johnson Date: Fri Mar 8 07:57:28 2013 -0800 Clarifying requirements for timing out launch_tor() The signal timeout we were doing isn't just *nix specific, but relies on global state. It turns out that no languages really have a good method for doing this sort of arbitrary timeout - pity... http://eli.thegreenplace.net/2011/08/22/how-not-to-set-a-timeout-on-a-computation-in-python/ commit a564ae2a6c24aebb822e777e7dea8bda3cffcf68 Author: Damian Johnson Date: Thu Mar 7 09:33:37 2013 -0800 stem.process.launch_tor() doesn't work without stdout logging The launch_tor() function uses tor's stdout logs to figure out when it has finished bootstrapping. This, of course, doesn't work if tor isn't logging to stdout... https://trac.torproject.org/7677 Changing launch_tor_with_config() to slip in a 'Log NOTICE stdout' option so it always works, and adding a note to launch_tor() warning users of the requirement. commit 32d39439b3a1cb2a72b2efc9d3010d2c90b174ee Author: Damian Johnson Date: Wed Mar 6 20:12:06 2013 -0800 Making the arrived_at arg of events optional The kwargs of stem.response.convert() are documented as being optional, however the arrived_at argument for events was mandatory. Caught by meejah. commit f40ab20110a1979efbffaa7854eca5ff77a5e9c3 Author: Damian Johnson Date: Wed Mar 6 19:56:57 2013 -0800 Dropping the _get_pid() helper from stem.process Huh, that's odd. There's a comment saying that I tried to use the mocking module for os.getpid() but it was a no-go, necessitating an ugly hack. I just tried again and it worked - maybe a bug we have since fixed with the mocking? commit de9b6220b21f0c09239181ab17a9f7554ea601eb Author: Damian Johnson Date: Wed Mar 6 09:39:34 2013 -0800 Mocking didn't properly revert static methods When working on our tutorials I expanded our mocking module to handle static methods. This worked well for our unit tests, but when we reverted the mocks we did so as standard rather than static methods. This in turn broke all of our integ tests that relied upon Controller.from_port(). This went undetected because it only manifests when *both* unit and integ tests are run together (and I've only been running one or the other recently). commit 27630c4d7f17052975238a0f7622ec3418bdbd5e Author: Damian Johnson Date: Tue Mar 5 11:01:56 2013 -0800 Replacing Controller's enable_caching constructor arg with setter It's clunky for cache toggling to be a constructor arg since most of our users will be using from_port() or from_socket_file() instead. commit 0c4cb0d9c0178a353df4894bb87ccb835bca2ace Merge: 954bced 7d4efa2 Author: Damian Johnson Date: Mon Mar 4 00:11:59 2013 -0800 Revising the tutorials Changing our tutorial page to be a friendly listing of the tutorials rather than a wall of text. Also rewrote our present tutorials and their corresponding tests. commit 7d4efa2102727a1135bd19204e7d35f7b8900ec4 Author: Damian Johnson Date: Mon Mar 4 00:10:07 2013 -0800 Tests for descriptor tutorial examples Unit tests for our new 'mirror, mirror' tutorial examples, and writing our prior tests to conform with the way-less-sucky style that separates the mocking from the example. commit a86d250b6702f7dae9ed74778a6e0a5cb2f9358a Author: Damian Johnson Date: Sun Mar 3 23:11:30 2013 -0800 Rewriting the descriptor tutorial Replacing the 'Mirror, Mirror' tutorial with a new one that gives a better overview of the various descriptors and how to get/use them. This keeps the old example (listing the fastest exits), but otherwise is a full rewrite. commit fce9b2e92f4d6bf9a01431f993fd0e42dcd6f6ad Author: Damian Johnson Date: Sun Mar 3 16:12:24 2013 -0800 Adding microdescriptors to our api docs commit 478ba3a44f3291367eeaea9ba9ce0f4cab24f517 Author: Damian Johnson Date: Sat Feb 23 11:49:02 2013 -0800 Cleaning up 'The Little Engine that Could' tutorial test Of all our tests the tutorial example tests stick out as being... just aweful. They work I guess but their mix of heavy mocking and the example being tested is both damn confusing and unmaintainable. Breaking it up so the mocking is entirely separate from the tutorial code. commit 1749855559044f2547920fa206f72c6fdf7c86f4 Author: Damian Johnson Date: Fri Feb 22 08:53:21 2013 -0800 Mocking stdout for tutorial tests Mocking stdout so the tests can be a little more faithful to the tutorial examples. commit 63bad222dc4de6b7e3c8510e4704ac7af0cabcdb Author: Damian Johnson Date: Fri Feb 22 08:55:14 2013 -0800 Revising 'The Little Relay that Could' tutorial Changing small bits if the tutorial page and 'hello world' tutorial, adding a little detail here and cleaning up the wording there. commit a3e486856810e99cfa065df9ebf24961ec7a3bff Author: Damian Johnson Date: Tue Feb 19 09:23:11 2013 -0800 Rearranging contents The contents determine the next/previous links in the page's upper right. In general these don't make sense (there's not really a logical order to our API), but for the tutorial it's nice if they link to each other. commit f4d2b20b2d954852faa59d39ce9f38f7cb11dcd2 Author: Damian Johnson Date: Mon Feb 18 22:12:01 2013 -0800 Moving tutorials to sub-pages Making pages for each of the tutorials and making tutorial.rst a pretty listing similar to the download page. This should hopefully make this page far more approachable by newcomers. commit 0fde6ad67f7250b3de573726343844eb7ea573fe Author: Damian Johnson Date: Mon Feb 18 08:58:10 2013 -0800 Moving download images into subdirectory I'm about to use '_static/section' for tutorial images, so moving the download page's resources to keep them separate. commit 954bced4f16b05bd6a4923cfe3a8297f0e3347a6 Author: Damian Johnson Date: Sun Mar 3 15:08:01 2013 -0800 Only first 'transport' line was being used The extrainfo descriptor parser only utilized the first 'transport' line, causing further lines to be ignored. Caught by asn. commit 7f3f8722d99b2747cafd2247b14e362c743d5290 Author: Damian Johnson Date: Sat Mar 2 22:16:20 2013 -0800 Adding get_microdescriptors() method to the Controller Controller method to fetch all microdescriptors. This is modeled after its counterparts for server descriptors and network status documents. However, as mentioned in 'https://trac.torproject.org/8323', the controller interface presently lacks a method to get them. In the meantime we're reading them from disk. commit db19cafb398e9626502cf9d1226803a5fb4cb4b1 Author: Damian Johnson Date: Sat Mar 2 19:57:57 2013 -0800 Replacing get_server_descriptor() and get_network_status() tests Our integration tests for get_server_descriptor() and get_network_status() have been skipped for quite some time due to... https://trac.torproject.org/7163 Replacing them with a simpler (and hopefully more reliable) test like what we're now doing for get_microdescriptor(). commit f5c7c226321f801e448f759e2ba3617c0ef5814b Author: Damian Johnson Date: Sat Mar 2 19:28:06 2013 -0800 Adding get_microdescriptor() method to the Controller Adding a method to query individual microdescriptors. This is very similar to its server descriptor and network status counterparts. commit 23d2332b2514fd9194e64cb1859614014087394b Author: Damian Johnson Date: Thu Feb 28 09:14:01 2013 -0800 Tor microdescriptor support Adding parsing support for tor microdesriptors... https://trac.torproject.org/8253 These have replaced server descriptors as the self-published descriptor content tor fetches by default. They're a bit clunckier to use compared to server descriptors, and lack much of the information controllers might be interested in, but the lighter weight of microdescriptors make them better for the overall network. Next up is to add support for these to our Controller. Unfortunately the tor control protocol only supports querying microdescriptors individually... https://trac.torproject.org/8323 commit d25f946d06a26e2ee0947eca5b1944f190e6299f Author: Damian Johnson Date: Sat Feb 23 12:16:32 2013 -0800 Notice that server descriptors are no longer available by default As of Tor 0.2.3.25 relays stopped fetching server descriptors by default. This is an important thing to know since it makes get_server_descriptor() and get_server_descriptors() useless unless you set 'UseMicrodescriptors 0'. velope has been emphatically pointing out that we want to discourage stem developers from disabling microdescriptors so advising them against it. commit ec3d0c01c9ba4aa30541ceb19e499fbe732453cb Author: Damian Johnson Date: Wed Feb 27 07:35:00 2013 -0800 Don't pass unsupplied optional args to wrapped file Karsten reports that the descriptor reader is failing for him with... ParsingFailure: readlines() takes exactly 1 argument (2 given) This is because to add python 3.x support we had to wrap the parse_file()'s file argument with a wrapper. In most cases this wrapper is a passthrough, but it converts the read() and readlines() output to unicode. I based the methods of the wrapper on a StringIO file object, but evidently other files (most likely the tarfile) deviate a bit. Addressing the readlines() instance and one other that might cause troubles. commit 3872befa5c62a91eb83d0d1376741b51a91acdb6 Author: Damian Johnson Date: Sat Feb 23 16:07:19 2013 -0800 Renaming controller and socket keyword arguments Less is more. I'm not sure why I called these keyword arguments 'control_addr', 'control_port', and 'socket_path' but these will be highly used constructors. We should aim to have them be as user friendly a possible, and that means less verbose. commit be9a5323a37ea0f1b7d497d7fc33e101453eb2cf Author: Karsten Loesing Date: Wed Feb 20 12:26:29 2013 +0100 Change fingerprints and digests to upper-case hex. commit 13de78a989b4e9ba68ffcecdd6022335baae4b39 Author: Damian Johnson Date: Mon Feb 18 07:42:46 2013 -0800 Catching unexpected exception from libc.setproctitle() Sathyanarayanan reports that on OSX libc.setproctitle() can raise an AttributeError... ERROR: test_set_process_name ---------------------------------------------------------------------- Traceback: File "/Users/sathya/Documents/stem/test/integ/util/system.py", line 412, in test_set_process_name stem.util.system.set_process_name(initial_name) File "/Users/sathya/Documents/stem/stem/util/system.py", line 774, in set_process_name _set_proc_title(process_name) File "/Users/sathya/Documents/stem/stem/util/system.py", line 831, in _set_proc_title libc.setproctitle(ctypes.byref(name_buffer)) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 366, in __getattr__ func = self.__getitem__(name) File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ctypes/__init__.py", line 371, in __getitem__ func = self._FuncPtr((name_or_ordinal, self)) AttributeError: dlsym(0x7fff6a41d1e0, setproctitle): symbol not found https://trac.torproject.org/8266 commit bdbb508ae2ffcaa5480992d33c28ec52a6e61b7e Author: Damian Johnson Date: Sun Feb 17 20:34:11 2013 -0800 Making get_process_name()'s ps lookup work on OSX The get_process_name() function uses a couple approaches to determine our process name, the first being to query ps. This tends to work pretty well across various platforms (barring Windows, of course), but there's a slight difference in the header on OSX/BSD. Accounting for this... Example ps output: stem git:(master) : ps -p 60437 -o args ARGS tor --socksport 9050 Caught by Sathyanarayanan on... https://trac.torproject.org/8266 commit 08529723708ed514f6f5e2489d3c1be8c0e9ae36 Author: Damian Johnson Date: Sun Feb 17 18:09:58 2013 -0800 Accounting for NULL access by ctypes Evidently accessing argc can raise a ValueError... https://trac.torproject.org/8266 commit 1a099106c43e37bf41f5c1f803f27cd2aad270f5 Author: Damian Johnson Date: Sun Feb 17 13:57:15 2013 -0800 Providing a string when str() is called on descriptors Python 2.x gets pretty confused when an object's __str__ method provides a unicode string. Calling... >>> str(desc) Traceback (most recent call last): File "", line 1, in UnicodeEncodeError: 'ascii' codec can't encode character u'\xab' in position 28: ordinal not in range(128) Providing an ascii str in python 2.x and unicode str in python 3.x. Thanks to Sathyanarayanan for the catch! commit d70a2d394f24e471379906e561d80fb47ecd2230 Author: Damian Johnson Date: Sun Feb 17 13:22:54 2013 -0800 Adding copyright headers Uggg I hate IP law. As pointed out by Juan on... https://trac.torproject.org/7954 ... we need copyright headers to properly comply with the requirements of being under the LGPL. I'm not looking forward to keeping this up to date (and likely won't), but oh well. I've added the header to all python source files except the unit and integ tests (patches welcome if someone wants to spend the time adding those too). These headers... * Declare a copyright from the year of the file's creation to now (2013). * Include Sean if he worked on it (he is the only contributor that hasn't made his contributions public domain to avoid copyright headaches). commit 4a2e7e74652c2c93654035bede41f203397ed16b Author: Damian Johnson Date: Sun Feb 17 12:25:43 2013 -0800 Cleaning up our TODO comments Several of our TODO comments were no longer relevant or could be expanded. commit 6927e68d80bf78d898afe321a599bf174e116e40 Author: Damian Johnson Date: Sat Feb 16 21:53:24 2013 -0800 Treat descriptor archive contents as individual files When the descriptor reader encountered an archive and read non-descriptor content it stopped reading. This has caused me almost two weeks of headaches in troubleshooting... https://trac.torproject.org/8049 Changing the reader's behaviour to instead handle each file within the archive separately. Thanks to Karsten for catching this! commit f83c7efce3380c60c81028802e198791aa30dfbd Author: Damian Johnson Date: Sat Feb 16 10:53:39 2013 -0800 Adding get_archive_path() method to descriptors We can't use a TarInfo's 'name' attribute for get_path() since that corresponds to its location within the archive. That said, I've often wanted both paths so both fixing get_path() for tarballs and adding a get_archive_path(). commit 8cdcb0883f0da8bb492da31cfea7155df5e9efed Author: Damian Johnson Date: Fri Feb 15 09:38:02 2013 -0800 Dropping the 'path' argument from stem.descriptor.parse_file() File objects have a 'name' attribute that we can use to guess the path. This isn't entirely reliable, but nothing is... http://stackoverflow.com/questions/2458676/absolute-path-of-a-file-object The path argument was only there to support the descriptor reader. Now that parse_file() is something for our users it's nice to get rid of arguments they can't use. commit 2a952ec98054cf656efd5d9b2bffa1960cdf5b42 Author: Damian Johnson Date: Sat Feb 9 15:29:00 2013 -0800 Accepting "NEVER" expiration in ADDRMAP events The expiry value in ADDRMAP events can be 'NEVER'. This is a little troublesome since it means that the field might or might not be quoted (making this unique among all tor events). Caught by Desoxy on 'https://trac.torproject.org/8162'. commit dba169f5cb3d28da29c7c2011ef5de7cc5825d37 Author: Damian Johnson Date: Sat Feb 9 13:07:23 2013 -0800 Fixing SingleLineResponse interlinking The SingleLineResponse class wasn't in the module's __all__, causing it to not appear in the sphinx output. commit 74b60f7552b587ea576db0be01e31e0e5531bb7d Author: Damian Johnson Date: Sat Feb 9 12:53:31 2013 -0800 Improving stem.response.convert() pydocs The convert() pydocs were pretty clunky. Replacing the listing with a nice table mapping the response_type to classes, like what we do elsewhere. commit 5f8b7b4231bcda7f69296821aa6417fd3e2ce5f3 Author: Damian Johnson Date: Sat Feb 9 12:35:11 2013 -0800 Adding a ControlMessage.from_str() function In discussions with Mike about using stem for txtorcon a major use has been response parsing. Using stem for this is dead easy, but requires a hack. Adding a function to negate the need for hackery. commit f96d5f64a94ecfa9d05b80e2b3a6c7c3ba78bcea Author: Damian Johnson Date: Fri Feb 8 08:32:44 2013 -0800 Noting '(Tor_internal)' addresses in the pydocs Noting that StreamEvents can have '(Tor_internal)' as a target address. Spec change: https://gitweb.torproject.org/torspec.git/commitdiff/3ad9d19e03bd816e1e0f0b9eeb839ee1eedcaedf commit ea5212863a97efa4f29e47280faf3c9566ec5349 Author: Damian Johnson Date: Fri Feb 8 08:29:22 2013 -0800 Using numeric 'flag-thresholds' values Now that the spec has been revised to specify numeric values we can provide 'flag => int/float' mappings (which are much nicer for our users). Spec change: https://gitweb.torproject.org/torspec.git/commitdiff/52d0eb4858ad3eb191df3afe324f43683467ae22 commit e8784466401a8120669128682b75eaa947e91c7a Author: Damian Johnson Date: Wed Feb 6 07:46:22 2013 -0800 Minor stylistic corrections Couple PEP8 bugs that slipped in concerning spacing between code and inline comments. commit 3687dde657dc8df40ffecadd45c4035ee3d44110 Author: Damian Johnson Date: Wed Feb 6 07:41:30 2013 -0800 Avoiding static /tmp usage Our tests had static /tmp paths at a couple places. Issue caught by Dererk and patch by Abhishek... https://trac.torproject.org/7926 commit 4e8aaa4daadb02db1686dde57c42c15aef6821e9 Author: Damian Johnson Date: Tue Feb 5 09:48:12 2013 -0800 Support for 'flag-thresholds' lines in network status votes Parsing the new 'flag-thresholds' in network status votes - thanks to Karsten for pointing this out. metrics-lib change: https://gitweb.torproject.org/metrics-lib.git/commitdiff/c2a0dbf8bf100a19660ad512b88d93f3d7c18a1e dir-spec addition: https://trac.torproject.org/8165 commit c6a9cde0f4eb5c627f5ba41f9d263be0c5854ae8 Author: Damian Johnson Date: Sun Feb 3 22:14:20 2013 -0800 Allowing for IPv4 'a' lines in router status entries Karsten reports on ticket #8036 that IPv4 addresses are indeed allowed on a router status entry's 'a' line. This is a little unfortunate since it means a less friendly attribute but not a big whoop. commit b8baf77c13154ccb6c43299a36979d812ca7b8be Author: Karsten Loesing Date: Mon Feb 4 06:17:28 2013 +0100 Try to make minor descriptor versions clearer. commit 6c99a28e83490537615de9388484c574aa1b85dd Author: Damian Johnson Date: Sun Feb 3 20:43:28 2013 -0800 Using port lists for addresses_v6 rather than ranges Huh, I wonder where I got the idea that 'a' lines had port ranges. Dropping that. According to the spec the 'a' lines should be parsed in the same way as 'or-address'. However, I suspect that the spec is a little off here - checking if it can contain IPv4 addresses... Caught by Karsten on... https://trac.torproject.org/8036 commit d44018a5566fe5c92a326a413768bdd8462c425e Author: Damian Johnson Date: Sun Feb 3 20:01:35 2013 -0800 Renaming check_whitespace.py to static_checks.py The check_whitespace.py module no longer... well, checks whitespace. Rather, it has become a dumping ground for all of the static checks that we do. Renaming it to something more appropriate. commit ea0b73a5aa221fadafc2ba718a0ef42e151e5ad6 Author: Damian Johnson Date: Sun Feb 3 13:00:56 2013 -0800 Providing alternative methods for parsing a NetworkStatusDocument Adding support in both the DescriptorReader and parse_file() function for three ways of parsing network status documents... a. Provide the router status entries (ie. the current behavior). b. Provide the document itself with the router status entries that it contains. This has the biggest cost in terms of upfront parsing time and memory usage, but provides the caller with everything they might want. c. Provide the document but skip reading the router status entries. A handy option of you just care about the document's header/footer. Now that we have these capability I'm further simplifying the descriptor API a bit. The network status docs encouraged users to use the NetworkStatusDocument constructors to achieve option 'b' above, but now that it's in the reader and parse_file() there's no reason for them to do that. Users should now *always* use either the DescriptorReader or parse_file(). If they don't then they're off the reservation. commit 479f535614081f097ee2e39ecde70ffaf658ba25 Author: Damian Johnson Date: Sun Feb 3 11:29:53 2013 -0800 Dropping Version.meets_requirements() in favour of comparisons Once upon a time you checked your requirements via simple comparisons... if my_version >= requirement: ... do stuff... I reluctantly changed this to a meets_requirements() method when we added the VersionRequirements class since it was no longer simple comparisons the __cmp__ method could handle. However, now that we're using rich comparison operators we can go back to the nicer style of comparisons. Apologies for any confusion this back-and-forth has caused. commit 30146d77243f7a25d54019d9d2641be258c444f8 Author: Damian Johnson Date: Sun Feb 3 11:12:57 2013 -0800 More succinct python 3 warning for parse_file() Read speeds and universal newline translation are both addressed by reading in binary mode. There's no need to have a separate warning for each. commit 9cd4c9feb479fe00987526c272a1b2a1853bd295 Author: Damian Johnson Date: Sat Feb 2 12:23:07 2013 -0800 Using binary mode for the controller socket file Yay! Now that I have a version of python 3 that doesn't segfault I can finish making our integ tests work. The socket file used for controller connections should be normalized to use binary mode. This is its behavior in python 2.x, and in 3.x having it in text mode can cause sadness. Exception in thread Tor Listener: Traceback (most recent call last): File "/home/atagar/Desktop/Python-3.3.0/Lib/threading.py", line 639, in _bootstrap_inner self.run() File "/home/atagar/Desktop/Python-3.3.0/Lib/threading.py", line 596, in run self._target(*self._args, **self._kwargs) File "/home/atagar/Desktop/stem/test/data/python3/stem/control.py", line 573, in _reader_loop control_message = self._socket.recv() File "/home/atagar/Desktop/stem/test/data/python3/stem/socket.py", line 115, in recv return recv_message(socket_file) File "/home/atagar/Desktop/stem/test/data/python3/stem/socket.py", line 539, in recv_message line = control_file.readline() File "/home/atagar/Desktop/Python-3.3.0/Lib/codecs.py", line 300, in decode (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 2005: invalid continuation byte After addressing this and a few encoding issues the controller integ tests now pass, but after we're done testing python spews out a dump following... *** glibc detected *** python3: munmap_chunk(): invalid pointer: 0x097f1620 *** At this point I'm pretty well persuaded that the python 3.x series leaves something to be desired in terms of stability. commit 3930f1f1eda99bcf22fe574b95665f8ef47ffdc4 Merge: 7921a46 2b2a645 Author: Damian Johnson Date: Sat Feb 2 10:12:02 2013 -0800 Python 3.x support Adding support for the python 3.x series. You can install the python 3 version of stem by running... python3 setup.py install The 2to3 conversion can be tested trough run_tests.py with the '--python3' argument. It passes all of the unit tests and the integ tests... er, don't technically fail. However, python 3.2 has a bug causing a segfault when it gets to the BaseController integ tests. Filed a ticket about it... http://bugs.python.org/issue17105 However, stem's descriptor functionality checks out and this issue has likely been addressed in later python releases so there's little point to hold off on merging. Ticket for python 3 support... https://trac.torproject.org/7843 commit 2b2a645ac778570d99961f9b9592769fdc9caa10 Author: Damian Johnson Date: Fri Feb 1 07:48:21 2013 -0800 Converting cookie auth token to unicode Well, this is dumb. Making a formatted string with ascii bytes includes the b'' wrapper in python 3. This broke our authentication calls, making calls like... AUTHENTICATE b'd55e81eb9c3a1e22a2db919ec2efd22df4aeb88ee0ab3d10e64dbb2450d06921' Converting the token to unicode to avoid this. ====================================================================== ERROR: test_authenticate_cookie ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/integ/connection/authentication.py", line 400, in _check_auth stem.connection.authenticate_cookie(control_socket, auth_arg) File "/home/atagar/Desktop/stem/test/data/python3/stem/connection.py", line 604, in authenticate_cookie raise CookieAuthRejected(str(auth_response), cookie_path, False, auth_response) stem.connection.CookieAuthRejected: Invalid hexadecimal encoding. Maybe you tried a plain text password? If so, the standard requires that you put it in double quotes. During handling of the above exception, another exception occurred: Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/integ/connection/authentication.py", line 303, in test_authenticate_cookie self._check_auth(auth_type, auth_value) File "/home/atagar/Desktop/stem/test/data/python3/test/integ/connection/authentication.py", line 411, in _check_auth failure_msg = _get_auth_failure_message(auth_type) File "/home/atagar/Desktop/stem/test/data/python3/test/integ/connection/authentication.py", line 99, in _get_auth_failure_message raise ValueError("No methods of authentication. If this is an open socket then auth shouldn't fail.") ValueError: No methods of authentication. If this is an open socket then auth shouldn't fail. commit 384411b26dd140e3d4b181f3c2b1a48057238096 Author: Damian Johnson Date: Fri Feb 1 07:16:25 2013 -0800 Replacing file() with open() I'm not sure why we were using file() at one point rather than open(), but it makes python 3 sad... ====================================================================== ERROR: test_authenticate_cookie ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/integ/connection/authentication.py", line 303, in test_authenticate_cookie self._check_auth(auth_type, auth_value) File "/home/atagar/Desktop/stem/test/data/python3/test/integ/connection/authentication.py", line 400, in _check_auth stem.connection.authenticate_cookie(control_socket, auth_arg) File "/home/atagar/Desktop/stem/test/data/python3/stem/connection.py", line 583, in authenticate_cookie cookie_data = _read_cookie(cookie_path, False) File "/home/atagar/Desktop/stem/test/data/python3/stem/connection.py", line 877, in _read_cookie with file(cookie_path, 'rb', 0) as f: NameError: global name 'file' is not defined commit a047a74abef0dc9fedf463c093c14defe958d81b Author: Damian Johnson Date: Fri Feb 1 07:12:04 2013 -0800 Accounting for ascii/unicode for network status documents Woohoo! Last descriptor type. Unlike the other descriptor types callers are encouraged to sometimes use our NetworkStatusDocument classes directly so swapping the input to unicode if we get ascii. With this all of the descriptor integ tests now pass with python 3! commit 284797ace4cf0b8e4469454b65809fc66c7ae1d2 Author: Damian Johnson Date: Fri Feb 1 06:57:54 2013 -0800 Pydoc missing version from descriptor type listing The 'network-status-microdesc-consensus-3' listing was missing the '1.0'. commit 4815afe52c4af2d4cb3d687b35b7012239d7471b Author: Damian Johnson Date: Thu Jan 31 09:54:27 2013 -0800 Using stem.descriptor.parse_file() for extrainfo integ tests Going through parse_file() so we do the proper unicode conversion. ====================================================================== ERROR: test_cached_descriptor ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/integ/descriptor/extrainfo_descriptor.py", line 150, in test_cached_descriptor for desc in stem.descriptor.extrainfo_descriptor._parse_file(descriptor_file): File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/extrainfo_descriptor.py", line 155, in _parse_file extrainfo_content = stem.descriptor._read_until_keywords("router-signature", descriptor_file) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/__init__.py", line 350, in _read_until_keywords line_match = KEYWORD_LINE.match(line) TypeError: can't use a string pattern on a bytes-like object commit 3afa4346d102d99eab93929be738cdb594105d9b Author: Damian Johnson Date: Thu Jan 31 09:44:18 2013 -0800 Using binary mode when reading descriptors Now Damian, repleat after me: text mode is bad. In python 2.x text mode and binary mode seem to be indistinguishable, but in python 3 there's one tiny little difference: text mode is around 33x slower. The integ test that read the cached-consensus took over five minutes (by comparison to ten seconds with python 2.7), and in one case simply hung for twenty minutes before I killed it. I'm not aware of any disadvantage to using binary mode, so opting for that. commit d9d46cb5b35f586bcf8025cb7682c4ad79e57e15 Author: Damian Johnson Date: Thu Jan 31 08:31:53 2013 -0800 Skipping deletion of pyc in __pycache__ I disabled the deletion of orphaned pyc files when testing python 3 but on reflection that wasn't enough. Python 2.x test runs still delete the python 3 bytecode. Changing the orphaned check to skip those files. commit bec2e9721e228f73c8b1438f1d398182893df27f Author: Damian Johnson Date: Thu Jan 31 08:22:38 2013 -0800 Fixing server descriptor test expecting unicode One of the server descriptor integ tests had a failing assertion because the expected text was ASCII bytes and the descriptor content was unicode. Fixing the test and moving the to_unicode helper to str_tools where it belongs. ====================================================================== FAIL: test_non_ascii_descriptor ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/integ/descriptor/server_descriptor.py", line 221, in test_non_ascii_descriptor self.assertEquals(expected_contact, desc.contact) AssertionError: '2048R/F171EC1F Johan BlÃ¥bäck ã\x81\x93ã\x82\x93ã\x81«ã\x81¡ã\x81¯' != '2048R/F171EC1F Johan Blåbäck こんにちは' - 2048R/F171EC1F Johan BlÃ¥bäck こんにちは + 2048R/F171EC1F Johan Blåbäck こんにちは commit f4ee5ab36ee512f7cb904bb371f593f7e2f1f8a6 Author: Damian Johnson Date: Wed Jan 30 09:39:45 2013 -0800 Checking that to_bytes has unicode before converting Adding a check to the to_bytes() helper so we don't attempt to convert ASCII bytes to ASCII bytes (which doesn't work so well). ====================================================================== ERROR: test_old_descriptor ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/integ/descriptor/server_descriptor.py", line 120, in test_old_descriptor desc = stem.descriptor.server_descriptor.RelayDescriptor(descriptor_contents) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/server_descriptor.py", line 642, in __init__ self._validate_content() File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/server_descriptor.py", line 687, in _validate_content key_der_as_hash = hashlib.sha1(stem.util.str_tools.to_bytes(key_as_bytes)).hexdigest() File "/home/atagar/Desktop/stem/test/data/python3/stem/util/str_tools.py", line 73, in to_bytes return _to_bytes(msg) File "/home/atagar/Desktop/stem/test/data/python3/stem/util/str_tools.py", line 54, in _to_bytes return codecs.latin_1_encode(msg)[0] TypeError: Can't convert 'bytes' object to str implicitly commit 46f1bfb98cee79f9a00b76513d2c70df1824bb33 Author: Damian Johnson Date: Wed Jan 30 09:36:09 2013 -0800 Changing is_python_2* prereq checks to include python 3 The is_python_26 and is_python_27 were checking if we were 2.6-2.x or 2.7-2.x. On reflection it makes more sense for these to be '2.y and above' checks rather than '2.y and above in the 2.x series'. commit 6c855996c0daed3b6db222996dd651534c134c0d Author: Damian Johnson Date: Wed Jan 30 09:32:03 2013 -0800 Providing ASCII bytes to hashlib.sha1() Another unicode/ASCII bytes conversion issue... ====================================================================== ERROR: test_metrics_descriptor ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/integ/descriptor/server_descriptor.py", line 89, in test_metrics_descriptor self.assertEquals("2C7B27BEAB04B4E2459D89CA6D5CD1CC5F95A689", desc.digest()) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/server_descriptor.py", line 666, in digest digest_hash = hashlib.sha1(for_digest) TypeError: Unicode-objects must be encoded before hashing commit 9ff618eebed92e5646090ce3faa802763a5f7720 Author: Damian Johnson Date: Wed Jan 30 09:10:18 2013 -0800 Skipping newline translation for descriptor integ tests Using a custom open() call for python 3's integ tests to prevent newline translation (and the resulting test failures). commit 8a3cfb57e1982d26bafe6289f8ed9dea47192604 Author: Damian Johnson Date: Wed Jan 30 08:53:06 2013 -0800 Checking for 2to3 and python3 when needed Warning the user if 2to3 or python3 aren't in our PATH when the user provides the '--python3' testing argument. commit 0288267be5a325aadd078a487751cf4ce83a2f73 Author: Damian Johnson Date: Wed Jan 30 08:24:59 2013 -0800 Skip universal newline translation in descriptor reader Python 3 introduces universal newline translation, converting '\n', '\r', and '\r\n' into the local system's newline style. This is a really neat feature and will solve many-a-headaches... but not for us. We conform to the tor spec which specifies when CRLF appears verses other newline types. Universal newline translation broke our ability to read the 'cr_in_contact_line' example which has multiple '\r' within a contact line (https://trac.torproject.org/5637). Fixing the reader to disable newline translation and adding a warning to our parse_file() pydocs. commit d0f4a0c429b1e3a2bead48403605bb361ec66832 Author: Damian Johnson Date: Wed Jan 30 08:07:50 2013 -0800 Adding --python3 to the run_tests.py help output commit b4c4835d957463d6a453d9d7f7ad6007627abc96 Author: Damian Johnson Date: Tue Jan 29 09:03:30 2013 -0800 Normalizing descriptor handling as unicode Our python 3 descriptor integration tests were having troubles due to the ASCII/unicode switch. Adding a file interceptor so we always parse descriptors as unicode. Yes, yes, I know. Ewwww. I definitely don't like the _UnicodeReader helper so suggestions welcome for an alternative. commit 9d53a2854178d12222480958795876e45ec0f1d6 Author: Damian Johnson Date: Tue Jan 29 08:54:03 2013 -0800 Improving error output for test_skip_nondescriptor_contents The descriptor reader's test_skip_nondescriptor_contents integ test is especially tickle since it can fail whenever there's new non-descriptor content in the test/integ/descriptor/data directory. Improving its error output to say what it expected and got rather than simply saying that the counts mismatch. commit e99dc2b618cb33d5e23166510d224429b8f67046 Author: Damian Johnson Date: Sun Jan 27 19:53:32 2013 -0800 Accounting for tarfile.is_tarfile() raising AttributeErrors When we lack read permissions the tarfile.is_tarfile() funcion raises an AttributeError with python 3 (rather than an IOError as it does with python 2). This is most likely a bug... http://bugs.python.org/issue17059 Working around it for now. commit 313e5251da4a390fa283c440ffbe0f0ef048ed38 Author: Damian Johnson Date: Sun Jan 27 19:31:42 2013 -0800 Skipping orphaned file check if testing with python 3 The orphaned *.pyc check is both broken with python 3 and pointless since it's an exported copy of the python 2 codebase (we're not acually making changes to it). commit 2850199fc513fc4a276e16eaa59ccbc2dc69ea23 Author: Damian Johnson Date: Sun Jan 27 19:21:20 2013 -0800 Process renaming requires ASCII bytes Yet another unicode/ASCII compatability issue... ====================================================================== ERROR: test_set_process_name ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/integ/util/system.py", line 409, in test_set_process_name stem.util.system.set_process_name("stem_integ") File "/home/atagar/Desktop/stem/test/data/python3/stem/util/system.py", line 764, in set_process_name _set_prctl_name(process_name) File "/home/atagar/Desktop/stem/test/data/python3/stem/util/system.py", line 809, in _set_prctl_name name_buffer.value = process_name TypeError: str/bytes expected instead of str instance During handling of the above exception, another exception occurred: Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/integ/util/system.py", line 412, in test_set_process_name stem.util.system.set_process_name(initial_name) File "/home/atagar/Desktop/stem/test/data/python3/stem/util/system.py", line 764, in set_process_name _set_prctl_name(process_name) File "/home/atagar/Desktop/stem/test/data/python3/stem/util/system.py", line 809, in _set_prctl_name name_buffer.value = process_name TypeError: str/bytes expected instead of str instance commit 7d1e8a4ffc08ca3b5e7fb34d3b025ec3c8755497 Author: Damian Johnson Date: Sun Jan 27 19:08:40 2013 -0800 Returning unicode from stem.util.system.call() The call() method is the basic call for shelling out and, like tor's stdout earlier, its output was ASCII bytes. This is fine in python 2.x, but causes problems for python 3.x. Both are fine with this being unicode output so going with that. ====================================================================== ERROR: test_get_cwd ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/integ/util/system.py", line 334, in test_get_cwd self.assertEquals(None, stem.util.system.get_cwd(99999)) File "/home/atagar/Desktop/stem/test/data/python3/stem/util/system.py", line 528, in get_cwd elif results[0].endswith("No such process"): TypeError: expected an object with the buffer interface commit 6a85a34c753f34c7cc79f4ad4e9f67d92bce1046 Author: Damian Johnson Date: Sun Jan 27 19:01:32 2013 -0800 Converting str stats for proc tests The proc tests were comparing strings with floats. It coincidentally passed, but wasn't really testing what we wanted. In python 3.x the comparisons fail due to the type mismatch. ====================================================================== ERROR: test_get_stats ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/integ/util/proc.py", line 74, in test_get_stats self.assertTrue(utime > 0) TypeError: unorderable types: str() > int() commit c2b92450016ccfbe46edec151136c8a8a9f694cb Author: Damian Johnson Date: Sun Jan 27 18:18:45 2013 -0800 Disabling newline translation in our socket In python 3 the socket file object tries to be helpful by translating newline characters. By default '\n', '\r', and '\r\n' are all translated to '\n'. Tor uses '\r\n' newlines and we check for this so disabling universal newline translation. ====================================================================== ERROR: test_get_pid_by_port_netstat ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/stem/connection.py", line 321, in authenticate protocolinfo_response = get_protocolinfo(controller) File "/home/atagar/Desktop/stem/test/data/python3/stem/connection.py", line 800, in get_protocolinfo protocolinfo_response = _msg(controller, "PROTOCOLINFO 1") File "/home/atagar/Desktop/stem/test/data/python3/stem/connection.py", line 837, in _msg return controller.recv() File "/home/atagar/Desktop/stem/test/data/python3/stem/socket.py", line 114, in recv return recv_message(socket_file) File "/home/atagar/Desktop/stem/test/data/python3/stem/socket.py", line 511, in recv_message raise stem.ProtocolError("All lines should end with CRLF") stem.ProtocolError: All lines should end with CRLF During handling of the above exception, another exception occurred: Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/integ/util/system.py", line 245, in test_get_pid_by_port_netstat elif not runner.is_ptraceable(): File "/home/atagar/Desktop/stem/test/data/python3/test/runner.py", line 413, in is_ptraceable tor_version = self.get_tor_version() File "/home/atagar/Desktop/stem/test/data/python3/test/runner.py", line 567, in get_tor_version control_socket = self.get_tor_socket() File "/home/atagar/Desktop/stem/test/data/python3/test/runner.py", line 534, in get_tor_socket stem.connection.authenticate(control_socket, CONTROL_PASSWORD, self.get_chroot()) File "/home/atagar/Desktop/stem/test/data/python3/stem/connection.py", line 323, in authenticate raise IncorrectSocketType("unable to use the control socket") stem.connection.IncorrectSocketType: unable to use the control socket commit 6219ed6e6c517e0ce7d25b5d54355ed85329615e Author: Damian Johnson Date: Sun Jan 27 18:02:49 2013 -0800 Making sure that the socket's file is writeable In python 2.x the socket.makefile() method *should* only be readable (according to the default argument) but I've been using it for write operations too all this time. In python 3.x doing this raises "io.UnsupportedOperation: not writable" so guess we gotta fix it now. :) ====================================================================== ERROR: test_get_connections ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/integ/util/proc.py", line 92, in test_get_connections elif not test.runner.get_runner().is_ptraceable(): File "/home/atagar/Desktop/stem/test/data/python3/test/runner.py", line 413, in is_ptraceable tor_version = self.get_tor_version() File "/home/atagar/Desktop/stem/test/data/python3/test/runner.py", line 567, in get_tor_version control_socket = self.get_tor_socket() File "/home/atagar/Desktop/stem/test/data/python3/test/runner.py", line 534, in get_tor_socket stem.connection.authenticate(control_socket, CONTROL_PASSWORD, self.get_chroot()) File "/home/atagar/Desktop/stem/test/data/python3/stem/connection.py", line 321, in authenticate protocolinfo_response = get_protocolinfo(controller) File "/home/atagar/Desktop/stem/test/data/python3/stem/connection.py", line 800, in get_protocolinfo protocolinfo_response = _msg(controller, "PROTOCOLINFO 1") File "/home/atagar/Desktop/stem/test/data/python3/stem/connection.py", line 836, in _msg controller.send(message) File "/home/atagar/Desktop/stem/test/data/python3/stem/socket.py", line 82, in send send_message(self._socket_file, message, raw) File "/home/atagar/Desktop/stem/test/data/python3/stem/socket.py", line 422, in send_message control_file.write(message) io.UnsupportedOperation: not writable commit 1e2e6204eb8ba618e79c7bb1459e53e5154147fa Author: Damian Johnson Date: Sun Jan 27 17:43:22 2013 -0800 Socket closures raise a ValueError in python 3 Evidently reading from a closed socket file raises a ValueError rather than socket.error in python 3... Exception in thread Tor Listener: Traceback (most recent call last): File "/usr/lib/python3.2/threading.py", line 736, in _bootstrap_inner self.run() File "/usr/lib/python3.2/threading.py", line 689, in run self._target(*self._args, **self._kwargs) File "stem/control.py", line 573, in _reader_loop control_message = self._socket.recv() File "stem/socket.py", line 114, in recv return recv_message(socket_file) File "stem/socket.py", line 467, in recv_message line = control_file.readline() File "/usr/lib/python3.2/socket.py", line 269, in readinto self._checkClosed() ValueError: I/O operation on closed file. commit 7507edec828603a860d9ebf0cbb049060d32fc3e Author: Damian Johnson Date: Sun Jan 27 17:41:08 2013 -0800 Converting tor initialization lines to unicode The stdout of the tor process is an ASCII byte stream, causing our following regex search() call to error. commit 8b95332960e5343dd9b716ef25682548f0de9d7e Author: Damian Johnson Date: Sun Jan 27 16:32:06 2013 -0800 Mocking for open() broken in python 3 In python 2.x there's both __builtin__.open() and io.open(), but in python 3.x there's just io.open() (which is aliased by builtin). Accounting for this when we mock it for the descriptor reader unit tests. I'm a little puzzled why the mocking wasn't failing more spectacularly before (the __builtin__ import should always fail on python 3), but oh well - fixed now. With this the unit tests finally pass with python 3!!! ====================================================================== ERROR: test_load_processed_files ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/unit/descriptor/reader.py", line 52, in test_load_processed_files self.assertEquals(expected_value, stem.descriptor.reader.load_processed_files("")) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/reader.py", line 179, in load_processed_files with open(path) as input_file: IOError: [Errno 2] No such file or directory: '' ====================================================================== FAIL: test_get_connections ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/unit/util/proc.py", line 171, in test_get_connections self.assertEquals(expected_results, proc.get_connections(pid)) AssertionError: Lists differ: [('17.17.17.17', 4369, '34.34.... != [] First list contains 2 additional elements. First extra element 0: ('17.17.17.17', 4369, '34.34.34.34', 8738) + [] - [('17.17.17.17', 4369, '34.34.34.34', 8738), - ] commit 8d5d2f14edf92c7240aae0b7894fea6e27b41699 Author: Damian Johnson Date: Sun Jan 27 16:17:34 2013 -0800 Converted _required_fields() encounter TypeErrors The server and extrainfo descriptors' _required_fields() functions get converted by 2to3 into a line that tries to concatenate a list with a tuple (raising a TypeError). On reflection the converted version which uses list comprehension is a little nicer than filter(), so using a variant of that which makes everyone happy. commit 264c9e673f71dde707a03ec6b24931f1bd0f3fa4 Author: Damian Johnson Date: Sun Jan 27 16:02:06 2013 -0800 Accounting for different exception type from base64.b64decode() When the base64.b64decode() function encounters a value with malformed padding it raises a TypeError in python 2.x and a binascii.Error in python 3.x. ====================================================================== ERROR: test_malformed_fingerprint ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/unit/descriptor/router_status_entry.py", line 231, in test_malformed_fingerprint self._expect_invalid_attr(content, "fingerprint") File "/home/atagar/Desktop/stem/test/data/python3/test/unit/descriptor/router_status_entry.py", line 525, in _expect_invalid_attr entry = RouterStatusEntryV3(content, False) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/router_status_entry.py", line 330, in __init__ super(RouterStatusEntryV3, self).__init__(content, validate, document) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/router_status_entry.py", line 150, in __init__ self._parse(entries, validate) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/router_status_entry.py", line 337, in _parse _parse_r_line(self, value, validate, True) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/router_status_entry.py", line 496, in _parse_r_line desc.fingerprint = _decode_fingerprint(r_comp[1], validate) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/router_status_entry.py", line 703, in _decode_fingerprint identity_decoded = base64.b64decode(stem.util.str_tools.to_bytes(identity)) File "/usr/lib/python3.2/base64.py", line 92, in b64decode return binascii.a2b_base64(s) binascii.Error: Incorrect padding commit cc3f1be711665ab203f8a657a8400e2828bb9aef Author: Damian Johnson Date: Sun Jan 27 13:41:01 2013 -0800 Dropping direct __dict__ usage Using an object's __dict__ attribute is generally thought to be a bad idea, and indeed it bites us from time to time. Functions like hasattr(), getattr(), setattr(), and vars() makes direct __dict__ usage unnecessary. This also fixes a python 3 mocking issue... ====================================================================== ERROR: test_examples ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/unit/descriptor/networkstatus/document_v3.py", line 129, in test_examples with support_with(io.StringIO(content)) as consensus_file: File "/home/atagar/Desktop/stem/test/data/python3/test/mocking.py", line 303, in support_with obj.__dict__["__enter__"] = return_value(obj) AttributeError: '_io.StringIO' object has no attribute '__dict__' commit de9ac90b6db59f29a42453b829a66dff7b3b1767 Author: Damian Johnson Date: Sun Jan 27 13:17:48 2013 -0800 NetworkStatus' meets_consensus_method() comparisons didn't account for None Another spot where comparison with None fails with python 3... ====================================================================== ERROR: test_with_directory_authorities ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/unit/descriptor/networkstatus/document_v3.py", line 800, in test_with_directory_authorities document = NetworkStatusDocumentV3(content) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/networkstatus.py", line 509, in __init__ self._footer = _DocumentFooter(document_file, validate, self._header) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/networkstatus.py", line 776, in __init__ if header.meets_consensus_method(9): File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/networkstatus.py", line 581, in meets_consensus_method return bool(self.consensus_method >= method or [x for x in self.consensus_methods if x >= method]) TypeError: unorderable types: NoneType() >= int() commit b96f434623cd829dc8e143e1e685167facad4e22 Author: Damian Johnson Date: Sun Jan 27 12:51:57 2013 -0800 Python 3 can't do comparison with other types Another change in python 2 => 3 is that comparison ('<' and '>') can no longer work with None, ints, and other types... In python 2: >>> 5 > None True In python 3: >>> 5 > None Traceback (most recent call last): File "", line 1, in TypeError: unorderable types: int() > NoneType() I expected the 'unorderable types' error would cause it to use the Version's comparison function but evidently that's not the case. Rather, when using python 3 callers need to check the values' type first (there's nothing we can do to make the comparison work). Personally I think that this is pretty stupid, but that's the workaround done by python devs on... http://bugs.python.org/issue13545 Oh well. Dropping the test that comparison with None and ints work. ====================================================================== ERROR: test_nonversion_comparison ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/unit/version.py", line 125, in test_nonversion_comparison self.assertTrue(test_version > None) TypeError: unorderable types: Version() > NoneType() ---------------------------------------------------------------------- Ran 9 tests in 0.043s commit 4ccbe411ff3320a09e21baa44280c357fd44df6e Author: Damian Johnson Date: Sun Jan 27 12:41:50 2013 -0800 The min/max() functions can't accept None in python 3 In python 2: >>> max(5, None) 5 In python 3: >>> max(5, None) Traceback (most recent call last): File "", line 1, in TypeError: unorderable types: NoneType() > int() ====================================================================== ERROR: test_comparison ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/unit/version.py", line 107, in test_comparison self.assert_version_is_equal("0.1.2", "0.1.2.0") File "/home/atagar/Desktop/stem/test/data/python3/test/unit/version.py", line 252, in assert_version_is_equal self.assertEqual(version1, version2) File "/usr/lib/python3.2/unittest/case.py", line 643, in assertEqual assertion_func(first, second, msg=msg) File "/usr/lib/python3.2/unittest/case.py", line 633, in _baseAssertEqual if not first == second: File "/home/atagar/Desktop/stem/test/data/python3/stem/version.py", line 214, in __eq__ return self._compare(other, lambda s, o: s == o) File "/home/atagar/Desktop/stem/test/data/python3/stem/version.py", line 197, in _compare my_version = max(0, self.__dict__[attr]) TypeError: unorderable types: NoneType() > int() commit e6e5005b8afd9b7e0a1239a92d7c7a3f39e3ef42 Author: Damian Johnson Date: Sun Jan 27 12:28:59 2013 -0800 Using byte string for binascii.a2b_hex in python 3 Like base64.b64decode(), binascii.a2b_hex() expects a byte string. ====================================================================== ERROR: test_valid_response ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/unit/response/authchallenge.py", line 31, in test_valid_response stem.response.convert("AUTHCHALLENGE", control_message) File "/home/atagar/Desktop/stem/test/data/python3/stem/response/__init__.py", line 117, in convert message._parse_message(**kwargs) File "/home/atagar/Desktop/stem/test/data/python3/stem/response/authchallenge.py", line 40, in _parse_message self.server_hash = binascii.a2b_hex(value) TypeError: 'str' does not support the buffer interface ---------------------------------------------------------------------- Ran 2 tests in 0.009s commit 2e7913aa6c59d147ef5ae926d456c54ec459dd52 Author: Damian Johnson Date: Sun Jan 27 12:17:43 2013 -0800 Using byte string for base64.b64decode in python 3 First correction for the ASCII -> Unicode switch over in python 3. ====================================================================== ERROR: test_get_network_status ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/router_status_entry.py", line 702, in _decode_fingerprint identity_decoded = base64.b64decode(identity) File "/usr/lib/python3.2/base64.py", line 83, in b64decode raise TypeError("expected bytes, not %s" % s.__class__.__name__) TypeError: expected bytes, not str During handling of the above exception, another exception occurred: Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/unit/control/controller.py", line 247, in test_get_network_status router = stem.descriptor.router_status_entry.RouterStatusEntryV2(desc) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/router_status_entry.py", line 254, in __init__ super(RouterStatusEntryV2, self).__init__(content, validate, document) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/router_status_entry.py", line 149, in __init__ self._parse(entries, validate) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/router_status_entry.py", line 261, in _parse _parse_r_line(self, value, validate, True) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/router_status_entry.py", line 495, in _parse_r_line desc.fingerprint = _decode_fingerprint(r_comp[1], validate) File "/home/atagar/Desktop/stem/test/data/python3/stem/descriptor/router_status_entry.py", line 707, in _decode_fingerprint raise ValueError("Unable to decode identity string '%s'" % identity) ValueError: Unable to decode identity string '/96bKo4soysolMgKn5Hex2nyFSY=' ---------------------------------------------------------------------- Ran 9 tests in 0.063s commit 4f108d1dffd2e23db8ced6d4cb755a336a0767f3 Author: Damian Johnson Date: Fri Jan 25 09:57:54 2013 -0800 Including signature method in DocumentSignature's comparison The DocumentSignature's method argument is new-ish, and its comparison function wasn't taking it into account. Both fixing this and using the default method (sha1) in the mock instance so our tests still pass. commit 5914ae523eb393383a120cd24c12c508e0ffaeac Author: Damian Johnson Date: Tue Jan 22 09:27:45 2013 -0800 Using rich comparison functions Python 3.x drops support for __cmp__, requiring that users opt for rich comparison methods instead... http://python3porting.com/problems.html#unorderable-types-cmp-and-cmp ====================================================================== ERROR: test_event_listening ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/unit/control/controller.py", line 285, in test_event_listening self.assertRaises(InvalidRequest, self.controller.add_event_listener, mocking.no_op(), EventType.BW) File "/usr/lib/python3.2/unittest/case.py", line 574, in assertRaises callableObj(*args, **kwargs) File "/home/atagar/Desktop/stem/test/data/python3/stem/control.py", line 1438, in add_event_listener if not self.get_version().meets_requirements(event_version): File "/home/atagar/Desktop/stem/test/data/python3/stem/version.py", line 175, in meets_requirements return self >= requirements TypeError: unorderable types: Version() >= Version() This change uncovered a couple bugs where events failed to strip off the trailing 'OK'. I'm not entirely sure why the previous comparisons didn't catch this, but oh well - fixed now. commit 137aed3eadb184fd3e96828cd207220d3e09a043 Author: Damian Johnson Date: Mon Jan 21 09:02:18 2013 -0800 Disabling static checks when using python 3.x Not only are our static checks useless with the python 3.x codebase (pyflakes probably ins't python 3.x compatable and who cares about PEP8 issues with the output?), the pyflake check errors... Traceback (most recent call last): File "./test/data/python3/run_tests.py", line 554, in style_issues.update(test.check_whitespace.pyflakes_issues(os.path.join(base_path, "stem"))) File "/home/atagar/Desktop/stem/test/data/python3/test/check_whitespace.py", line 132, in pyflakes_issues line_match = re.match("^(.*):(\d+): (.*)$", line) File "/usr/lib/python3.2/re.py", line 153, in match return _compile(pattern, flags).match(string) TypeError: can't use a string pattern on a bytes-like object commit 62086346044c9eab1ce68d830669559773ef22b9 Author: Damian Johnson Date: Mon Jan 21 08:57:02 2013 -0800 Accounting for __builtin__ renaming Python 3.x renames __builtin__ to builtins... ====================================================================== ERROR: test_parse_circ_path ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/data/python3/test/unit/control/controller.py", line 24, in setUp mocking.revert_mocking() File "/home/atagar/Desktop/stem/test/data/python3/test/mocking.py", line 404, in revert_mocking if module == __builtin__: NameError: global name '__builtin__' is not defined commit 84df4318abce565fd2f682ee6e12ea2fe9acf072 Author: Damian Johnson Date: Mon Jan 21 08:56:35 2013 -0800 Adding is_python_3() to prereq module Something tells me that we'll need this... commit 54eac73e62739cb40bced307ff6967548eb3af42 Author: Damian Johnson Date: Mon Jan 21 00:42:46 2013 -0800 Dropping hardcoded python 2.x check Our prereq module asserted that stem isn't compatable beyond the 2.x series. Clearly I'm trying to 'fix' that with this branch. Ideally we'd keep this check for our python 2.x codebase, and strip it when we do a 2to3 conversion. However, that's more error prone than I'd like so simply dropping the check entirely. commit 77b6fa86df045298564b1c0b9f66bd3d71d66d2e Author: Damian Johnson Date: Mon Jan 21 00:36:24 2013 -0800 Making the Enum's keys() method provide a list The 2to3 converter is evidently confused because we have a keys() method that provides a tuple rather than a list. Dictionaries provide lists via their keys() method so there's probably not a terribly compelling reason for us to do differently. commit f9a2f147b6fe6187e3ba8d1b5bffa5efce14fac1 Author: Damian Johnson Date: Mon Jan 21 00:18:20 2013 -0800 Making static checks skip the data directory Our static checkers (most noticeably pep8 and pyflakes) run over the test data directory. This was fine until I started using it for the python3 export. Now that it has python code these checkers have a lot to say about it, none of it helpful. Simply ignoring the output if it concerns the data directory. This is a sucky solution since pyflakes/pep8 are still running over an extra copy of our codebase, taking a noticeable bit of extra time. However, pyflakes at least lacks any options for ignoring directories. I'll probably simply move our python3 export to another location later if this annoys me. commit 919502c53baff9f823ed537dab86ec930cf2eaf7 Author: Damian Johnson Date: Sun Jan 20 23:42:29 2013 -0800 Python 3 testing support Adding python 3 support to the tests. This introduces two new testing arguments... --python3 Does a 2to3 export of our codebase and tests against that. If a python3 export already exists then reuses that (... it takes an annoyingly long time to run). --clean Wipes our prior python3 export. Next to start sorting out the 2to3 conversion issues... commit 7921a46ba8655baa7c4b81d900a9854444675564 Author: Damian Johnson Date: Thu Jan 24 09:21:35 2013 -0800 Populating BridgeNetworkStatusDocuments with v2 entries Karsten reports that bridge network status documents have v2 router status entries, not v3. https://trac.torproject.org/8036 commit 0b1d47d92d447614095cf43fdb35214650db8d82 Author: Damian Johnson Date: Thu Jan 24 09:20:47 2013 -0800 NetworkStatusDocumentV3's params misdocumented as being a list The params is a dict, not a list. Caught by amj703 on... https://trac.torproject.org/8036 commit ce8fcf13521567c439696d7e87894eb09e238347 Author: Damian Johnson Date: Wed Jan 23 09:25:03 2013 -0800 Broken link for bad relays wiki Wrong format for external links. /home/atagar/Desktop/stem/stem/__init__.py: docstring of stem:56: ERROR: Unknown target name: "https://trac.torproject.org/projects/tor/wiki/doc/badrelays". commit ab992cd1716b4bc9f61660b746281c61dd878ab5 Author: Damian Johnson Date: Wed Jan 23 09:21:53 2013 -0800 Fixing ExitPolicyRule's get_address_type() pydocs We had an ending colon rather than a backtick for the AddressType references. /home/atagar/Desktop/stem/stem/exit_policy.py: docstring of stem.exit_policy.ExitPolicyRule.get_address_type:1: WARNING: Inline interpreted text or phrase reference start-string without end-string. /home/atagar/Desktop/stem/stem/exit_policy.py: docstring of stem.exit_policy.ExitPolicyRule.get_address_type:3: WARNING: Inline interpreted text or phrase reference start-string without end-string. commit 2fe1e2a9ee709f745fb65d2716f6b42c7ba24b28 Author: Damian Johnson Date: Wed Jan 23 09:16:24 2013 -0800 Fixing extend_circuit exception pydocs Correcting a couple mistakes in extend_circuit's pydocs... /home/atagar/Desktop/stem/stem/control.py: docstring of stem.control.Controller.extend_circuit:30: ERROR: Unexpected indentation. /home/atagar/Desktop/stem/stem/control.py: docstring of stem.control.Controller.extend_circuit:31: WARNING: Block quote ends without a blank line; unexpected unindent. commit 5bcb53923214330d3d6b62eea48c930dde65adf8 Author: Damian Johnson Date: Wed Jan 23 09:09:03 2013 -0800 Linking the RouterStatusEntry's document attr Fixing the pydocs for the RouterStatusEntry's document attribute. Interlinking requires a fully qualified class. commit ea4964267e519c33b039f88d9d857a8f6452b3a3 Author: Damian Johnson Date: Sun Jan 20 23:55:05 2013 -0800 Empty bridge-ip-versions value broke parser A 'bridge-ip-versions' line without a value broke our parser... File "/home/karsten/tasks/task-7828/stem/stem/descriptor/extrainfo_descriptor.py", line 858, in __init__ super(BridgeExtraInfoDescriptor, self).__init__(raw_contents, validate) File "/home/karsten/tasks/task-7828/stem/stem/descriptor/extrainfo_descriptor.py", line 413, in __init__ self._parse(entries, validate) File "/home/karsten/tasks/task-7828/stem/stem/descriptor/extrainfo_descriptor.py", line 882, in _parse raise stem.ProtocolError("The bridge-ip-versions should be a comma separated listing of '=' mappings: '%s' (entry = '%s')" % (line, entry, )) ProtocolError: The bridge-ip-versions should be a comma separated listing of '=' mappings: 'bridge-ip-versions ' (entry = '') Reported by Karsten on... https://trac.torproject.org/7967 commit d9553f57fedb4692b44c497942ad1a952fb38a02 Author: Damian Johnson Date: Sun Jan 20 16:37:32 2013 -0800 Adding validate argument to the DescriptorReader Request by Aaron for the validate attribute to be available on the reader. commit 998abb262eb4ed6a3d88a7be865412685ab26d3a Author: Damian Johnson Date: Sun Jan 20 13:59:05 2013 -0800 Exposing the validate arg through parse_file() Users may want to opt out of validation, either for performance or to accept quasy-malformed content. Giving them that option. commit 9183f64f2085d0f22d66b6cc381b6e66eef227f7 Author: Damian Johnson Date: Sun Jan 20 13:44:28 2013 -0800 Parsing multiple descriptors from non-cached files We were parsing single descriptors from metrics archives because... well, that's what they contained. This is unintuitive to users making then parsing their own descriptors, so always using the _parse_file() functions when able. An extra advantage is that all server descriptors parsed through parse_file() will support server annotations (so things like "@downloaded-at 2012-09-17 17:44:50" will no longer trip it up). https://trac.torproject.org/8005 commit 7bd5b49ea27158b44f97e3f5f400aa7df84c2e4b Author: Damian Johnson Date: Sat Jan 19 21:24:54 2013 -0800 Adding Arch Linux to the download page Sjon has made a stem package so adding it to our download page. commit 80f11181966c5390a0aabbb531c20bbcd06a3860 Author: Damian Johnson Date: Sat Jan 19 20:51:12 2013 -0800 Py_GetArgcArgv unavailable with pypy Peer reports that stem fails with an AttributeError on pypy. Unsurprisingly this is in the process renaming voodoo - adding a try/catch to make this more relient. Traceback (most recent call last): File "app_main.py", line 51, in run_toplevel File "first_pass.py", line 62, in for router in parse_file(consensus_file): File "stem/descriptor/__init__.py", line 94, in parse_file import stem.descriptor.server_descriptor File "stem/descriptor/server_descriptor.py", line 40, in import stem.version File "stem/version.py", line 65, in import stem.util.system File "stem/util/system.py", line 74, in Py_GetArgcArgv = ctypes.pythonapi.Py_GetArgcArgv File "/usr/lib/pypy/lib-python/2.7/ctypes/__init__.py", line 371, in __getattr__ func = self.__getitem__(name) File "/usr/lib/pypy/lib-python/2.7/ctypes/__init__.py", line 376, in __getitem__ func = self._FuncPtr((name_or_ordinal, self)) File "/usr/lib/pypy/lib_pypy/_ctypes/function.py", line 250, in __init__ ptr = self._getfuncptr([], ctypes.c_int) File "/usr/lib/pypy/lib_pypy/_ctypes/function.py", line 421, in _getfuncptr self._ptr = cdll.getfunc(self.name, ffi_argtypes, ffi_restype) AttributeError: No symbol Py_GetArgcArgv found in library commit 3d3e69417975499fbe5d2a3eb591d808c6800874 Author: Damian Johnson Date: Sat Jan 19 15:53:06 2013 -0800 Always processing @type headers The parse_file() function only consumed the @type annotation when we didn't have a descriptor_type nor recognized filename. Changing it so we always consume the @type header. I'm also changing the priority order of parse_file() so if the file could be both a metrics archive or cached descriptor (for instance a file named 'cached-consensus' that starts with a @type annotation) then we process it as a metrics archive. commit ca0375e0c47b8162c94745ec4ae6a648c9c0021c Author: Damian Johnson Date: Fri Jan 18 22:41:09 2013 -0800 Clearing request cache on sighup Sighups can render our cache as being stale. Clearing it. commit fa6ef6bca6e703e9b69a140ec5abe4c972122872 Author: Damian Johnson Date: Fri Jan 18 09:24:11 2013 -0800 Adding support for '@type dir-key-certificate-3 1.0' We already parsed key certificates for network status documents. I didn't know though that they had their own @type annotation. Test data is from... https://metrics.torproject.org/data/certs.tar.bz2 https://trac.torproject.org/7987 commit 42ddd84ce3106d350890db064c631eaafe9b8d17 Author: Damian Johnson Date: Fri Jan 18 08:48:57 2013 -0800 Updating test for the network status example I change one of the pydoc examples in the networkstatus module to use stem.descriptor.parse_file() rather than the module's (now private) parse_file method. Updating the unit test for this example. Updating this test has the side benefit that it now gives us test coverage of the descriptor_type argument (I had missed that previously). commit 8b3e9ed828070b2da9010220e989017668cb994d Author: Damian Johnson Date: Fri Jan 18 00:04:30 2013 -0800 Table providing desctriptor_type to class mappings commit d34b30903a071cc0fd8601c9af8a35da3f57f3d2 Author: Damian Johnson Date: Thu Jan 17 23:28:53 2013 -0800 Replacing our tuple descriptor_type arg with a str Changing the type of the descriptor_type arg I added earlier today. A string would be more intuitive to use than a tuple. commit b060be224ef31342f813b8224fcac7f17fdcb3a4 Author: Damian Johnson Date: Thu Jan 17 09:37:37 2013 -0800 Malformed url in docs Missing the proper reStructuredText formatting for a url. commit 5818640e54542ff079b0a9328f82bb0c5897bc9a Author: Damian Johnson Date: Thu Jan 17 09:28:56 2013 -0800 Only providing a single parse_file() function Ok, our interface for parsing descriptors was damn confusing. Users had three options... * stem.descriptor.reader This was the method I had intended for users to always read descriptor files with, but folks keep finding parse_file() first... * stem.descriptor.parse_file() I didn't intend for this to be used by our users (I made it to support the DescriptorReader). However, it's pretty convenient for reading individual descriptor files. * stem.descriptor.*.parse_file() The parse_file() in individual descriptor classes definitely weren't intended for external users. They were made to support stem.descriptor.parse_file(), but turned out to be used most of all... I'm making the last group private to discourage their usage and changing stem.descriptor.parse_file() to be both more capable and user friendly... * users can now explicitly give it a descriptor type, as per the metrics types (https://metrics.torproject.org/formats.html#descriptortypes) * the path is now optional (it was only required because I expected this to be used by the DescriptorReader) * ... and better pydocs This breaks backward compatibility, but should greatly help our usability going forward. Thanks to Aaron for pointing out the pain points. commit a85aa8cfcdabfaf2176497abeaf0c90cb80a9624 Author: Damian Johnson Date: Thu Jan 17 08:43:17 2013 -0800 Moving Flag enum to stem's base module I'm making an effort to put most general purpose enums in the base module. Oddly I never referenced the Flag enum in the descriptor pydocs... commit 7d013d66db31d9bef398b00e12e2fb78c7222a32 Author: Damian Johnson Date: Thu Jan 17 08:37:58 2013 -0800 Adding documentation for Flag enums We were missing docs for the Flag enums along with a couple values (BADDIRECTORY and V3DIR). commit 1fb3a96c75f5eb6f1706ab7d34d582df3ee2cce7 Author: Damian Johnson Date: Wed Jan 16 09:27:27 2013 -0800 Notifying status listeners of SIGHUPs Our status listeners should be notified when three things happen... * we newly connect to a controller * we disconnect from a controller * tor's state is reset by a SIGHUP signal We had implemented the first two, but not the third. Correcting this oversight. commit 5d9c25300e6783b092a64fc66877f39566fc4a4f Author: Damian Johnson Date: Wed Jan 16 09:21:18 2013 -0800 Letting add_event_listener() work when unauthenticated If we called add_event_listener() prior to authentication then we'd throw an error when calling get_version() to figure out if we meet the requirements. Changing the behavior so that we check requirements and issue SETEVENTS if able, but if not then simply enquing the listener. It'll then be attached during our post-authentication hook. The change that I'm more interested in this though is that our post-authentication hook is less likely to go belly up. If we, say, added a listener then attached the Controller to an older tor instance then our SETEVENTS call could fail. This was an all-or-nothing call. :( Changed it so that the hook will re-attach the events that it can, then warn about the rest. commit a0e13fbcc8fd8e0678b7906329d11e7f567839d1 Author: Damian Johnson Date: Wed Jan 16 09:09:06 2013 -0800 test_get_exit_policy failed if we get a public address I forgot to account for the exit policy including a 'reject [public_addr]:*' entry. This is a bit of a pain both because it's dynamic, and because it may or may not be included at all. Accounting for this by just checking the policy prefix and suffix instead. commit 32092205f1be4ccc01ac2cd4bd78a63fd732ae45 Author: Damian Johnson Date: Wed Jan 16 07:38:29 2013 -0800 Adding a BaseController is_authenticated() method I've been running into some situations where I need to know if the controller is not only connected, but authenticated (since that determines if it's... well, usable). A use case where this could get confused is... * User gets a ControlSocket instance. * Manually authenticates via the send() and recv() methods. * Makes a new Controller object from this control socket. Unfortunately this is *exactly* what our test runner did, so tweaking it so that it hits our post_authentication method (I'm surprised that it hadn't bitten us before). commit 1f498f1ce24e149e166ce280872283a61fa76ef4 Author: Damian Johnson Date: Wed Jan 16 07:33:22 2013 -0800 Moving event version requrements to the version module Moving the location to the Requirement enum so users can more easily check if an event is available or not. commit f09e5efc22264c8c83bd39b9ae679f3f6beeb752 Author: Damian Johnson Date: Mon Jan 14 08:34:37 2013 -0800 Dropping expect_alive arg from _notify_status_listeners On reflection the expect_alive attribute can be implied by the state. Simplifying things. commit 32b9de7c8e2690d4642ac72019c7ed684e3b3576 Author: Damian Johnson Date: Sun Jan 13 21:12:08 2013 -0800 Caching ExitPolicy's can_exit_to() results That's odd. I wonder why I didn't add a cache here - can_exit_to() is a method that's highly likely to be called repeatedly so if we can get contant time lookups then great. commit b03613cc62d7ae19faa1bbf63e95cffb3547eeae Merge: 427fce8 716f8a6 Author: Damian Johnson Date: Sun Jan 13 17:38:12 2013 -0800 Controller get_exit_policy() method commit 716f8a693e9b814da5e2c9df551dbe6768f4f324 Author: Damian Johnson Date: Sun Jan 13 17:19:33 2013 -0800 Controller method to get the exit policy What started out with 'we should have a method to get our exit policy' turned out to not be as simple as I thought. A relay's exit policy is really the combination of three things: * 'reject private:*, reject [public address]:*' if ExitPolicyRejectPrivate is set. * Our ExitPolicy. Tor provides the torrc value rather than a proper policy spec, so I added a get_config_policy() function to account for the differences. * The default exit policy (as per 'GETINFO exit-policy/default'). commit f0ae1eaeec229e51ce8ce223dd1b862d769c1a0c Author: Damian Johnson Date: Sun Jan 13 17:14:55 2013 -0800 Dropping redundant policy rules While working with 'reject private:*' entries I realized that ExitPolicyRejectPrivate makes for really long, ugly policies. In general this is just life - exit policies are more complicated than just the ExitPolicy torrc option. Hoever, in the case of 'reject *:*' we can safely boil things down. commit f264a626d682f6b8a4cdea8b58a260b1c1069524 Author: Damian Johnson Date: Sun Jan 13 16:33:03 2013 -0800 Providing ExitPolicy via get_config_policy() On reflection it desn't make sense to provide a list of ExitPolicyRules. Usually we want a policy itself, and if we do want a rule list then that's easily done via 'list(get_config_policy())'. commit 67c00e3c89b0cebd00ce5f6f19ce194b200eb8f4 Author: Damian Johnson Date: Sun Jan 13 16:19:17 2013 -0800 Truncating exit policies after a catch-all It's pointless for an exit policy to contain rules after an 'accept *:*' or 'reject *:*'. Cropping policies if they go on after that. commit 54d2c103940b31195b3c91ac440d7f07cb6b9489 Author: Damian Johnson Date: Sun Jan 13 12:57:02 2013 -0800 Function to parse configuration exit policies Tor exit policies found in the torrc (and 'GETCONF ExitPolicy') differ slightly from the exitpattern definition found in the spec. Adding a get_config_policy() function that converts these into proper exit policies. commit 427fce8bd6777542f234e1738b35ee553853a366 Author: Damian Johnson Date: Sat Jan 12 23:42:11 2013 -0800 Support process renaming Renaming our python process from "python our_script.py" to something more sensible is a very handy capability to have. Thanks to Jake arm has had this for years, which makes it easier to identify in top and ps listings. We'll want this capability for other controllers so snagging arm's utility and cleaning it up. commit 84664f1d6f388c93ff686dd049dfe48804ead41f Author: Damian Johnson Date: Sat Jan 12 21:07:47 2013 -0800 Missing files for prior commit Oops. Did a 'git commit -a' while forgetting that we were adding a couple new files. commit b236ac4e0ba830352c447537be6cf59d85650ae0 Author: Damian Johnson Date: Sat Jan 12 21:04:01 2013 -0800 Support for bridge network status documents Tor metrics has network status documents for bridges. These are not part of the dir-spec, and presently not even in the metrics spec. However, they're trivial to parse, consisting of just a 'published' line followed by v3 router status entries. This resolves... https://trac.torproject.org/7938 commit 98a8842c9ab800f9891a01caf35d02f2dfc60809 Author: Damian Johnson Date: Sat Jan 12 10:46:39 2013 -0800 Supporting footers in pre-version 9 method documents All version 3 network status documents can have a footer. The caveat is that prior to consensus method 9 they just contained signatures, and after that they were a properly marked 'directory-footer' section. Reproed the issue that Karsten discovered in 'https://trac.torproject.org/7932' via our unit tests then fixed. commit 04391b319fd63fb10c6bf925b86796d2d26ec526 Author: Damian Johnson Date: Fri Jan 11 09:07:35 2013 -0800 String explanation for FileSkipped exceptions For #7828 I wrote a little consenses parser script. That script included a very simple skip listener... lambda path, exc: LOGGER.warning(" skipped due to %s" % exc) "How could it fail?" I asked myself. Yet when I just ran it the listener gave me the very helpful warning of "skipped due to ". It turns out that we weren't providing our exceptions with a string explanation - go me. :P commit 153ada322653f95eb21c360d3cac4fe6ad436ed7 Author: Damian Johnson Date: Thu Jan 10 08:37:17 2013 -0800 Supporting '-legacy' authority entries Directory authorities have an alternative format if their nickname has a '-legacy' suffix. Caught by Karsten on... https://trac.torproject.org/7866 commit 29af06cd5854ad70290b8066bdb1f3219831c458 Author: Damian Johnson Date: Wed Jan 9 21:27:41 2013 -0800 Using https link for cloning from git commit 4267b6c42a0de9546f664b7498bb3f2ca9b065b2 Author: Damian Johnson Date: Wed Jan 9 20:29:19 2013 -0800 Fixing issues caught by pyflakes Mostly extra imports, though in one case it caught an error (default arguments for get_server_descriptors() and get_network_statuses() would cause a NameError). commit c88e3da8c481dfc4e726a5e3d69bc19ffffddb57 Author: Damian Johnson Date: Wed Jan 9 20:20:28 2013 -0800 Adding a '--all' argument to the tests It'll be a whole lot easier to ask patch submitters to run 'run_tests.py --all' rather than 'run_tests.py --unit --integ --style'. :) commit 73e69ac2a612d3eef7b4a6f8efd99063ecb7babb Author: Damian Johnson Date: Wed Jan 9 20:15:38 2013 -0800 Integrating pyflakes with our tests Inspired by txtorcon, using pyflakes for static error checking of our codebase. Unlike pylint it has precious few false positives for our codebase. For the false positives we do have I've added a 'pylint.ignore' config mapping so they can be whitelisted. commit d29ec4d12567a6af6f483400658d2d2d17769a5a Merge: d580192 6cd814d Author: Damian Johnson Date: Mon Jan 7 00:57:28 2013 -0800 Changing our style to mostly match PEP8 Stem now conforms with PEP8 with the following exceptions... * four space indentations * strict 79 character line widths * no spaces around '=' in keyword args If you're curious about why then see 'test/check_whitespace.py' for a short explanation, but I'm not interested in discussing this bike shed's color further. If both Sean and Ravi both want one of these to change then I'll accept a patch. This also adds the pep8 style checker to our tests... http://pypi.python.org/pypi/pep8 You can run it via the '--style' argument. commit 6cd814dcb3c24a301f5e4c6396388a424629fd3f Author: Damian Johnson Date: Mon Jan 7 00:56:47 2013 -0800 Adding '--style' to run_tests.py's help output commit 44c018886e0ff43e7eb42f7ec34402a301d40cbf Author: Damian Johnson Date: Mon Jan 7 00:48:21 2013 -0800 Skipping E111 and E121 (four space indentations) I strongly prefer two space indentations. It reads perfectly well and leads to shorter lines. I've never heard a coherent argument why four spaces, eight spaces, tabs, or any of the other measurement is actually superior. When it comes to this bike shed I'm keeping the color as it is for now. commit 48776aee58767f0b8fef3b49d6e77ec65c09681f Author: Damian Johnson Date: Mon Jan 7 00:42:05 2013 -0800 Conforming to W293 (blank line contains whitespace) Finally we get to the main thing that started this whole odyssey. Sed was my friend with this one... find . -name "*.py" | xargs sed -i 's/[ \t]*$//' commit 26b27b6eff21214b17b4018a16ac97f3bd674734 Author: Damian Johnson Date: Mon Jan 7 00:31:22 2013 -0800 Conforming to E302 (two blank lines above methods and functions) Another long one. I hope this is almost done... commit 1eed64be89d0cf0382234786c020ab4760a4e8e4 Author: Damian Johnson Date: Mon Jan 7 00:00:20 2013 -0800 Conforming to E701 (multiple statements on one line) Well, that took a while. commit ed070ee78b2a3d0baed01158b669611646957d97 Author: Damian Johnson Date: Sun Jan 6 22:48:59 2013 -0800 Skipping E251 (no spaces around keyword / parameter equals) I got a little ways through switching us over to this one then stopped. It looks aweful and makes the code less readable. If Sean and Ravi both decide that they want this then fine - one of them can be the person to swap it over. commit 23b06d380c4e2b3e0bfb0348ff03c5685e59754b Author: Damian Johnson Date: Sun Jan 6 22:37:33 2013 -0800 Conforming to E261 (at least two spaces before inline comment) commit 82a2994d21120b7505c7ebc9f78fbf6774843f35 Author: Damian Johnson Date: Sun Jan 6 22:15:58 2013 -0800 Conforming to W391 (blank line at end of file) I've been including an empty line at the end of files since it makes it a bit easier for me to work with via vim. I don't care overly much though so appeasing pip8. commit ff92770fba42f0daa40204795b9568fb6389d7cc Author: Damian Johnson Date: Sun Jan 6 21:57:07 2013 -0800 Skipping E127 (continuation line over-indented for visual indent) Bug with pep8 where detection only works for this if we have four space indents (which we aren't swapping to at present). commit 1af5594eda89aa0008f20d3c63f6447b0a4e44bf Author: Damian Johnson Date: Sun Jan 6 21:50:32 2013 -0800 Conforming to E241 (multiple spaces after ':') commit 9e45887a491b6b1938303bc3077d00673fca119a Author: Damian Johnson Date: Sun Jan 6 20:40:51 2013 -0800 Conforming to E128 (continuation line under-indented for visual indent) Meh. commit b8b32e0aa751d8f2c2fbe43502b1f400871f0919 Author: Damian Johnson Date: Sun Jan 6 20:29:16 2013 -0800 Conforming to E226 (missing optional whitespace around operator) Another nice one. commit b2a087ad2c84eaaa89d147fa92a5ef3c5f2f1b9a Author: Damian Johnson Date: Sun Jan 6 20:26:17 2013 -0800 Conforming to E231 (missing whitespace after ':') Another nice catch. commit f3fada29235b9d504c4daa34f451d31249058fa9 Author: Damian Johnson Date: Sun Jan 6 20:23:09 2013 -0800 Conforming to E201 (whitespace after '{') Agreed on this one. commit 6a32f1a257cc1f7bf7d8499dea7d2a5df5e9d9c0 Author: Damian Johnson Date: Sun Jan 6 20:21:25 2013 -0800 Conforming to E203 (whitespace before ':') Meh. I'm having trouble thinking of this as either any better or worse. commit 708bcad9746e4b757fd14555ed8bf13e50990475 Author: Damian Johnson Date: Sun Jan 6 20:18:55 2013 -0800 Conforming to E124 (closing bracket does not match visual indentation) This one seems to be another false alarm, but can't expect it to always be right. commit e08c0f6604a80509c001899e39553b17a21ebc0f Author: Damian Johnson Date: Sun Jan 6 20:16:12 2013 -0800 Conforming to E211 (whitespace before '[') Another nice catch by pep8. commit 64113e5a505d8c7b57933ab2d1e3c0a4d6feac6d Author: Damian Johnson Date: Sun Jan 6 20:14:40 2013 -0800 Conforming to E222 (multiple spaces after operator) Extra space around an assignment. Oops. commit 28a39f1127c7b455076e86364eb4880ae19e9c18 Author: Damian Johnson Date: Sun Jan 6 20:12:50 2013 -0800 Conforming to E225 (missing whitespace around operator) Ok, PEP8 is trying to make me like it again. These I definitely agree with. Shame on me for letting 'em slip in. commit f735fbc11973fd03435df0bc306d800433bbbeae Author: Damian Johnson Date: Sun Jan 6 20:06:55 2013 -0800 Conforming to E221 (multiple spaces before operator) I'd forgotten that PEP8 forbade aligned assignments. This is another spot that I think PEP8 is in the wrong in terms of readability but I don't care overly much. commit 23b6ffd899370f3076a845c7de4bc92220a49eaa Author: Damian Johnson Date: Sun Jan 6 20:01:28 2013 -0800 Conforming to E126 (continuation line over-indented for hanging indent) Ok, I haven't a clue what the hell this one was about. In one spot pep8 complained about the hanging indent. However, it looks right to me. For some odd reason it wants the hanging indent to match the 'stem.descriptor.networkstatus' import above - maybe this is a bug with pep8. Oh well. Changing our code to shut it up. commit 366f77882261031f86ac3d9a6217ae0379dc48cd Author: Damian Johnson Date: Sun Jan 6 19:52:58 2013 -0800 Conforming to E262 (inline comment should have a space after the '#') Another nice catch. commit 6bc6c231e16122e66205e8e04b849f316fa6ada7 Author: Damian Johnson Date: Sun Jan 6 19:51:13 2013 -0800 Conforming to E271 (multiple spaces after keyword) Caught an extra space in the ordereddict we brought in. Nice catch. commit b25d5332a8b10c70211c83df9f48ea7b97710fde Author: Damian Johnson Date: Sun Jan 6 19:47:55 2013 -0800 Conforming to E502 (backslash to split lines) Ok, this one I think is stupid. "Thou shalt not have long lines" says PEP8, "... and thou shalt not spit up thy long lines." In two of the instances I agree, it wasn't buying us anything. But in the third it was actually making things more readable. Oh well, I'm not gonna put up a fuss over just one line of code. commit 026dbce5bf191c38a48605f041b10f10629d22b2 Author: Damian Johnson Date: Sun Jan 6 19:44:13 2013 -0800 Conforming to E303 (too many blank lines) Had an extra newline at one point. I'm starting to like this style checker thing... commit cca1b3c9854fc04ad3c4ff13a22d1ff9c632b5fe Author: Damian Johnson Date: Sun Jan 6 19:40:43 2013 -0800 Conforming to E711 (None checks by identity) E711 is actually something that we *were* trying to comply with (as are several other PEP8 warnings we're getting, actually). Fixing the one occurance. commit 4c7af528d2ab80737ff6ec09d88d194ba12cce2f Author: Damian Johnson Date: Sun Jan 6 19:38:42 2013 -0800 Still run tests if style checking is the only thing Revising the check where we abort if there's no tests to run. commit 3ef0964914f670c05252ad139f88611b636aab37 Author: Damian Johnson Date: Sun Jan 6 19:34:58 2013 -0800 Prompting testers to install pep8 if unavailable Style checks will require pep8 so prompting the user to install it if it's unavailable when they run with '--style'. commit 662a1e00746ecbfdece8136b92f9da9494007ad5 Author: Damian Johnson Date: Sun Jan 6 19:26:00 2013 -0800 Running PEP8 checks under a '--style' arg Well... damn. I had hoped that we would be able to wrap the PEP8 style checks into our normal testing runs (the reason check_whitespace.py works so well is that it's trivial to do with every test run). However, pep8 takes on the order of ten seconds to run over our codebase. This is twice as long as all of our unit tests and hence too long to include in tests without opting in. I'm keeping my fingers crossed that they're doing something stupid with ignored issues so we'll get better performance when we've made it happier. We'll see. On the up side PEP8 is wonderfully configurable with respect to what it accepts and ignores so we can start using it right away. Also, the output is well formed so it was trivial to wrap it into our present style checker (which means consolidated and colorized output - yay!). commit d580192a95d0b323165cf22cfb8bbaca11af9600 Merge: 04546f8 37fd201 Author: Damian Johnson Date: Sun Jan 6 15:07:44 2013 -0800 Expansion of Controller unit tests Several nice additions from Sean... https://trac.torproject.org/7874 commit 37fd201683180377ae7dfe0e908c199433166b12 Author: Sean Robinson Date: Sat Jan 5 14:23:45 2013 -0700 Add unit tests for Controller.get_network_status() Signed-off-by: Sean Robinson commit 043fe6aede76ece7c1a2d44f2e738f471892f3ff Author: Sean Robinson Date: Sat Jan 5 14:22:52 2013 -0700 Add unit tests for Controller.get_protocolinfo() Signed-off-by: Sean Robinson commit e70af604c1716d6baff3e0e908ecc94e861b2fd3 Author: Sean Robinson Date: Sat Jan 5 14:21:36 2013 -0700 Move test_get_socks_listeners_*() methods up in module Controller.get_socks_listeners() is next after get_version(). Signed-off-by: Sean Robinson commit 87b56850d18a39dcb020d3c3e3d45dce2a374d0b Author: Sean Robinson Date: Sat Jan 5 14:16:04 2013 -0700 Add unit tests for Controller.get_version() Signed-off-by: Sean Robinson commit e69530c8af668f7b5f20c350c460e21c0b0cf081 Author: Sean Robinson Date: Sat Jan 5 14:15:21 2013 -0700 Explicitly enable Controller value caching I'll be playing with turning caching on and off later, so I'll set a known state. Signed-off-by: Sean Robinson commit 6005da73c3e667352b7bcf78b2f0b41da5ce70ae Author: Sean Robinson Date: Sat Jan 5 14:14:40 2013 -0700 Move _parse_circ_path() test to bottom of module Moves the test method in its module to an analogous position as the tested function in its module. Signed-off-by: Sean Robinson commit 04546f858b1c7c7fe737acd0360d3a4c25e83d06 Author: Sean Robinson Date: Fri Jan 4 15:41:16 2013 -0700 Use tuple-of-types with isinstance Move to more Pythonic idiom for multiple type check. This form of isinstance was added in Python 2.2, so it's safe for us to use. Signed-off-by: Sean Robinson commit 1bb38dd2460c0674d97d88504763b2c63ba6e72b Author: Sean Robinson Date: Fri Jan 4 15:32:34 2013 -0700 Use isinstance for type checks Change the four uses of type() == x to the more Pythonic isinstance idiom. Signed-off-by: Sean Robinson commit c482e68ebff140bdde174e15566b6be05f5b5dbd Author: Damian Johnson Date: Sun Jan 6 14:48:31 2013 -0800 Fixing test_reattaching_listeners to work with RUN_ALL Checking the runner's torrc to determine what kind of authentication it's using. This way we can skip this part of the test (rather than fail) for the RUN_COOKIE and RUN_SCOOKIE targets. commit 01cfab7d61cb167a786c509afb505ac9546e728b Author: Sean Robinson Date: Sun Jan 6 08:48:51 2013 -0700 Test different authentication styles with passwords This restores testing various authentication paths and keeps password use, if available. RUN_COOKIE and RUN_SCOOKIE targets still do not work. Signed-off-by: Sean Robinson commit 19d9a1ccde704ea05bd8e4b4231820a0b28df974 Author: Damian Johnson Date: Sat Jan 5 20:35:31 2013 -0800 Fixing test_reattaching_listeners when there's authentication The test_reattaching_listeners wasn't making any effort to authenticate connections, causing the test to break for every integ target except RUN_NONE and RUN_OPEN (the later being the default which is why this went undiscovered so long). Oops. commit 69a2d6d8aba1e538bdaf7f0ca8d2c66c57bb99de Author: Damian Johnson Date: Sat Jan 5 19:48:07 2013 -0800 Missing a requirement from test_get_latest_heartbeat All controller tests (including test_get_latest_heartbeat) require an open control socket. Missing this requirement broke the RUN_NONE target. commit c78d9cda3ab5fc307264bce774187b6f4188f96d Author: Damian Johnson Date: Sat Jan 5 18:01:49 2013 -0800 Parsing server descriptor's ntor-onion-key line Parsing for the newly introduced 'ntor-onion-key' line. I'm not entirely clear of the format that our users will want so checking with Nick and just parsing the raw string. commit ec306804fc325b8670758df36ec76a1e10fe3604 Author: Damian Johnson Date: Sat Jan 5 16:14:12 2013 -0800 Renaming MicrodescriptorExitPolicy to MicroExitPolicy Suggestion by Sean on "Stem code review 2013-01-03". The new name both fits better (since the rule class is called MicroExitPolicyRule) and a friendlier length to type. commit 9cd9b4004995015c0a0a04a32bb9c68a1addf728 Author: Damian Johnson Date: Sat Jan 5 16:04:50 2013 -0800 Replacing 'assert' keyword in extend_circuit() Issue spotted by Sean in "Stem code review 2013-01-03". The check itself is fine but I'm trying to avoid using the 'assert' keyword. commit 33cd87351b3bfa945a5f741078d6795c18dd3972 Author: Sean Robinson Date: Sat Dec 22 03:27:01 2012 -0700 Corrects pydocs for mocking.get_object Signed-off-by: Sean Robinson commit 9dbb7fc438192af25c93f08d43686563df18e8ac Author: Sean Robinson Date: Sun Dec 30 15:08:51 2012 -0700 Fix variable name is close_circuit() test This fixes a problem I created in commit 9119f9f0d0091a03. Signed-off-by: Sean Robinson commit 80f03bdd427813eedf88d847e9229415ca42541f Merge: ec1078c 9808fbc Author: Damian Johnson Date: Fri Jan 4 19:12:19 2013 -0800 Controller get_streams() method and close_stream() test Merging patches by Sean for... * adding a get_streams() method to the controller https://trac.torproject.org/7859 * integ test for the close_stream() method he added earlier https://trac.torproject.org/7687 commit 9808fbc0abf13ea0a6e5cef3249c286e36592494 Author: Damian Johnson Date: Fri Jan 4 09:59:44 2013 -0800 Minor get_streams() testing revisions Minor tweaks for the get_streams() tests, for instance the 'stream' argument was unused (likely used to assert that the stream lists were equal, which didn't work due to mock objects leading it to be replaced when a len() check). commit 7b2f6be091f4bb340277a49565bf37ec3a97e3c2 Author: Damian Johnson Date: Fri Jan 4 09:34:39 2013 -0800 Adding a default arg to get_streams() Adding a default argument to the new get_streams() method (see ffdd61e). commit 40a8d3da7505c3b62c8d1a2c353b17d3635bcab8 Author: Damian Johnson Date: Fri Jan 4 09:31:28 2013 -0800 Raising get_streams to the top of the category This might be yet another "Damian's conventions are weird" thing that I'll come to regret, but I placed the getters above the modification funtions for circuits, so doing the same for streams. commit 4f8efd38bbaccb573d19f8ce98f1d590bd8918f5 Author: Sean Robinson Date: Fri Jan 4 07:37:33 2013 -0700 Add integration tests for Controller.close_stream() Signed-off-by: Sean Robinson commit 7e5377906b6b0e0bcc20fb442fb11554df929868 Author: Sean Robinson Date: Wed Jan 2 05:52:29 2013 -0700 Add a Controller.get_streams method and its tests Re-use Ravi's idea for parsing get_info("circuit-status") responses with get_info("stream-status"). The stream status GETINFO reply contains just the mandatory parts of a STREAM event, so the event optional keyword arguments are not (yet) used. Signed-off-by: Sean Robinson commit ec1078ccc39af8d154ef2f70781f1a3787be6789 Author: Damian Johnson Date: Thu Jan 3 09:42:39 2013 -0800 Whitelist support for shell commands Shell commands have at times been the bane of my existance. They live outside the PATH (causing our is_available() to fail), and now it turns out that subprocess.Popen() fails to find them too. In retrospect both make sense but at first both are worthy of some head scratching. Adding a whitelist we can add shell commands to so they'll behave as we expect. commit cde6de6bb3c132ba657ed2135ece83b9d3b60190 Author: Damian Johnson Date: Thu Jan 3 08:46:05 2013 -0800 Config changes broke integ tests Shame on me for not running the integ tests (they're sluggish enough that I run the unit tests a lot more). I both missed a get_str_csv() usage (grepped the wrong thing) and an integ test with a synchronize() call. commit 76205611acb95231226bf42c4c6b20388ef3cfe7 Author: Damian Johnson Date: Thu Jan 3 08:35:09 2013 -0800 system.call() support for a default The 'suppress_exc' argument is a relic from long ago when I frequently made suppression the default in arm. Stem does things better, following "Errors should never pass silently. Unless explicitly silenced." A default argument lets us do the same thing as a 'suppress_exc' boolean, but with more control. It's also more uniform with our controller. commit 5ee4b1b67876dc1ddba9aa21dbec6ef8871a8ba6 Author: Damian Johnson Date: Thu Jan 3 08:24:26 2013 -0800 expand_path() pydoc inaccurate The pydocs for system.expand_path() said that it was unix specific. However, Beck expanded this to support Windows in 62e51e9 and others. commit 0439570531bb909ba85b9b4ede4070e96994083d Author: Damian Johnson Date: Wed Jan 2 20:21:33 2013 -0800 Minor improvement to to_camel_case() function On reflection arm has a better function signature than us for to_camel_case(). Adopting it. commit 8a746ba1023a521cdb09a068ab68102a0ca75498 Author: Damian Johnson Date: Wed Jan 2 18:48:38 2013 -0800 Error message printed wrong arg Drats. Just got an unexpected exception for this that I can't repro, and the stacktrace had the wrong attribute... >:( commit 45f82fab334265d8ca6162192080a3e03853f323 Author: Damian Johnson Date: Tue Jan 1 15:15:31 2013 -0800 Dropping the Config's synchronize() method Choice is highly overrated. Actually, in the world of libraries providing too many options of doing a single can make things damn confusing. The Config's synchronize() method was a great improvement in its day, but now that we have config_dict() it's redundant. Hopefully our module's docs are now more friendly for newcomers. commit 93e1e8a1c4c5c52f09a9b974f868a223a1cda1ca Author: Damian Johnson Date: Tue Jan 1 14:03:53 2013 -0800 Dropping Config's get_int_csv() and get_str_csv() Many, many moons ago when arm first began it used integer enumerations (as is often the case for python projects). I added get_int_csv() to allow users to configure those enums, and later added get_str_csv() when we moved to our Enum class. Long story short, both are legacy methods for a bygone age before we had our new config_dict() pattern. We don't use it, and now neither does arm. Time to drop the cruft. commit 2b43313d72c4a73ff5055e2bf5c5b162d8f1f4b6 Author: Damian Johnson Date: Tue Jan 1 13:04:56 2013 -0800 Configuration parse_enum() function Convenience function to do the same thing as parse_enum_csv() but for a single enum value. commit 3521953c1be9110f3c22cc20bf697990a5287e3a Author: Damian Johnson Date: Mon Dec 31 23:26:31 2012 -0800 Configuration parse_enum_csv() function While swapping over arm's config usage I realized that we likely don't need the int or str csv functions. In practice the csvs have been for enumerations. Adding a helper function that better handles configurations for enum values. commit 661eee8dbe2f02cd714a363a48ed3ffd050a013c Author: Damian Johnson Date: Mon Dec 31 16:53:15 2012 -0800 Wrong class in docs for CircuitEvent We were missing the 'response' part of the path, breaking interlinking. commit 7cbb066c214c227fb65dece7b282aa314a731d41 Author: Damian Johnson Date: Mon Dec 31 15:16:58 2012 -0800 Updating TorCtl url TorCtl is no longer on the volunteer page so pointing to its git repo instead. commit 683b1ba479f2aff3c277e0ff53bdc8b1f81664af Author: Damian Johnson Date: Mon Dec 31 01:43:00 2012 -0800 Support for consensus 'a' lines Adding support for the 'a' lines in version 3 router status entries. These contain IPv6 addresses and port lists. The port lists will make this unweildy for users but that's out of our hands. commit 7df236509dca6d32d5cebb8aa0b634d3b3d962b1 Author: Damian Johnson Date: Mon Dec 31 00:47:58 2012 -0800 Duplicate network status 'm' lines were ignored The 'm' lines in version 3 router status entries can appear multiple times, but we were only parsing the first instance. commit cdf2c13bbe5db5a85670760572f425dffb6ca897 Author: Damian Johnson Date: Mon Dec 31 00:33:30 2012 -0800 Server descriptor parsing for 'ipv6-policy' lines Adding support for IPv6 exit policies in server descriptors. Descriptors with these are already in the wild. https://trac.torproject.org/7826 commit 87b05342286546f603ea4bca2aee68e2b95d875e Author: Damian Johnson Date: Mon Dec 31 00:12:40 2012 -0800 Network status integ tests should check for unrecognized lines Probably the most useful integration test that we have is the check for unrecognized lines in the server descriptors. That one test has found dozens of upstream tor bugs (mostly new additions without spec updates). It turns out that we were missing this check for network statuses, and it's already tripping on an issue. Oops. commit 974b71c65a3e67da25254418856c4a65c5b0993a Author: Damian Johnson Date: Mon Dec 31 00:07:34 2012 -0800 Descriptor reader callbacks for when reading a file While writing a script for 'https://trac.torproject.org/7828' I realized that it would be nice to be able to add hooks for when reading a file. commit d35b9c737e320500fa6cfdfa874534cfbe6f1b3a Author: Damian Johnson Date: Sun Dec 30 21:26:34 2012 -0800 Using a lightweight rule counterpart for microdescriptor policies The exit policies used in microdescriptors are wonderfully simple. There's no point in storing full ExitPolicyRule instances for them, so making an alternate lightweight subclass. Advantages are faster for parsing and less memory usage. Disadvantage is that this doesn't use the ExitPolicyRule constructor, so we'll need to take care that ExitPolicyRule changes are reflected in this subclass. This dropped the runtime of the network status integ tests on my system from 9.4 to 8.3 seconds. commit da65f282699d3e8e763e1f3ba3c7612cc970178a Author: Damian Johnson Date: Sun Dec 30 21:00:50 2012 -0800 Lazily parsing ExitPolicy Why parse exit policies that we never use? ExitPolicy instances exist numerous places, note notably in the descriptors. Users may never use or care about the exit policy, so there's no point in putting effort into parsing them until they're needed. The advantages is that this means faster untime and less memory usage if they go unused. Down side is that we won't pick up on malformed content until they're used (handy for tests, but not somethign that users care about). This dropped the runtime for the server descriptor integ tests on my system from 10.1 to 7.7 seconds. commit c34e90b55d744b0e19922d61b638c8987b089233 Author: Damian Johnson Date: Sun Dec 30 19:39:54 2012 -0800 Tor 0.2.2.9 required for tests with new/extend_circuit() Our tests utilizing new_circuit() and extend_circuit() do not provide an explicit path. This breaks the tests if tor isn't version 0.2.2.9 or later. https://trac.torproject.org/7833 commit af6fd83f74f6f39981cb32e1f871421755edfdaa Merge: dc003e8 f716fed Author: Damian Johnson Date: Sun Dec 30 19:05:14 2012 -0800 Circuit attachment and getter defaults Originally I set out to merge in Ravi's branch for attach_stream (https://trac.torproject.org/7152) but this turned into an odyssey of general Controller improvements. The most notable bits (besides attach_stream) are... * All of the Controller's getter methods now support a 'default' argument. https://trac.torproject.org/7832 * Added methods to get circuits. This was part of Ravi's branch though I expanded it a bit. * The new_circuit() and extend_circuit() now have an 'await_build' option to block until the circuit is ready to be used. commit f716fedc97ea3cc522571ad8bd505e847379c0d7 Author: Damian Johnson Date: Sun Dec 30 18:59:11 2012 -0800 Allow for blocking circuit build and extension Callers of new_circuit() and extend_circuit() will often (maybe usually?) want their call to block until the circuit is actually ready to be used. Unfortunately this is a little tricky for them to hack together on their own since it requires a listener (polling doesn't work reliably - after a couple of hours of hair pulling I now know that all too well). Hence doing this work for them. This includes a CircuitExtensionFailed exception so we can communicate the circuit event that reported the failure. commit f87d9d75ef76b8755e5a3f78ae85002050ef8db0 Author: Damian Johnson Date: Sun Dec 30 16:40:32 2012 -0800 Using get_socks_listeners() for attachstream test Using our shiny new get_socks_listeners() method in test_attachstream. Besides being nicer, this will work with older versions of tor. commit 11ef43c64479af8a9fe50b01e980d02234671977 Author: Damian Johnson Date: Sun Dec 30 16:32:16 2012 -0800 Reverting __LeaveStreamsUnattached when test fails Putting the cleanup activity for test_attachstream() into a finally block so we still do it when the test fails. We also forgot to remove the handle_circ listener. commit 9401596d2d07a2baadab5c4d144566f99d16df7d Author: Damian Johnson Date: Sun Dec 30 16:28:17 2012 -0800 Renaming attach_stream()'s hop argument Renaming the method's 'hop' argument to 'exiting_hop' to be more descriptive. Also tweaking how we make the query a bit. commit 86de12ecab643b809d2e75265512e0dc4267dd12 Author: Damian Johnson Date: Sun Dec 30 16:23:42 2012 -0800 Reordering Controller methods Rearranging the Controller's methods to match the header. This'll both make it easier to find things and rearrange our documentation. commit ffdd61ea41ca844047ac57725639a7cf0e22d41d Author: Damian Johnson Date: Sun Dec 30 16:03:30 2012 -0800 Uniformly providing a default argument for getters As disussed on 'https://trac.torproject.org/7832' it's important from a usability standpoint for all of our getter methods to support a 'default' argument. It's also nice to have class-wide uniformity on this since it makes the Controller class more intuitive to use. This also reorganizes the method listing in the header to group related methods, and has a couple small bits of backward incompatability (trying to get these breakages in early before making our initial release announcement)... * renamed protocolinfo() to get_protocolinfo() to match the other getters * renamed one of the 'circuit' arguments to 'circuit_id' (we did that for all of the methods but one) commit 22f167fa48bf51d0f66736624293050bb022a936 Author: Damian Johnson Date: Sun Dec 30 15:03:49 2012 -0800 Adding a default arg to get_circuits() Getter methods should have a default argument (though admittedly I've been doing a bad job of requiring this, #7832). commit cb9b4b605e8bdb639b22b2e58e5d2e80da9dcb56 Author: Damian Johnson Date: Sun Dec 30 14:52:41 2012 -0800 Adding a get_circuit() Controller method While reading through the tests I realized that we frequently wanted a method for 'gimme the circuit with id X', so adding a method for that. commit b01a2e9b1474f6ff858ff85f40bcdcb68b8c66ac Author: Ravi Chandra Padmala Date: Sun Dec 30 23:17:13 2012 +0530 use SocksListenAddress instead of SocksPort commit 4d9a71295f5f4afcfb41f3f0f75ef4ecc45b2124 Author: Ravi Chandra Padmala Date: Sun Dec 30 23:03:48 2012 +0530 Add missing import commit d70e98ab0c58aa5a209b24394852ecb7ae39e563 Author: Ravi Chandra Padmala Date: Wed Dec 26 14:44:38 2012 +0530 Make test_attach_stream more robust commit 66af2a98e85db0442687b5a684013a41bdf50d9c Author: Ravi Chandra Padmala Date: Sun Dec 23 22:55:32 2012 +0530 Make test_extend_circuit and test_close_circuit use get_circuits commit bcd03831cace7352a7cbdf4e9f4dd651d86a48af Author: Ravi Chandra Padmala Date: Sat Dec 22 17:25:19 2012 +0530 Make test_close_circuit use Controller.get_circuits() commit 299a872c0b7ba26babec3febf8c0a8adad09f4c7 Author: Ravi Chandra Padmala Date: Sat Dec 22 17:17:47 2012 +0530 Fix test_repurpose_circuits to work with string ids commit 631b210c59fcbeadd3e75a1b1ac9e9f9a635d142 Author: Ravi Chandra Padmala Date: Sat Dec 22 14:33:26 2012 +0530 Fix test_get_circuits to work with string ids commit e0929d2cde0b57232dea966192dd4fe954667d93 Author: Ravi Chandra Padmala Date: Mon Dec 17 08:09:18 2012 +0530 Variable naming consistency commit 32a54a71f8391b5300e6cae5a3f558f190160995 Author: Ravi Chandra Padmala Date: Tue Jul 3 10:35:52 2012 +0530 Implement Controller.attach_stream commit 1b9ffa597408e9b7f0f8863e8c7a2f66d9922e1c Author: Ravi Chandra Padmala Date: Wed Dec 5 08:08:18 2012 +0530 Add a Controller.get_circuits method commit dc003e8bb26bf9261718ee2ac6ba5a3109972c44 Author: Damian Johnson Date: Sun Dec 30 02:23:12 2012 -0800 Moving the ExitPolicyRule's masked_bits behind a method In looking at TorCtl's ExitPolicyLine class I'm realizing that we may want to use our mask attribute as our main subnet representation rather than the masked_bits. Changing the class so the masked_bits are accessed via a method so we'll have the flexability to change this in the future. commit d1d00b635c53c885ff930fe28771cef51f1edebb Author: Damian Johnson Date: Sun Dec 30 01:18:59 2012 -0800 Flagging the ExitPolicy's set_default_allowed() as private The ExitPolicy's set_default_allowed() method is there to support microdescriptor policies, not for our users. Acually, exit policies were entirely immutable with the exception of this attribute. Changing the method to be private. commit bbb2d5a44a898e1a7202d876fe27cc87065ffd3f Merge: bf28436 79fc8cd Author: Damian Johnson Date: Sat Dec 29 23:34:13 2012 -0800 Adding get_socks_listeners() to Controller Adding a get_socks_listeners() to the controller to provide the (address, port) tuples where tor is listening for SOCKS connections. This also includes some mocking changes to better support the tests. Changes are from Sean... https://trac.torproject.org/7783 https://trac.torproject.org/7784 commit 79fc8cd7514cda072abf855b6a1908f0dd10bb17 Author: Damian Johnson Date: Sat Dec 29 23:27:06 2012 -0800 Revisions for prior mocking and controller changes Changes include... * Rewrite of get_socks_ports() with the following changes... * Renaming to get_socks_listeners() since it provides (address, port) tuples, not just ports. * Better error handling. Previously malformed content such as invalid ports would cause us to raise ValueErrors. * The method wasn't documented as raising any exceptions, but if the get_conf() calls fail then a ControllerError will be raised. * Minor corrections to the return_for_args() examples. * Renamed return_for_args()'s "method" arg to "is_method" (it's nice when the an arg name can imply the type). * I'm not sure what was meant by "Also, remove the the space in the join so that there is only one parameter passed to the string substitution.". Personally I'd find a message like... "Unrecognized argument sent for return_for_args(). Got 'foo, bar' but we only recognize 'foo, blarg'." ... to be a bit nicer than... "Unrecognized argument sent for return_for_args(). Got 'foo, bar' but we only recognize 'foo;blarg'." commit 3a2875f22d1bf5dba6171e98807ef22f2a02c729 Author: Sean Robinson Date: Sat Dec 22 06:16:56 2012 -0700 Add get_socks_ports method to Controller Adds a method to find the SOCKS listeners on a controlled tor. This method is tested on tor versions 0.2.1.32 - 0.2.4.6-alpha and provides a similar response regardless of the underlying tor process. Also includes unit and integration tests for the new method. Signed-off-by: Sean Robinson commit 667cafa0c5fbdcc74dfb781ef8f55ac84d727112 Author: Sean Robinson Date: Sat Dec 22 06:14:37 2012 -0700 Replace SocksPort with SocksListenAddress in base torrc This is the most widely accepted format for specifying a SOCKS port, it works with tor 0.2.1.x - 0.2.4.x. SocksPort is preferred for tor 0.2.4.x and later, but this also acceptable. Signed-off-by: Sean Robinson commit d3cd0841393b20b4b333be0b39fb4890609426cf Author: Sean Robinson Date: Sat Dec 22 04:34:36 2012 -0700 Add explicit declaration of method use for mocking.return_for_args The previous way of removing 'self' for MockClass objects only worked for fully mock objects. By using an explicit declaration, mocked methods for real objects can be used. Signed-off-by: Sean Robinson commit 1dbb7befdbd10169a35e27977a0736a9dc9465d0 Author: Sean Robinson Date: Sat Dec 22 04:29:46 2012 -0700 Handle keyword arguments with mocking.return_for_args() Allow a structured way to match keyword arguments to be included in the mapping keys for return_for_args(). This changes the API for return_for_args to require tuples as the mapping keys. Signed-off-by: Sean Robinson commit ef5c552e59017936699048d680f5f377f1b60d66 Author: Sean Robinson Date: Sat Dec 22 04:17:38 2012 -0700 Convert keys before reporting error Non-string argument keys caused the ", ".join() to fail with a TypeError. Also, remove the the space in the join so that there is only one parameter passed to the string substitution. Signed-off-by: Sean Robinson commit bf28436368aea5a0cad3545de9406934434a9f1a Author: Damian Johnson Date: Sun Dec 23 11:58:38 2012 -0800 Fixing eronious 'missing with' warning Running our tests gave a working that network.py was missing an import of the 'with' statement. This was wrong - the with statement was in its pydocs. The checker tries to be smart about what is and isn't documentation by looking for block quotes but an unusual string confused it... """ Base error for proxy issues. """ I avoid using block quotes unless things span multiple lines which is why I didn't account for this. We could fix the checker (... which isn't a bad idea) but for the moment I'm just dropping the odd block quotes since it'll probably introduce extra whitespace before and after the message in the sphinx docs. commit 0c4cb6d69e1b47faad757402fb666d6310252c09 Merge: cfc745e 8aba54d Author: Damian Johnson Date: Sat Dec 22 16:29:51 2012 -0800 Utility to test client usage in integ tests Utility class to make it easy to use the tor instance that we have for testing for client traffic... https://trac.torproject.org/7693 commit 8aba54d53bc2738ecea503ec09e71a32edb5cc0a Author: Damian Johnson Date: Sat Dec 22 16:27:32 2012 -0800 Dropping test.network.Socks interlink The ":class: `~test.network.Socks`" had a couple issues... * there's a space between the :class: and back-tic (not sure if that's a problem for sphinx, but definitely different) * we can't use things like :class: within the preformatted text block commit 31cbbbad01230b26722c7524617518f2228e55f1 Author: Sean Robinson Date: Tue Dec 18 06:21:13 2012 -0700 Advertise network module inside test Signed-off-by: Sean Robinson commit 270f173deddd2e3dded794d23306f1f32b78d400 Author: Sean Robinson Date: Mon Dec 17 20:03:29 2012 -0700 Add SocksPatch to socksify network-using Python code A context manager which helps re-direct network socket traffic, from existing code, through the tor process. Signed-off-by: Sean Robinson commit 4485657e3840f0ea9d9fc96623dd7f50a476a79d Author: Sean Robinson Date: Mon Dec 17 20:02:22 2012 -0700 Add Socks class to handle a socket connection through a SOCKS5 proxy I attempted to keep this generic and open enough that IPV6 support can be easily added at a later date. Signed-off-by: Sean Robinson commit bf6651aa0121e51e938e87608a4df69c0e80c917 Author: Sean Robinson Date: Mon Dec 17 19:20:23 2012 -0700 Add exception classes to test.network module Signed-off-by: Sean Robinson commit ef0cee8123578ed8ace0e3304735eb683dc6ca5e Author: Sean Robinson Date: Mon Dec 17 19:12:37 2012 -0700 Add a new module to test to support network connections through tor This module aims to provide basic connectivity options on which tests may be built. Signed-off-by: Sean Robinson commit cfc745e95010b4c86a8d794c118bfafafee4d59d Author: Sean Robinson Date: Fri Dec 21 09:17:23 2012 -0700 Documentation and output string fixes Signed-off-by: Sean Robinson commit a5b05edefbbcbc7261e40a5842a1ef901a60a25c Merge: f60e327 5687229 Author: Damian Johnson Date: Sat Dec 22 14:30:29 2012 -0800 Integ test fixes concerning string circ/stream ids Fixes from Sean to fix integration tests broken by commit 6711458... https://trac.torproject.org/7778 commit 568722919c499c1e409680fe38fb3471a97b425c Author: Sean Robinson Date: Sun Dec 16 20:02:49 2012 -0700 Fix test_extendcircuit by removing int() conversion Traceback: File "./test/integ/control/controller.py", line 464, in test_extendcircuit self.assertTrue(filter(lambda x: int(x.split()[0]) == circ_id, controller.get_info('circuit-status').splitlines())) AssertionError: [] is not true Also, change to standard variable name "circuit_id". commit 848d4c95b5c6f908755ac7db4019a3ac2bf2e9eb Author: Sean Robinson Date: Sun Dec 16 19:46:41 2012 -0700 Fix test_repurpose_circuit crash by changing directive in format string Traceback: File "./test/integ/control/controller.py", line 486, in test_repurpose_circuit circ = filter(re.compile("^%i " % circ_id).match, circuit_output.splitlines())[0] TypeError: %d format: a number is required, not str Also, change to standard variable name "circuit_id. Signed-off-by: Sean Robinson commit 9119f9f0d0091a030654002a2e7ae6ccf91a789f Author: Sean Robinson Date: Sun Dec 16 19:32:22 2012 -0700 Fix test_close_circuit crash by concatenating strings Traceback: File "./test/integ/control/controller.py", line 521, in test_close_circuit self.assertRaises(stem.InvalidArguments, controller.close_circuit, circ_id + 1024) TypeError: cannot concatenate 'str' and 'int' objects Also, change to standard variable name "circuit_id". Signed-off-by: Sean Robinson commit f60e3272f78acdc3c66c07f897688d1e3a381ca3 Author: Damian Johnson Date: Fri Dec 21 08:19:10 2012 -0800 Dropping duplicate stop() call The test runner is ran within a finally block that calls() stop. Reguardless of if tor fails to start or not stop() will be called, so this call was a duplicate. Calling stop() multiple times is fine (the method is idempotent), but it logs each time so we were saying "Shutting down tor... done" twice. It's still possible for us to give double messages in some other use cases so we should address this more generally later, though I'm not sure offhand what that fix should be. commit bf2f7f2a461b535ddd21728e29dc2091ef35a74d Author: Damian Johnson Date: Fri Dec 21 08:06:15 2012 -0800 Reporting testing as having failed if tor can't start As pointed out by Simon on... https://lists.torproject.org/pipermail/tor-dev/2012-December/004296.html ... we presently say that testing has passed if tor fails to start. This is wrong. commit e22a753058a2b9c61301c62e2d2123ccea046f29 Author: Damian Johnson Date: Fri Dec 21 08:02:10 2012 -0800 Sanity testing the that the tor command points to tor Adding some basic sanity tests to launch_tor() so that we provide a better error message if our tor command is wrong. This is to address the following, which was admittedly really confusing error output... https://lists.torproject.org/pipermail/tor-dev/2012-December/004296.html Caught thanks to Simon. commit b20d7acc9cd6fad1fd17f8dddccf8473d0034a09 Author: Sean Robinson Date: Sat Dec 15 15:32:47 2012 -0700 Add more test cases for re-attaching event listeners This covers the three anticipated cases when re-attaching event listeners is required: Controller.authenticate(), stem.connection.authenticate(), and BaseController.msg("AUTHENTICATE"). Signed-off-by: Sean Robinson commit f5620f2dbd4bbd5c3bdcb2c19abad765f4e324c3 Author: Sean Robinson Date: Sat Dec 15 15:28:11 2012 -0700 Use threading.Event in event listener tests wait() up to two seconds, for a threading.Event to be set() in the test event listener before continuing assertions. This should speed up success or failure in these tests, even if not by much. test_event_handling() goes from a constant time of 5 seconds, to a variable 4-8 seconds, with 4 seconds most common in my testing. test_reattaching_listeners() goes from a constant 6 seconds, to a variable 4-6 seconds. Signed-off-by: Sean Robinson commit 6bc92816dc4356a204fdde0f160ee344875143a3 Author: Damian Johnson Date: Sun Dec 16 18:50:46 2012 -0800 Adding get_latest_heartbeat() Adding a method to the BaseController to get the time that we last heard from tor. This is useful to figure out if our controller connection has stalled, but only works if we're continually using our controller (such as by listening for BW events). commit 229a6319109372dc529106516a9fcb65f5c2eb73 Author: Damian Johnson Date: Sun Dec 16 18:35:41 2012 -0800 Making the Controller's enabled_features private The Controller's 'enabled_features' attribute wasn't documented so it was probably meant to be private. Changing its name to reflect that. commit 844073f3813d496fe4d5037283413d837d1ae325 Author: Damian Johnson Date: Sun Dec 16 13:42:10 2012 -0800 Adding the requirement for 'PortForwarding' Adding in the version requirement for the 'PortForwarding' torrc option. This was hardcoded in arm. commit 5b45d3125c811067d7c3ba0a11e324423c88d925 Author: Damian Johnson Date: Sun Dec 16 11:03:07 2012 -0800 Revising error handling for get_conf() The Controller's get_conf() function was buggy if the configuration value was unset. It's documented as returning the default, but instead it returned None or [None] depending on if 'multiple' was set. On reflection the pydocs for these functions were pretty unfriendly, so giving them an overhaul. Functionally this changes two things... 1. When you query for an unset value it will... * provide the default value if you gave it one * provide None if multiple was False, and [] if it was True 2. The get_conf_map()'s 'default' argument is now the defaultly *mapped value* rather than the default *response*. For instance... # old behavior get_conf_map('NoOption', 'Blarg') => 'Blarg' # new behavior get_conf_map('NoOption', 'Blarg') => {'NoOption': 'Blarg'} This means that the get_conf_map() pydocs can now assure the caller that they'll always get a dict, and it also better matches the get_conf() handling for default. commit fc4bdf7ab686b4bbd12e3aefc83d78d0a171df60 Author: Damian Johnson Date: Sat Dec 15 16:22:30 2012 -0800 The str_tools functions didn't handle negative values Functions like str_tools.get_size_label() provided labels using the smallest unit type if the count was negative. For instance, '-2000000' would be translated into '-2000000 B' rather than '-1 MB'. An inline comment seems to indicate that this was on purpose, but I haven't a clue why. Fixing it. commit a201c0c5f4b92b119f89fa39bd43680d67a64817 Merge: e30f2df 256676d Author: Damian Johnson Date: Sat Dec 15 13:46:08 2012 -0800 Unit tests for event error conditions Testing expansion from Sean for malformed event content, plus a couple fixes. https://trac.torproject.org/7736 commit 256676d5e686fb544927a715335c567d693254cc Author: Damian Johnson Date: Sat Dec 15 13:06:41 2012 -0800 Event testing revisions Minor revisions for the prior event testing additions... * Adding comments saying how the 'BAD' events are malformed. Some of them took me a few seconds to figure out. :) * Renaming the 'ORCONN_CONNECTED_BAD_*' events to 'ORCONN_BAD_*' since it's not especially important that these are connection events (and in one case it isn't). * Putting ORCONN_BAD_* events on a single line to match the non-bad counterparts. The rough convention that I've been using is to add a newline if there's either a key/value argument or a really long positional arg. * Renaming ADDRMAP_BAD_3 to ADDRMAP_ERROR_EVENT. I like the ADDRMAP_BAD_1 and ADDRMAP_BAD_2 names since they make it clear that they're malformed input. However, ADDRMAP_BAD_3 is a valid response where tor is conveying that an error occurred. * Actually, I think that your STREAM_SENTCONNECT_BAD_1 event reveals a bug. The spec doesn't seem to indicate that the target is optional, so making it mandatory. commit b3aca85301439fd2441d098775b9c56d7365c506 Author: Sean Robinson Date: Fri Dec 14 16:28:25 2012 -0700 Add test cases for STREAM events These test events exercise all the exceptions in StreamEvent._parse() and the case of an empty target address:port. Signed-off-by: Sean Robinson commit 38190ab9cbedf9844022fe02f4385f1b129c5f52 Author: Sean Robinson Date: Fri Dec 14 16:15:53 2012 -0700 Move NEWDESC sample events to alphabetic position Signed-off-by: Sean Robinson commit 58c48067ba1b901bdb1d4151181949591c97b60a Author: Sean Robinson Date: Fri Dec 14 16:12:28 2012 -0700 Add test case for invalid STATUS_* event type Signed-off-by: Sean Robinson commit 146f32473a3c0f9f91c0e4a721e92906d293a63e Author: Sean Robinson Date: Fri Dec 14 16:11:37 2012 -0700 Change status event type for CONSENSUS_ARRIVED According to control-spec.txt, CONSENSUS_ARRIVED is in the STATUS_GENERAL family. While it is not vital for the faked test events, let's make it accurate. Signed-off-by: Sean Robinson commit 04633d4243f88f232a39daf5cbffbb228ee8f6bf Author: Sean Robinson Date: Fri Dec 14 15:44:35 2012 -0700 Add test cases for ORCONN events Rename ORCONN_CONNECTED to ORCONN_LAUNCHED and use ORCONN_CONNECTED for more complex _parse() method test. Adds three bad events to trigger exceptions in _parse(). Move sample lines to alphabetic position among other sample cases. Signed-off-by: Sean Robinson commit df01f3737143545493652437680db52ea8e7f337 Author: Sean Robinson Date: Fri Dec 14 15:20:13 2012 -0700 Add test cases for CLIENTS_SEEN event These sample events hit every exception possible in the ClientsSeenEvent _parse() method. Signed-off-by: Sean Robinson commit 6cf5422575dd1e3e6fe2237d6bbc2f45e2096e1a Author: Sean Robinson Date: Fri Dec 14 15:09:00 2012 -0700 Add test cases for CIRC_MINOR events Checks the same error conditions added to CIRC event tests in previous commit. Also, swap test_circ_minor_event() and test_clients_seen_event() to alphabetic order of appearance. Signed-off-by: Sean Robinson commit c5db5cb1be07da0da92b30559557e72285d3fc37 Author: Sean Robinson Date: Fri Dec 14 15:00:49 2012 -0700 Add test cases for CIRC events Check ProtocolError is raised when the TIME_CREATED time cannot be parsed or the CiruitID is not valid. Signed-off-by: Sean Robinson commit 48daff039d95d8223ecd83cbd3af59a2ee00dc37 Author: Sean Robinson Date: Fri Dec 14 14:58:08 2012 -0700 Add test cases for BUILDTIMEOUT_SET events Check integer and float conversion failures raise a ProtocolError. Signed-off-by: Sean Robinson commit 97dc4108404a1c0f676845538a542f98085ffb4a Author: Sean Robinson Date: Fri Dec 14 14:34:59 2012 -0700 Add test case for missing BW read value Signed-off-by: Sean Robinson commit f1b81e062c2b1ec646f99b346b98dd3d4ec38057 Author: Sean Robinson Date: Fri Dec 14 14:30:40 2012 -0700 Test error conditions for ADDRMAP events Add tests for missing quotes before (ADDRMAP_BAD_1) and after (ADDRMAP_BAD_2) the expiry positional argument. Add a test for an error event (ADDRMAP_BAD_3), constructed from control-spec.txt. Also, move the sample event lines to their alphabetic position in the list of other event samples. Signed-off-by: Sean Robinson commit 26a39c9ac5bf77860f2a10dd776c95ce46ef6380 Author: Sean Robinson Date: Fri Dec 14 14:13:38 2012 -0700 Add tests for stem.response.events.Event This tests the base class used to build named event classes (e.g. ADDRMAP, BW, CIRC, etc.). Mainly this documents existing behavior so we can check future changes to the class do not surprise us with breakage. Signed-off-by: Sean Robinson commit 8a0c9f58046670c7a808e2e4a79b82244405e847 Author: Sean Robinson Date: Fri Dec 14 14:05:13 2012 -0700 FIX: include positional argument name which is missing proper quoting This was discovered with a spot coverage test. These were the only lines not run in the Event class while I was adding tests for the stem.response.events module. I just *had* to write a test for these two lines and it turned out they had a problem. Signed-off-by: Sean Robinson commit e30f2df26221004fc2c535e5ef5e0b2c8518e3b2 Author: Sean Robinson Date: Fri Dec 14 17:12:27 2012 -0700 Remove extra checks for circuit id and stream id validity CircuitID and StreamID may not be None, so raise ProtocolError if either ID is None. Inside is_valid_circuit_id() and is_valid_stream_id(), isinstance(entry, str) already returns False if None is checked. Add None as invalid IDs in tests. NOTE: Do not try this with is_valid_fingerprint() and is_valid_nickname(), as the fingerprint or nickname may be None in certain cases. Signed-off-by: Sean Robinson commit 22286e9dc3fea710182984dd320f4e419cc5a425 Merge: 6711458 9a1601d Author: Damian Johnson Date: Fri Dec 14 09:27:50 2012 -0800 Script for republishing stem's site Now that we're moving our site from ReadTheDocs to a tpo host (staticiforme) we need our own method for automatically republishing our sphinx docs. To do this we're using a cron process that runs every five minutes which... * Pulls stem's git repo. If it has changed... * Run 'make html' to rebuild the docs. * Republish the content by calling 'static-master-update-component'. Many thanks to weasel for working with me on this, and pointing out the numerous mistakes I've been making. commit 9a1601db33f58b044aa2375db8f97f364bdd849c Author: Damian Johnson Date: Fri Dec 14 09:12:48 2012 -0800 Logging site republication runtime Again, mostly a commit to exercise that site republication is now working with a cron setup. commit 85241a8cdb145edce2655dda69d2c822276095a1 Author: Damian Johnson Date: Fri Dec 14 08:59:16 2012 -0800 Only logging an init message when repeating I'm about to change the republication script to be a cron process rather than a daemon, but first I want to drop its initializtion log message when running in a one-off fasion. Otherwise we'll get a log entry every five minutes for it. commit 91fc3b8a0d7b33eddfee52d4bef29604923fa19f Author: Damian Johnson Date: Fri Dec 14 00:32:01 2012 -0800 Correcting mistake in the tutorial Correction thanks to arma... 23:57 < armadev> tutorial looks nice! 23:57 < armadev> s/lets/let's/g though commit ba930a28b5e2cd7f43c4b85cdd61366d58103892 Author: Damian Johnson Date: Thu Dec 13 23:40:33 2012 -0800 Revising our site url Honestly I'm just making a trivial fix so I can see if my republisher script finally works or not (fingers crossed!). commit 2fcdc4bbd6ebcb27725d45cffa3461b4d833913e Author: Damian Johnson Date: Thu Dec 13 23:34:43 2012 -0800 Propagating cwd for republish calls The script's a little nicer (and closer to working) when we pass through a cwd rather than trying to set it separately. commit 0700b831ff4068bc6ea83e33e8b5c63fdeaaeb0b Author: Damian Johnson Date: Thu Dec 13 23:10:01 2012 -0800 Dropping disown from republish The disown command isn't available on the webserver, and on some experimentation seems to be unnecessary. The daemon process survives after logout. commit ffde535388f2996a9adccdccc75a22a0176d0d35 Author: Damian Johnson Date: Thu Dec 13 22:56:17 2012 -0800 Republishing --repeat should... well, repeat Forgot to make the script continually run when --repeat was provided. commit d2b2b9757cf56be09d8841184f493ae478247c0b Author: Damian Johnson Date: Thu Dec 13 22:53:52 2012 -0800 Passing args along to republish.py commit 9dbf340fde1a7c17b40cf3e01c0bd99684b91ce1 Author: Damian Johnson Date: Thu Dec 13 22:52:28 2012 -0800 Adding execute to republish commit 72832baa56dfbf0cff0980746db708c897017603 Author: Damian Johnson Date: Thu Dec 13 22:50:34 2012 -0800 Script to republish stem's website content Script for stem's webserver (staticiforme) to provide a daemon that periodically publishes our site's content. commit 6711458be83b2050b058b7528bccb4e15c2c6a25 Author: Damian Johnson Date: Thu Dec 13 08:39:18 2012 -0800 Circuit and stream ids are strings Ids for streams and circuits presently look like ints, but stem should handle them as strings as per the control-spec... ; Unique identifiers for streams or circuits. Currently, Tor only ; uses digits, but this may change StreamID = 1*16 IDChar CircuitID = 1*16 IDChar IDChar = ALPHA / DIGIT Issue caught by Lunar^. This should be invisible to callers (they can probably still provide us with ints), the only noticeable difference being that new_circuit() and extend_circuit() now provides a str rather than int. commit 9b5ff19be35ebbf8c75772a63406ba23762b37de Author: Damian Johnson Date: Wed Dec 12 09:32:02 2012 -0800 Lowering cache hits to the TRACE runlevel In practice logging GETINFO and GETCONF cache hits can be very noisy (within a minute of running it with arm there were around a hundred duplicates). Lowering the visability of those events. commit a7b275a3d64301da4d23137dbadd4842cc7a041f Author: Damian Johnson Date: Wed Dec 12 08:13:39 2012 -0800 log_to_stdout() only printed at TRACE level Ok, the logging module is *trying* to make me hate it. I haven't a clue what's wrong with the basicConfig() function, but when I try to use it the 'level' argument is ignored and it always spews its output at the TRACE runlevel. This is worthless (trace is often too noisy to be helpful) so making my own StdoutLogger. Tested by running this on a separate system, then copying the change over by hand. commit 92a7c6f8224ce9e69caab7c52b027b77e77dc1e5 Author: Damian Johnson Date: Tue Dec 11 19:38:10 2012 -0800 Default ignored when geoip was unavailable We ignored the default argument and always raised a ProtocolError if the geoip database was unavalable - oops. commit 872af06dc1a5770fb57e0af72e2412aa879f7e1f Author: Damian Johnson Date: Tue Dec 11 19:30:45 2012 -0800 Logging a warning when the geoip db is unavailable Issuing a warning when we've made enough failed geoip queries to conclude that the database is unavailable. This is partly for parity with arm which issues a warning in this situation (and that seems appropriate for other callers too). commit f4061ace5d87f2b0d4b47fb2a0aabbdcbd97b964 Merge: a8b20ad 6176abb Author: Damian Johnson Date: Mon Dec 10 07:47:52 2012 -0800 Adding a close_stream() to the Controller Feature being discussed on... https://trac.torproject.org/7687 commit 6176abb3f1c46192126a256aea6d6b27b4d66df9 Author: Sean Robinson Date: Sun Dec 9 18:38:47 2012 -0700 Minor revisions to the previous close_stream() revisions These tweaks change some pydoc verbiage and put NONE back in CircClosureReason while removing it from RelayEndReason. Signed-off-by: Sean Robinson commit 29a6d14fa054c05e4c3c6bb97b0a91fd2fe3569a Author: Damian Johnson Date: Sun Dec 9 14:41:53 2012 -0800 Minor revisions to close_stream() * Dropping the NONE RelayEndReason. It feels a little hacky to include it just so the index_of() will align with the integer values. Instead having close_stream() account for the offset. * The reason parameter of close_stream() is a stem.RelayEndReason rather than int so fixing its pydocs. * When doing formatted strings it's not necessary to wrap values in str(). If it's being assigned to a '%s' then that happens as part of the insertion. commit 33547a2b346b07f900d81be08f2dcd19451ea434 Author: Damian Johnson Date: Sun Dec 9 14:33:21 2012 -0800 Deduplicating RelayEndReason vs StreamClosureReason StreamClosureReason is a superset of RelayEndReason so there's no need to list both. Merging the enum descriptions and having StreamClosureReason refer to RelayEndReason. Also added RelayEndReason to __all__ so wildcard imports will pick it up. commit 786624951b627ae76370ab3afe38997552b80ae2 Author: Damian Johnson Date: Sun Dec 9 14:23:42 2012 -0800 Consolidating how StreamClosureReason is made commit 77481da93b23a54d339133ba379c0a6f3741f799 Author: Sean Robinson Date: Sat Dec 8 19:46:56 2012 -0700 Add a close_stream method to Controller Signed-off-by: Sean Robinson commit 545f251b698281b5cd6bd3f704242c9fdafaa36b Author: Sean Robinson Date: Sat Dec 8 19:24:28 2012 -0700 Build StreamClosureReason as a superset of RelayEndReason Explicitly make StreamClosureReason rely on RelayEndReason for its contents. Signed-off-by: Sean Robinson commit 69a2c8193fd613ce057b6d0adb466a21437ef5ff Author: Sean Robinson Date: Sat Dec 8 19:20:38 2012 -0700 Add enumeration of RELAY_END reasons This is for the reason to be given with the CLOSESTREAM command. NONE is a filler to make the remaining reasons start at offset one and will be rejected by the server, so it is not listed in the pydoc. Signed-off-by: Sean Robinson commit a8b20adb7044971935380b4170254c1c2d7f2051 Author: Sean Robinson Date: Sun Dec 9 08:37:54 2012 -0700 Improve tests for close_circuit exceptions Signed-off-by: Sean Robinson commit b3fe1659c0d2c7bf7c8ce282e0f858eb6350e80e Author: Sean Robinson Date: Sat Dec 8 19:57:13 2012 -0700 Raise different exceptions for different CLOSECIRCUIT errors Signed-off-by: Sean Robinson commit ec13ebb581fc7b57f055c59934f5f7f41c2111bb Merge: 4cacd2f 3be3a8d Author: Damian Johnson Date: Sun Dec 9 13:54:20 2012 -0800 Testing event version requirement checks Patches by robinson to both fix the mock_method() function and use it to test the tor version requirement checks for events. event version check: https://trac.torproject.org/7674 mock_method() fix: https://trac.torproject.org/7686 commit 3be3a8d94bb666b56bdc65e732f83c88ccfe0a15 Author: Damian Johnson Date: Sun Dec 9 13:52:20 2012 -0800 Minor revisions for the new add_event_listener tests Just some minor tweaks... * Replaced 'lambda x: x' with 'mocking.no_op()'. * Dropped the 'assertIsNone()' call. The purpose of this line is to make sure that add_event_listener() doesn't raise an exception, not to make an assertion on what the function returns. * Minor import tweaks to conform with 3da47d3. commit f5e2d127581a487f9cba7edf86943c0f78445180 Author: Sean Robinson Date: Sat Dec 8 18:29:54 2012 -0700 Add version requirement tests for Controller.add_event_listener Signed-off-by: Sean Robinson commit 22389a8ffd94251e857d3dd9d0abd40a01bc574e Author: Sean Robinson Date: Sat Dec 8 18:25:19 2012 -0700 Update mocking.mock_method pydocs Signed-off-by: Sean Robinson commit a00e2dda071bb2d6c04b6c78ece33942ba0def53 Author: Sean Robinson Date: Sat Dec 8 18:08:39 2012 -0700 Use getattr()/setattr() for mocking.mock_method mocking.revert_mocking also needs setattr because it may be trying to revert to the original method in an object. This fixes the following error when trying to use mock_method: TypeError: 'dictproxy' object does not support item assignment Signed-off-by: Sean Robinson commit 4cacd2f13d1cd003db0c15addcf265fa4262ee37 Author: Damian Johnson Date: Sun Dec 9 12:33:06 2012 -0800 Dropping '--' from config options being used to launch tor The '--' prefix in '--[config option]' commands are unnecessary. Thanks to lunar on #7666 for pointing this out. commit 120907822f06fd476f2c47b7135e816cd730b1c7 Author: Damian Johnson Date: Sat Dec 8 23:41:21 2012 -0800 Only using TAKEOWNERSHIP if we're on the local system We were using TAKEOWNERSHIP if our pid matched tor's __OwningControllerProcess. However, doing this doesn't make sense if we're connecting remotely. Caught by Robert on... https://trac.torproject.org/7666 commit aebe936d6e3f6a62d3d5f70944f06d599c207475 Author: Damian Johnson Date: Sat Dec 8 23:11:24 2012 -0800 Providing '__ReloadTorrcOnSIGHUP 0' when lauching tor with a custom config Lauching tor with the '__ReloadTorrcOnSIGHUP 0' option when we provide it with a custom configuration. This is to prevent the process from erring out when it gets a sighup due to not having a torrc. Suggestion by Robert on... https://trac.torproject.org/7682 commit 6f94c909d21b710fe86030cf3c1824102fc52613 Author: Damian Johnson Date: Sat Dec 8 23:04:30 2012 -0800 Dropping eronious regex from test A relay's LongName identifier can be missing the relay name. This was caught by Eoin and Robert on... https://trac.torproject.org/7558 commit 3da47d3b9d6d1ae5c6b2013a4247c45c13461a05 Author: Damian Johnson Date: Sat Dec 8 22:02:29 2012 -0800 Reordering import statements Dropping my oddball import ordering scheme in favor of the python standard. Now that we're getting more contributors making the library newcomer-friendly defintely trumps my weird notions of aesthetically pleasant imports. Taking this opportunity to also drop use of the 'as' import keyword. I'm still a little unsure what the most 'proper' style for imports is, but I've defintely seen 'from' used a lot more than 'as'. commit 9fddf70f2d1028dc6434ab866e010ca1fa53fb1d Author: Damian Johnson Date: Sat Dec 8 15:35:38 2012 -0800 Minor correction for the get_info() pydocs The prior comment about when a stem.ProtocolError was raised was mostly right, though it missed an important detail: our 'detection' for the geoip db being unavailabe is based on repeated failures so the that exception likely *won't* be raised for db unavailability. commit bd386445e7c6bf93348fde50503765fb13341971 Author: Sean Robinson Date: Sat Dec 8 15:31:57 2012 -0800 FIX: ProtocolError is no longer in stem.socket Signed-off-by: Sean Robinson commit 9fc23f64d3534a7e68bb0b66dddbd18b3e8db63c Author: Damian Johnson Date: Sat Dec 8 15:24:50 2012 -0800 Unexpected TypeError when get_conf() called for non-existant value Calling the Controller's get_conf() method would result in a TypeError if the configuration option didn't exist. For instance... ====================================================================== ERROR: test_reattaching_listeners ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/integ/control/controller.py", line 134, in test_reattaching_listeners controller.authenticate() File "/home/atagar/Desktop/stem/stem/control.py", line 954, in authenticate stem.connection.authenticate(self, *args, **kwargs) File "/home/atagar/Desktop/stem/stem/connection.py", line 358, in authenticate authenticate_none(controller, False) File "/home/atagar/Desktop/stem/stem/connection.py", line 435, in authenticate_none auth_response = _msg(controller, "AUTHENTICATE") File "/home/atagar/Desktop/stem/stem/connection.py", line 787, in _msg return controller.msg(message) File "/home/atagar/Desktop/stem/stem/control.py", line 308, in msg self._post_authentication() File "/home/atagar/Desktop/stem/stem/control.py", line 1511, in _post_authentication owning_pid = self.get_conf("__OwningControllerProcess", None) File "/home/atagar/Desktop/stem/stem/control.py", line 1009, in get_conf return _case_insensitive_lookup(entries, param, default) File "/home/atagar/Desktop/stem/stem/control.py", line 1644, in _case_insensitive_lookup for entry in entries: TypeError: 'NoneType' object is not iterable commit 40827b365e3a5f3dac6d1f8e8ffddd589be6cd70 Author: Damian Johnson Date: Sat Dec 8 15:10:08 2012 -0800 Checking event version prereqs before setting any Checking that our tor version meets all of the event's version prereqs before setting any. This has a few advantages... * It makes add_event_listener() an all-or-nothing operation (ie, failing one of the checks means that no listeners are attached, rather than a few) * Avoids repeated acquisitions of _event_listeners_lock (so other lock users can't slip in-between). * Lets me fix a bug where we were calling SETEVENTS for each event type, rather than a single call when we're done adding the listeners. commit 8506a43e04df43166f009c523cb63762d2243aeb Author: Sean Robinson Date: Sat Dec 8 15:04:18 2012 -0800 Add version requirements for event types Patch from Sean to add the version requirements for numerous event types... * AUTHDIR_NEWDESCS (added in tor commit b9d37a2d58e76ee2) * BUILDTIMEOUT_SET (version information taken from control-spec.txt) * CIRC_MINOR (added in tor commit 5519e633ec4c48c5) * CLIENTS_SEEN (added in tor commit 5519e633ec4c48c5) * CONF_CHANGED (added in tor commit 5a801a8c8b71c955) * DESCCHANGED (version information taken from control-spec.txt) * GUARD (added in tor commit 31e09bb5242b24d5) * NS (version information taken from control-spec.txt) * NEWCONSENSUS (version information taken from control-spec.txt) * SIGNAL (version information taken from control-spec.txt) * STATUS_* (added in tor commit 4401c47f638a60da) * STREAM_BW (added in tor commit 4a74812c70b75df8) commit 6da7588be3d0331d4e66a41901510cc680aa7cba Author: Sean Robinson Date: Fri Dec 7 05:39:42 2012 -0700 Note that some events are original to control-spec V1 Do not explicitly assign a minimum version to some events. Instead, use the Event class version, but add pydocs about when the event was added to Tor. While these events were all added in tor version 0.1.1.1-alpha, not all were in the same commit. Tor commit 9b4aa8d2abbce713 adds V1 BW, CIRC, NEWDESC, ORCONN, and STREAM. Tor commit c2f6fe9b85255185 adds V1 logging (i.e. DEBUG, INFO, etc.). Tor commit 64fc1f7bae174d62 adds ADDRMAP. Signed-off-by: Sean Robinson commit 86524bf589014f2f6d7a4a9704cc0801663654b2 Author: Sean Robinson Date: Fri Dec 7 04:03:19 2012 -0700 Begin version requirements for events Check that the tor process is of a minimum version able to generate the events requested. Checking versions here improves on the 'Unrecognized event "XXX"' response from the tor process. The SETEVENT control message was added to tor version 0.0.9-pre5 (commit 347d3f9d629df74b). But control protocol V1 changed the message syntax in tor version 0.1.1.1-alpha (commit 9b4aa8d2abbce713) and V0 is not supported. Therefor, version 0.1.1.1-alpha is the baseline for any and all events in Stem. Signed-off-by: Sean Robinson commit e2c69117b388abbbeee9240e67365e893b137cd3 Author: Sean Robinson Date: Tue Dec 4 18:38:27 2012 -0700 Fix various pydoc typos Corrections from Sean for our is_alive methods, get_info, and other methods. commit 7fba50ff561578f746ceb3383b1bb16d709a14e3 Author: Damian Johnson Date: Sat Dec 8 14:13:43 2012 -0800 Supporting TAKEOWNERSHIP when spawing a tor process Adding a 'take_ownership' option to the stem.process module's launch_tor*() functions. When it's set we'll do the following... * Assert ownership over the tor process that we spawn by setting its __OwningControllerProcess to be our pid. * If a Controller connects to it then we replace our ownership via the pid with ownership via the socket connection (issuing a TAKEOWNERSHIP request and dropping __OwningControllerProcess). Thanks to lunar for the initial TAKEOWNERSHIP patch on #7666, and rransom for his advice on #7667. commit 885a294646703a537c37cd2a5ac9aa8728561744 Author: Damian Johnson Date: Sat Dec 8 12:35:34 2012 -0800 Re-attaching event listeners When a controller disconnects then reconnects we want our event listeners to resume getting events. There's a couple things about this commit that I really don't like... * It includes a hack in our msg() method to add the hook. * Our tests include sleep() calls. This sucks since it makes our tests take considerably longer, and can break on systems under heavy load. commit 3eb1da90792de75dc55492e37c3785dd0db28bc7 Author: Damian Johnson Date: Fri Dec 7 18:31:01 2012 -0800 Reverting 'Tidying up event testing boilerplate' This reverts commit fb0aec5d95e9d2e6d64b7ab7c51018f3581565c5. commit 700b54c10c4f5cf37f250189fd76ffa6bd24cca6 Author: Damian Johnson Date: Fri Dec 7 17:30:53 2012 -0800 Prompt notice when tor is starting When the prompt script is first ran it can take a while for tor to start. Giving a notice that tor is starting. This also changes the Controller instance variable from 'control' to 'controller'. I'm trying to standardize on the later to avoid a naming confict with stem.control imports. commit 0d01ad60bb2a4ea2f0ac1f9626360d5db22858ea Author: Damian Johnson Date: Fri Dec 7 09:13:46 2012 -0800 Support for bridge-ip-versions extrainfo desc lines Adding parsing support for 'bridge-ip-versions' lines in extra-info descriptors. These are fields only relevant to bridges (though the spec doesn't prohibit them from appearing elsewhere). They're pretty much the same thing as the IPVersions in CLIENTS_SEEN events, so borrowing its parsing. Spec addition: https://gitweb.torproject.org/torspec.git/commitdiff/9a518d9 Metrics-lib addition: https://gitweb.torproject.org/metrics-lib.git/commitdiff/17e9149 commit bd6e5cea6525f76b95966b023a1c1fa88065adc3 Author: Damian Johnson Date: Fri Dec 7 08:48:12 2012 -0800 Support for geoip6-db-digest extrainfo desc lines Adding parsing support for the newish 'geoip6-db-digest' lines in extra-info descriptors. Lacking this support has been breaking our integ tests for a while, so I was able to exercise this both with the unit test addition and an actual consensus. Bug: https://trac.torproject.org/7354 Spec addition: https://gitweb.torproject.org/torspec.git/commitdiff/9a518d9 Metrics-lib addition: https://gitweb.torproject.org/metrics-lib.git/commitdiff/43b9390 commit 9c0cc18909a1dd16cc810cf3178acfbcb0bc064f Author: Damian Johnson Date: Fri Dec 7 07:22:43 2012 -0800 Avoiding address type query for cached get_mask() requests Oops. There's no reason to call get_address_type() when we already have a lazily loaded _mask attribute. commit 45403096a550186cfbecc035cd54c125e3b0237c Author: Damian Johnson Date: Thu Dec 6 22:44:17 2012 -0800 Reducing ExitPolicyRule's memory requirements Frequently there's several ExitPolicyRule entries per an exit policy, so when we pull the consensus we get quite a few instances of this class. Making the following changes to reduce the memory requirements... * Dropping the rule attribute. The string representation of the rule should be good enough, if not better for callers. * Replacing the address_type attribute with a method for getting it. This lets us store the address type as an integer within our class. * Replacing the mask attribute with a method for getting it. The ip mask representation is very rarely useful, so there's little reason to store it unless it's requested. * Lazily loading the integer reprentation of our address and mask, both speeding up our constructor and avoiding it entirely if our caller never uses is_match() This lowers the memory requirement for loading the full consensus on my netbook from 5.5% to 5.1% (a 7% drop). This is a drop in the bucket compared to the prior commit, but between the faster constructor runtime and squeezing out a little more performance it's still worth it. commit bbd702c81e1923bfbab236d0d3649fd2f87b3d95 Author: Damian Johnson Date: Thu Dec 6 22:05:07 2012 -0800 Vastly lowering memory usage for exit policies A pox upon good indentions. When dealing with microdescriptor exit policies I enumerated all ports involved, the goal being to provide constant time is_match() lookups. We certainly did that, but there's quite a few valid ports and enumerating them for all relays has consistantly cost so much memory on my system that it triggers the OOM killer. Dropping the port set optimization. Memory usage trumps is_match() performance. Reading the full consensus into memory now takes 5.5% of the memory on my netbook, verses before where it would locked me up. commit 51141d37b9a26a8a05cfa09cf85f3939ab79011a Author: Damian Johnson Date: Thu Dec 6 09:02:35 2012 -0800 Skip pointless extra read of network status documents When parsing a network status document we read to the end of the router status entries solely to determine where the end of the file is. This is a waste of time and memory. Instead, checking for the end of the section as we go along. commit a71163021f1731302c7f922ce4e5b3191d2fb938 Author: Ravi Chandra Padmala Date: Thu Dec 6 20:05:18 2012 +0530 Improve Controller.extend_circuit documentation Realized it might not be that obvious how paths are specified, and added some sample code to the Controller.extend_circuit documentation. Fixed a typo/grammer. commit 7882d3cfeec98a478046a779e531a6013558c348 Author: Ravi Chandra Padmala Date: Thu Dec 6 19:39:16 2012 +0530 Controller.extend_circuit now checks for failures properly commit 69f72efc9367092c989819bb2e408598ecc9dcdb Author: Damian Johnson Date: Tue Dec 4 08:37:59 2012 -0800 Checking for None by identity According to PEP 290 comparisons to None should be done via identity (the 'is' keyword): http://www.python.org/dev/peps/pep-0290/#testing-for-none I had already been doing this, albeit inconsistently. Inequality checks were still using '!=' - swapping all None comparisons to be by identity. commit d50b8a06744a8476bc16c04268fef75b20712b2b Author: Damian Johnson Date: Tue Dec 4 08:21:09 2012 -0800 Purging use of the variable 'i' A pox upon whoever taught me to use variables like i, j, and k for loop constants. It's ungrepable and a practice that should be purged via fire and pitchforks. I'm keeping it in for list interpolation (since it's localized to a single line), but replacing it with 'index' everywhere else. I'm also replacing a few points that do... for i in range(len(my_list)): my_item = my_list[i] ... with calls to enumerate(). commit 7011a6a2f1c306e204eb6e518604911d5561fc71 Author: Damian Johnson Date: Tue Dec 4 08:00:58 2012 -0800 Including all entries in the Event's positional_args Fixing a semi-bug where the Event's positional_args only contained entries that we didn't recognize. This was a bug according to the Event's pydoc, but by design according to an inline comment. I'm not really sure what I did that - oh well... commit 115607af56f7e8497c9c3343a4f7eae850b7a8a9 Author: Damian Johnson Date: Tue Dec 4 07:53:23 2012 -0800 Comments for private Event attributes Adding comments to describe arguments like _POSITIONAL_ARGS and _SKIP_PARSING. Also dropping _QUOTED entries for key/value entries since it's no longer being used for that purpose. commit fb0aec5d95e9d2e6d64b7ab7c51018f3581565c5 Author: Damian Johnson Date: Mon Dec 3 23:20:24 2012 -0800 Tidying up event testing boilerplate Most event unit tests initially check two things... * That the event class matches an expected value. * That the string content for the event matches what we were constructed with. Blending these two checks with the function that provides us with events. commit ae1f59b6b0ec1550434bc1508c7cd9ce866fa616 Author: Damian Johnson Date: Mon Dec 3 21:04:10 2012 -0800 Better handling quoted key/value mappings in events We were being pretty picky about key/value mappings with quoted values, only accepting them if they were in both our _KEYWORD_ARGS and _QUOTED. The trouble with this was... 1. It make our parsing code more convoluted than it needed to be. 2. If tor added a new quoted key/value mapping then it would likely break our parser. This is because we'd mis-interpret the value as being positional fields. For instance if... 650 MY_EVENT NEW_USER username=atagar ... became... 650 MY_EVENT NEW_USER username=atagar message="hello world" Then we'd misinterpret the event as having no keyword arguments, and rather have four positional arguments (the last being 'world"'). This isn't strictly wrong according to the control-spec. It's good about specifiying how new positional and keyword arguments should be handled, but quiet about quoted values. However, accepting quoted mappings by default is more intuitive and less likely to cause sadness down the road so making it so. commit 22b4ced8548202a5715119a1d331900e6b3e8fff Author: Damian Johnson Date: Mon Dec 3 20:41:53 2012 -0800 Simplify check for if event attributes are unrecognized Whenever event parsing handles an attribute for which we have an enum it checks to see if the value is recognized by stem. If not then it logs at the INFO level. Replacing these boilerplate checks with a helper function. I'm also adding a test to check that we're actually logging these messages. commit fb03666ebe9fa1ec134e6260bb79b6ed363e9994 Author: Damian Johnson Date: Mon Dec 3 19:22:55 2012 -0800 Revisions for STREAM_BW support Couple very minor tweaks to the prior commits: * Adding STREAM_BW to the Event enum table. * Chaning is_valid_stream_id() to be a function rather than a module alias. This is effectively the same thing, but is less likely to confuse sphinx and imho is a bit more intuitive. commit b3d91930be616ad7b7de10e3bbe581637583202c Author: Sean Robinson Date: Sat Dec 1 13:50:48 2012 -0700 Add unit test for STREAM_BW event handler Signed-off-by: Sean Robinson commit ac4eac1f15a3e8fce01bfe7e3f92aeff077894a5 Author: Sean Robinson Date: Sat Dec 1 13:28:04 2012 -0700 Add STREAM_BW event handler Signed-off-by: Sean Robinson commit 959e1506892c5f11e24a9a4fdfb3eafd73b4bd56 Author: Sean Robinson Date: Sat Dec 1 13:27:35 2012 -0700 Add is_valid_stream_id helper function This is an alias to is_valid_circuit_id since stream and circuit identifiers follow the same pattern. Signed-off-by: Sean Robinson commit b3a1cbc538dced7c97da7cd3cd48f42937ed3565 Author: Sean Robinson Date: Sat Dec 1 13:41:10 2012 -0700 Add unit tests for is_valid_circuit_id function Signed-off-by: Sean Robinson commit 3ba521dfa2f2a6f023e7b92fa24bcfb910f1fe78 Author: Damian Johnson Date: Mon Dec 3 09:00:49 2012 -0800 Support for CIRC_MINOR events Implementation and testing for CIRC_MINOR events. Snagged test data by listening for events while bootstrapping. commit bb458df4ac47c1405ce7e7f7e3e15ececf485b4b Author: Damian Johnson Date: Mon Dec 3 08:21:50 2012 -0800 Support for CONF_CHANGED events Implementation and testing for CONF_CHANGED events. Got the test data by issuing a SETCONF request... SETEVENTS CONF_CHANGED 250 OK SETCONF ExitNodes=caerSidi MaxCircuitDirtiness=20 ExitPolicy 650-CONF_CHANGED 650-ExitNodes=caerSidi 650-ExitPolicy 650-MaxCircuitDirtiness=20 650 OK 250 OK commit 47f608b05c0085a5acfb76b08a9efe916b1e09e9 Author: Damian Johnson Date: Sun Dec 2 22:31:22 2012 -0800 Support for SIGNAL events Implementation and testing for SIGNAL events. This moves tor's signals to an enum, which also makes the Controller's signal() method a little nicer. commit 0687e99789827aa43dd8523006eb49aed837c2ef Author: Damian Johnson Date: Sun Dec 2 20:20:33 2012 -0800 Support for BUILDTIMEOUT_SET events Impementation and testing for BUILDTIMEOUT_SET events. Snagged test data by listening for the events while bootstrapping... 650 BUILDTIMEOUT_SET COMPUTED TOTAL_TIMES=124 TIMEOUT_MS=9019 XM=1375 ALPHA=0.855662 CUTOFF_QUANTILE=0.800000 TIMEOUT_RATE=0.137097 CLOSE_MS=21850 CLOSE_RATE=0.072581 650 BUILDTIMEOUT_SET COMPUTED TOTAL_TIMES=125 TIMEOUT_MS=8915 XM=1375 ALPHA=0.860984 CUTOFF_QUANTILE=0.800000 TIMEOUT_RATE=0.136000 CLOSE_MS=21850 CLOSE_RATE=0.072000 650 BUILDTIMEOUT_SET COMPUTED TOTAL_TIMES=126 TIMEOUT_MS=8923 XM=1375 ALPHA=0.860563 CUTOFF_QUANTILE=0.800000 TIMEOUT_RATE=0.134921 CLOSE_MS=21850 CLOSE_RATE=0.071429 650 BUILDTIMEOUT_SET COMPUTED TOTAL_TIMES=127 TIMEOUT_MS=8814 XM=1375 ALPHA=0.866231 CUTOFF_QUANTILE=0.800000 TIMEOUT_RATE=0.133858 CLOSE_MS=21850 CLOSE_RATE=0.070866 commit 42872dd08e81d6b341654ab85969df2ed77a7397 Merge: e1772bb c1e88e4 Author: Damian Johnson Date: Sun Dec 2 18:30:14 2012 -0800 Tor event handling Merging support for tor's asynchronous events. This isn't entirely complete, missing... * STREAM_BW (being added in https://trac.torproject.org/7616) * BUILDTIMEOUT_SET * SIGNAL * CONF_CHANGED * CIRC_MINOR ... but better not to let the perfect be the enemy of good. This branch was partly code reviewed by robinson on... https://trac.torproject.org/7328 commit c1e88e4257c606593ecd2abf4212cc5266b34bf4 Author: Damian Johnson Date: Sun Dec 2 18:24:54 2012 -0800 Support for NEWCONSENSUS events Implementation and testing for NEWCONSENSUS events. This is pretty much the exact same thing from our perspective as NS events. commit 92419a54b83d80d7b124482df9bdf42fb931c7a0 Author: Damian Johnson Date: Sun Dec 2 17:23:02 2012 -0800 Support for CLIENTS_SEEN events Implementation and testing for CLINETS_SEEN events. This would be a difficult event to get examples for, but luckily the spec has one. commit 92af0a5c0f9c6924cfddfb6388f581f3b6014732 Author: Damian Johnson Date: Sun Dec 2 16:16:20 2012 -0800 Support for NS events Implementation and testing for NS events. To get test data I wiped my cached_consensus, then listened for NS events while bootstraping. The first time I did this it provided a flood of entries (probably the whole consensus), but then when I restarted tor and tried again it just proved one. commit 2585969d6efbd8b1a6660d5de1ede437792dc22b Author: Damian Johnson Date: Sun Dec 2 14:02:21 2012 -0800 Support for GUARD events Implementation and testing for GUARD events. The spec is a little vauge, so we're missing enum descriptions and a good understanding of the 'Name' attribute... https://trac.torproject.org/7619 Got the test data by listening for GUARD events while bootstrapping... 650 GUARD ENTRY $5D0034A368E0ABAF663D21847E1C9B6CFA09752A BAD 650 GUARD ENTRY $071C2D9D0E03D75FF30DCCA850BB29CA4A4BB383 BAD 650 GUARD ENTRY $36B5DBA788246E8369DBAF58577C6BC044A9A374 NEW 650 GUARD ENTRY $071C2D9D0E03D75FF30DCCA850BB29CA4A4BB383 GOOD 650 GUARD ENTRY $5D0034A368E0ABAF663D21847E1C9B6CFA09752A GOOD commit 4959378f9aa83d96bc1431d72bea6fa7a1b6e2c7 Author: Damian Johnson Date: Sun Dec 2 11:51:34 2012 -0800 Allowing zero Terget port in STREAM events The STREAM event's target port can be zero in DNS_REQUEST requests. Caught thanks to robinson on... https://trac.torproject.org/7598 commit e59de80010d1894c5eecb7d4b9d953ee195a5259 Author: Damian Johnson Date: Sun Dec 2 03:00:51 2012 -0800 Testing for STATUS_* events I had grand plans to have a StatusEvent subclass for each of the causes. However, on reflection that would encompass a bundle of work that's almost as large as all of the other events put together. It might still be a neat thing to do someday, but there's plenty of higher priority things to do first. Adding tests for all of the event examples that I collected earlier. This includes a bit of a hack to make quoted key/value mappings work. Previously we've only parsed quoted mappings if they belonged to the _QUOTED listing, but STATUS_* events sprinkle them all around so for forward compatability we can't simply enumerate them. commit aeebbe762579951b4a78ddc8c68347d2d181fc49 Author: Damian Johnson Date: Sun Dec 2 02:05:09 2012 -0800 Handling typo in STATUS_SERVER event name Prior to tor 0.2.0.22 STATUS_SERVER events claimed to be 'STATUS_SEVER'. Accepting both names as per the spec's suggestion. commit 1af965f1d12f5a58e921559901a2b7f0ca1e4cba Author: Damian Johnson Date: Fri Nov 30 23:04:07 2012 -0800 Support for STREAM events with a DIR_FETCH purpose The SOURCE_ADDR field of STREAM events can have zero as their port. Fixing this and adding a test. This was caught thanks to robinson on... https://trac.torproject.org/7598 commit 628dd1f3246566bf4d7b4b2e2932f459bec32986 Author: Damian Johnson Date: Thu Nov 29 18:49:50 2012 -0800 Initial support for STATUS_* events The STATUS_GENERAL, STATUS_CLIENT, and STATUS_SERVER events all follow a nice, well specified format. The specific action instances for these events warrant subclasses, but to start with adding the StatusEvent class which covers the basic attributes. commit f7cc47eae4e733872ba11687a81fc7cddb22fb1e Author: Damian Johnson Date: Tue Nov 27 08:48:50 2012 -0800 Filling in AuthDescriptorAction descriptions The AUTHDIR_NEWDESCS spec was clarified in 'https://trac.torproject.org/7533'. I thought that I'd fixed all the TODO notes, but guess I missed this one. commit 0f4e52bf20ae82a8951ed910639f0e4fa41aeac8 Author: Damian Johnson Date: Fri Nov 23 19:54:58 2012 -0800 Parsing AUTHDIR_NEWDESCS events The specification for the AUTHDIR_NEWDESCS event type was expanded in... https://gitweb.torproject.org/torspec.git/commitdiff/a1331ca We still don't have an example of the event so no tests, but it's enough to implement the parsing. It would be nice if the event type included a field for the descriptor type, but oh well. commit af527bdc1b87586c201ffd7fa288f40e29e33598 Author: Damian Johnson Date: Fri Nov 23 17:01:52 2012 -0800 Dropped StreamPurpose.UPLOAD_DESC Dropping UPLOAD_DESC, which was evidently never used... https://gitweb.torproject.org/torspec.git/commitdiff/36d7d05 commit df9a051783a94b831aa1c93c8592d6c46397a229 Author: Damian Johnson Date: Fri Nov 23 12:36:28 2012 -0800 Adding descriptions for ORStatus and ORClosureReason Spec has been expanded with descriptions for the ORCONN fields. I'm still not sure what these events mean nor what many of these values are actually trying to indicate, but it's enough for me to parrot the descriptions. bug report: https://trac.torproject.org/7513 fix: https://gitweb.torproject.org/torspec.git/commitdiff/036e256 commit 404025ec9a7f0614742ce35ae13a1a001e4d9f9e Author: Damian Johnson Date: Fri Nov 23 11:43:13 2012 -0800 Documenting the StreamPurpose values Description for the STREAM event's PURPOSE field were added to the spec to reflecting that here. bug report: https://trac.torproject.org/7508 fix: https://gitweb.torproject.org/torspec.git/commitdiff/e23a24a commit 4ce0be17442a4c21612eefb6cbe426aebbfc722f Author: Damian Johnson Date: Fri Nov 23 11:31:14 2012 -0800 Adding CircClosureReason.MEASUREMENT_EXPIRED description Spec was updated with a description so reflecting that here. bug report: https://trac.torproject.org/7506 fix: https://gitweb.torproject.org/torspec.git/commitdiff/de62160 commit 92ab865999a3e6a4ed21311560e4b15c22ad4139 Author: Damian Johnson Date: Fri Nov 23 11:02:44 2012 -0800 Replacing GMT refrences with UTC Accouning for the spec change... https://gitweb.torproject.org/torspec.git/commitdiff/d581e63 commit 7f73e1c2c92bd949c29b1430200778d46be4d0cc Author: Damian Johnson Date: Wed Nov 21 09:05:40 2012 -0800 Supporing DESCCHANGED events Adding support for DESCCHANGED events. These are trivial since... well, they don't contain any attributes. I'm also filling in some missing EventType enum documentation and changing the API docs for events to automatically pick up new event types (there's no point in using autoclass rather than automodule in this case). commit 84ec250529b81cb86899897784602e6c41fb8d98 Author: Damian Johnson Date: Tue Nov 20 09:37:54 2012 -0800 Stub for AUTHDIR_NEWDESCS events Implementing the parts that I can for AUTHDIR_NEWDESCS events. The spec doesn't provide enough detail for us to parse them (https://trac.torproject.org/7533) and I need an example of an event before I can test them (https://trac.torproject.org/7534). commit ab6e7a365cfa1e31b269f1df0e722ea11a2c53b1 Author: Damian Johnson Date: Mon Nov 19 00:09:09 2012 -0800 Support for ADDRMAP events There's a special spot in hell for whoever decided to allow for quoted values in events. This implements and adds testing for ADDRMAP events. Unlike TorCtl we aren't falling back on a regex for the... er, 'wonderful' quoted stuff, but rather including quoted value support in the Event parser. Got test data by visiting a few sites in TBB... 650 ADDRMAP check.torproject.org 38.229.72.22 "2012-11-18 22:48:34" EXPIRES="2012-11-19 06:48:34" 650 ADDRMAP ocsp.digicert.com 5.63.145.124 "2012-11-18 21:53:42" EXPIRES="2012-11-19 05:53:42" 650 ADDRMAP www.atagar.com 75.119.206.243 "2012-11-19 00:50:13" EXPIRES="2012-11-19 08:50:13" commit 54f81be931196e92870b09119fa7e02a7a62cdbf Author: Damian Johnson Date: Sun Nov 18 21:06:29 2012 -0800 Support for NEWDESC events Implementing and testing NEWDESC events. Got test data by listning for events right after starting a TBB instance... 650 NEWDESC $9C2007BE0D3785D5BAB204F0CFE07CF29F4B75EB~Unnamed 650 NEWDESC $A10C4F666D27364036B562823E5830BC448E046A=DFRI1 650 NEWDESC $A1130635A0CDA6F60C276FBF6994EFBD4ECADAB1~tama 650 NEWDESC $A4E3D4A42E68D8302E4A3EA5D117EDCF0B157F72=tor1rufus 650 NEWDESC $B3BE230EEA146002C4A2A436B4EF7D897DE6C9B6~WombleNode01 650 NEWDESC $B3FA3110CC6F42443F039220C134CBD2FC4F0493=Sakura 650 NEWDESC $B4BE08B22D4D2923EDC3970FD1B93D0448C6D8FF~Unnamed 650 NEWDESC $B839D6CCC5438A21E547DB62043CB0192C068F51=brajah1 650 NEWDESC $B8B29766ACFADF832ABD1BCAA86497A20F54B584=stillhavenoipinfo 650 NEWDESC $BBE409F5791DAA52C2C3C9117CBA5AA55F3E2E88=Rarity 650 NEWDESC $BE938957B2CA5F804B3AFC2C1EE6673170CDBBF8=Moonshine 650 NEWDESC $C863FB2A6109C9CE2993C8855BAC59583B15475B=coinet 650 NEWDESC $CA37697301883DE9E895F642D76F3CD31E855434=Bitcoin 650 NEWDESC $CBC5F626F16052B2B987106D4845DF0B5197901D=mergeslittletornode 650 NEWDESC $CC19411EDD0BE139E8828EECE2F11E5DFE5F5C09~RadiantPickle 650 NEWDESC $CF334A712B78A84C6E8F2FF71987122E17951176=skye ... commit 62b6f153e66a3b1a8adff3ac837c161e43576f2d Author: Damian Johnson Date: Sun Nov 18 14:47:27 2012 -0800 Reordering event tests We're using an alphabetical order elsewhere for the events (except logging events which are batched together at the top), so matching that in the unit tests. commit a1a5784d480421af4394b3e012d6f4bbb8ee6c8b Author: Damian Johnson Date: Sun Nov 18 14:44:05 2012 -0800 Support for ORCONN events Implementation and tests for ORCONN events. These have several holes in its documentation (https://trac.torproject.org/7513) so I'm not really sure what these events actually are. Reguardless, got some samples by connecting to TBB and issuing a NEWNYM. AUTHENTICATE 250 OK SETEVENTS ORCONN 250 OK 650 ORCONN $1D024F41EDBF3F061E1341D516543090D8A44B42=AccessNowKromyon21 CONNECTED 650 ORCONN $7ED90E2833EE38A75795BA9237B0A4560E51E1A0=GreenDragon CONNECTED 650 ORCONN $A1130635A0CDA6F60C276FBF6994EFBD4ECADAB1~tama CLOSED REASON=DONE commit dff8a0edf35a6e52f2d0e1d52fee71f891146c23 Author: Damian Johnson Date: Sun Nov 18 13:03:46 2012 -0800 Alphabetizing event types I had been following an order similar to the control spec but on reflection that doesn't make sense. Ordering these alphabetically would be better. commit d1cd96db70da3a737f6fee24354d61f5e52893e5 Author: Damian Johnson Date: Sun Nov 18 13:00:50 2012 -0800 Few spelling corrections commit c96c76ce16d42aeef6641bfeaaf937673934dc5c Author: Damian Johnson Date: Sun Nov 18 12:55:52 2012 -0800 Moving attribute enums to top level module Enumerations for our event attributes don't really belong in the control module. They're a bit more like our exceptions in that they're a data type that might be used in multiple spots. Hence moving the enums to live with them. commit d735e1e267877d9e9e7fa0686c2cd49c63930f0b Author: Damian Johnson Date: Sun Nov 18 11:48:00 2012 -0800 Support for STREAM events Implementaton and tests for STREAM events. I got the test data by... * starting TBB * used netstat to get the control port (shouldn't have needed to do this - https://trac.torproject.org/7512) * connecting to it with telnet * AUTHENTICATE * SETEVENTS STREAM * visited google's front page in firefox Full test data: AUTHENTICATE 250 OK SETEVENTS STREAM 250 OK 650 STREAM 18 NEW 0 encrypted.google.com:443 SOURCE_ADDR=127.0.0.1:47849 PURPOSE=USER 650 STREAM 18 SENTCONNECT 26 encrypted.google.com:443 650 STREAM 19 NEW 0 encrypted.google.com:443 SOURCE_ADDR=127.0.0.1:47850 PURPOSE=USER 650 STREAM 19 SENTCONNECT 26 encrypted.google.com:443 650 STREAM 18 REMAP 26 74.125.227.129:443 SOURCE=EXIT 650 STREAM 18 SUCCEEDED 26 74.125.227.129:443 650 STREAM 19 REMAP 26 74.125.227.129:443 SOURCE=EXIT 650 STREAM 19 SUCCEEDED 26 74.125.227.129:443 650 STREAM 20 NEW 0 encrypted.google.com:443 SOURCE_ADDR=127.0.0.1:47851 PURPOSE=USER 650 STREAM 20 REMAP 0 74.125.227.129:443 SOURCE=CACHE 650 STREAM 20 SENTCONNECT 26 74.125.227.129:443 650 STREAM 21 NEW 0 encrypted.google.com:443 SOURCE_ADDR=127.0.0.1:47852 PURPOSE=USER 650 STREAM 21 REMAP 0 74.125.227.129:443 SOURCE=CACHE 650 STREAM 21 SENTCONNECT 26 74.125.227.129:443 650 STREAM 20 REMAP 26 74.125.227.129:443 SOURCE=EXIT 650 STREAM 20 SUCCEEDED 26 74.125.227.129:443 650 STREAM 21 REMAP 26 74.125.227.129:443 SOURCE=EXIT 650 STREAM 21 SUCCEEDED 26 74.125.227.129:443 650 STREAM 22 NEW 0 www.google.com:443 SOURCE_ADDR=127.0.0.1:47853 PURPOSE=USER 650 STREAM 22 SENTCONNECT 26 www.google.com:443 650 STREAM 23 NEW 0 www.google.com:443 SOURCE_ADDR=127.0.0.1:47854 PURPOSE=USER 650 STREAM 23 SENTCONNECT 26 www.google.com:443 650 STREAM 21 CLOSED 26 74.125.227.129:443 REASON=CONNRESET 650 STREAM 20 CLOSED 26 74.125.227.129:443 REASON=CONNRESET 650 STREAM 22 REMAP 26 74.125.227.147:443 SOURCE=EXIT 650 STREAM 22 SUCCEEDED 26 74.125.227.147:443 650 STREAM 23 REMAP 26 74.125.227.147:443 SOURCE=EXIT 650 STREAM 23 SUCCEEDED 26 74.125.227.147:443 650 STREAM 24 NEW 0 ocsp.thawte.com:80 SOURCE_ADDR=127.0.0.1:47855 PURPOSE=USER 650 STREAM 24 SENTCONNECT 26 ocsp.thawte.com:80 650 STREAM 25 NEW 0 ocsp.thawte.com:80 SOURCE_ADDR=127.0.0.1:47856 PURPOSE=USER 650 STREAM 25 SENTCONNECT 26 ocsp.thawte.com:80 650 STREAM 24 REMAP 26 199.7.52.72:80 SOURCE=EXIT 650 STREAM 24 SUCCEEDED 26 199.7.52.72:80 650 STREAM 25 REMAP 26 199.7.52.72:80 SOURCE=EXIT 650 STREAM 25 SUCCEEDED 26 199.7.52.72:80 650 STREAM 26 NEW 0 ssl.gstatic.com:443 SOURCE_ADDR=127.0.0.1:47857 PURPOSE=USER 650 STREAM 26 SENTCONNECT 26 ssl.gstatic.com:443 650 STREAM 27 NEW 0 ssl.gstatic.com:443 SOURCE_ADDR=127.0.0.1:47858 PURPOSE=USER 650 STREAM 27 SENTCONNECT 26 ssl.gstatic.com:443 650 STREAM 23 CLOSED 26 74.125.227.147:443 REASON=CONNRESET 650 STREAM 26 REMAP 26 74.125.227.143:443 SOURCE=EXIT 650 STREAM 26 SUCCEEDED 26 74.125.227.143:443 650 STREAM 27 REMAP 26 74.125.227.143:443 SOURCE=EXIT 650 STREAM 27 SUCCEEDED 26 74.125.227.143:443 650 STREAM 25 CLOSED 26 199.7.52.72:80 REASON=DONE 650 STREAM 27 CLOSED 26 74.125.227.143:443 REASON=CONNRESET 650 STREAM 26 CLOSED 26 74.125.227.143:443 REASON=DONE 650 STREAM 24 CLOSED 26 199.7.52.72:80 REASON=DONE 650 STREAM 22 CLOSED 26 74.125.227.147:443 REASON=DONE 650 STREAM 19 CLOSED 26 74.125.227.129:443 REASON=DONE 650 STREAM 18 CLOSED 26 74.125.227.129:443 REASON=DONE Connection closed by foreign host. commit a349a01fece5534195a6619d72df4944065001b8 Author: Damian Johnson Date: Sat Nov 17 18:49:23 2012 -0800 Support for CIRC events Implementation and testing for CIRC events. This work also concerns the 'GETINFO circuit-status' method, which is defined as providing the same output as CIRC events. This is part of the reason why I put the enums for the event attributes in 'stem.control'. commit d7dc386533f0b7885f2966922b9cb051c1b89384 Author: Damian Johnson Date: Sun Nov 11 12:36:37 2012 -0800 Enum docs for stem.control.EventType I had been doing a table for the EventType enum under the add_event_listener() method. This made sense and matched with what I'm doing for enumeration documentation except that this should be a interlinkable :data: field in our header. commit 119579bd8c53c15e949e9a77a0c30e12be07d1cd Author: Damian Johnson Date: Sun Nov 11 12:28:51 2012 -0800 Enum docs for stem.control.State While documenting enums I must have messed up by search term for stem.control since, rather than adding enum documentation, I concluded that there weren't any enums and removed the import statement. Fixing this. commit 7ba579494215a5353cff1db9fbf7e59d6b666af9 Author: Damian Johnson Date: Mon Nov 5 09:14:07 2012 -0800 Support for logging events Adding an event class for DEBUG, INFO, NOTICE, WARN, and ERR. commit 1d2d684e3975e0e55369485b3d75155e90ea6aef Author: Damian Johnson Date: Mon Nov 5 08:36:05 2012 -0800 BW event unit tests Adding a few simple unit tests for the BandwidthEvent class. commit 1d56eb59e256977d34f9d3d77e4e0b8c2df49f86 Author: Damian Johnson Date: Mon Nov 5 08:21:30 2012 -0800 EventType to Event subclass table Adding a table that'll list the EventType enumerations and map them to Event subclasses. commit 1a6a4d55701aaa199c14e1ef1ee1b6455922d761 Author: Damian Johnson Date: Mon Nov 5 08:02:15 2012 -0800 Unit test for add_event_listener() example Adding a unit test for the pydoc example in add_event_listener(). I really, really don't like how the tests for these documentation examples are turning out. The end result looks precious little like the example and is convoluted as hell. We should revisit how we do documentation example testing in the future, but that'll be a subproject in itself. commit cc530bb2005f0e156c62fe322d5b43ffe16c3e99 Author: Damian Johnson Date: Mon Nov 5 07:20:49 2012 -0800 Adding a add_event_listener() example On reflection the example that I included on ticket #7328 is pretty good at exemplifying the function's usage, so including it in our pydocs. commit c35a626af19a5d22b8cc1ca1cef5bae212e3a726 Author: Damian Johnson Date: Sun Nov 4 21:06:05 2012 -0800 BandwidthEvent class Adding a class to handle BW events. This is the easiest type of tor event to test because it's emitted every second. The other event types we'll probably need to go with unit tests. commit cac0c434ca9a4ed1f9bb6aa940dba1e006b92471 Author: Damian Johnson Date: Sun Nov 4 18:38:28 2012 -0800 Base class for events Adding a base class for tor events which does the initial parsing of positional and keyword arguments. This is using a similar design to the TorCtl patch I wrote for #3679 (but a little better). commit ba280f94da9962e1a2b1bfaf22d5fbd17ffa3103 Author: Damian Johnson Date: Sun Nov 4 15:53:44 2012 -0800 Adding event handling support to the Controller My initial plan was to add event listener support similar to TorCtl but with automated handling of the SETEVENT calls. While working on this meejah suggested accepting functors instead (like txtorcon). On reflection this is both much nicer for our callers and easier for us to support. Adding functions for adding and removing listeners, with a simple integ test. Next up will be the Event classes... commit e1772bb9ad9ad2f6f085e2c0f489214d2f7fd6ee Author: Damian Johnson Date: Fri Nov 23 12:14:12 2012 -0800 Revised behavior for document signature methods The spec has been revised to explain how signature methods worked. I had assumed that they were included in microdescritor consensuses and excluded from standard consensuses. Turns out however that they can be included with both and, if excluded, have a default value of 'sha1'. This is much nicer from a parsing and behavior standpoint. bug report: https://trac.torproject.org/7072 fix: https://gitweb.torproject.org/torspec.git/commitdiff/96427e8 commit 69c55aba4146861d0adead57850bea8fa9ded3ea Author: Damian Johnson Date: Thu Nov 29 18:55:10 2012 -0800 Descriptor reader hung if malformed content was in an archive The descriptor reader was missing a catch clause when parsing archives, causing the reader to die (and callers to hang) if it contained invalidly formed descriptor data. Caught by eoinof on... https://trac.torproject.org/5810 commit 6f2c1c33931b6fda738468d03ee0f835f0caca6b Author: Damian Johnson Date: Sun Nov 25 13:50:04 2012 -0800 Checking when EXTENDCIRCUIT is/isn't provided The EXTENDCIRCUIT's path argument is optional unless... * the ciruit id is non-zero * we're prior to tor version 0.2.2.9 where it was made optional: https://gitweb.torproject.org/tor.git/commitdiff/ac68704f07c2b703 Pointed out by robinson and integrates his suggestions on... https://trac.torproject.org/6666 commit d82a70a4fb874ca295c1644e3c77f24afddcbf06 Author: Damian Johnson Date: Wed Nov 28 21:16:04 2012 -0800 Minor whitespace changes Just adding empty lines in a few spots to improve readability. Conventionally we include blank lines between pydoc description, :param:, :returns:, and :raises: entries. It also find that it helps to have empty lines around conditionals and comments, though that might just be a matter of taste. No functional changes. commit 5da6b9790da266f96258c7c6d6a439ca2ef06529 Author: Eoin o Fearghail Date: Tue Nov 27 21:04:14 2012 +0000 Refinements to previous checkin after code review/feedback cf https://trac.torproject.org/projects/tor/ticket/5810 Removed most of the logging code _digest function now returns the digest in uppercase hex digest value is now calculated once & cached for evermore. moved key string manipulation code to a separate function as it is used more than once, cf _get_key_bytes() reverted change to test/integ/descriptor/server_descriptor as _digest now returns uppercase hex added some documentation to _digest() added some documentation to sign_descriptor_content() commit e0095fbe54759c45cbf6d1b120d2b17b47a0ec21 Author: Eoin o Fearghail Date: Fri Nov 23 22:16:22 2012 +0000 Implementing Relay Descriptor verification cf https://trac.torproject.org/projects/tor/ticket/5810 1) Implemented relay descriptor verification using the python-crypto lib. Code is only run if python-crypto can be imported. [cf stem.prereq.is_crypto_available()] NOTE: constructing a RelayDescriptor will now raise an exception if invalid descriptor content is used. 2) Refactored the digest() function in server_descriptor.py. 3) Added a function to the mocking lib to sign a descriptor with an auto-generated key 4) Add usage of new sign_descriptor_content() in unit tests where necessary. 5) Updated the non-ascii-descriptor file to be correctly signed. 6) Updated extra info descriptor test to use new fingerprint in non-ascii-descriptor file 7) Removed server descriptor tests that do not make sense if data is being generated dynamically. e.g. Removed test fingerprint valid test, since data now dynamically generated. commit 368fedfaffc6e86dc8febd1d6d82e54f91b6ad01 Author: Damian Johnson Date: Sun Nov 25 14:01:18 2012 -0800 Whitespace fixes The prior couple patches lacked whitespace on empty lines, causing our whitespace checker to complain. I wonder if I should stop fighting with the world here - they're clearly wrong, but I'm pretty alone in this... commit 3b6ac7bd10c38b9ef52d73bdbf134eced065e291 Author: Sean Robinson Date: Sun Nov 25 07:33:05 2012 -0700 Fix close_circuit test Signed-off-by: Sean Robinson commit d682c8cd13a0f6bffcd056a2a938db2dfa65f500 Author: Sean Robinson Date: Thu Nov 22 08:09:42 2012 -0700 Add a close_circuit method to Controller Signed-off-by: Sean Robinson commit 4c65738d100ab0802a75cc4b9d64065c8a8ef881 Author: Damian Johnson Date: Sun Nov 25 13:03:46 2012 -0800 Borrowing setup.py corrections from robinson Borrowing a couple things from robinson's patch on... https://trac.torproject.org/7565 ... including the description, updated stem url, and provides/keywords args. Tested by simply running 'python setup.py build'. commit 337f2221109ae61ed90fbf640d23fa434bed8049 Author: Damian Johnson Date: Sun Nov 25 12:11:59 2012 -0800 Adding a setup.py I've had a setup.py sitting around since the GSoC mentor summit, but didn't check it in since I haven't made a method for testing the Python 3 conversion. robinson submitted a patch adding a setup.py so guess I should stop waiting on that. commit e73c4f353612ad26b47ca2c66d8c6f4ac78f165b Author: Damian Johnson Date: Sat Nov 17 10:57:30 2012 -0800 Disabling flaky bits of descriptor tests As per 'https://trac.torproject.org/7163' the network status and server descriptor integ tests are flaky right now. We read our cached consensus and network status document for a relay, then query the control socket for that relay. This really seems like it should work, but sometimes tor replies with an unknown key error (ie, saying the relay doesn't exist). There's no point in keeping this test while it's flaky. We should investigate what's going wrong at some point so keeping the ticket open for that. commit 28d6bc73796a46ba0463768a2de35eeb3bc0a1fe Author: Eoin o Fearghail Date: Fri Nov 16 18:57:38 2012 +0000 Minor bug fix. The _get_descriptor_content function was adding whitespace to the relay descriptor that it was generating. This was incorrect. commit 505b5a6f82222f4d16e1e1cdac96c66096a52bf1 Author: Damian Johnson Date: Sat Nov 10 23:17:31 2012 -0800 Adding link for stem's development wiki We don't have anything linking stem's front page to the development wiki. In general this is ok, the dev wiki is only of interest to people wanting to hack *on* stem, not use it. However, we should have at least one link to let new contibutors find it on their own. commit 0e99a269b94f657575690734f4ffff15a17944e0 Author: Damian Johnson Date: Fri Nov 9 09:46:54 2012 -0800 Revised enumeration documentation Our enumerations were only defined in our 'Module Overview' section which sucks for a few reasons... 1. We can't interlink with the enum definitions. 2. Blocks lack any formatting that could improve readablity. 3. In the long term I'd like to replace the 'Module Overview' blocks with something better. Sphinx's python domain directives include a 'data' type, used for global values. That's exactly what our enums are so using the directive, with tables to list our enum values. commit 07c5bae7fb4fd6139afcb2a14abdddad75580e0e Author: Damian Johnson Date: Thu Nov 8 08:29:01 2012 -0800 Git image linked to gitweb rather than stem's repo Oops, not sure how I got this wrong. We want to link to our repository, not the list of projects. commit 142450fc68d8df967bc99b632154f43d4b2e3451 Author: Damian Johnson Date: Tue Nov 6 08:56:44 2012 -0800 Including exceptions in api docs Moving the exceptions to stem/__init__.py meant that they were no longer covered my any of our automodule declarations. Adding a special 'Exceptions' section to the controller's api page. commit 82747342cac4a7204b42928fe3a3f32fadea62c6 Author: Ravi Chandra Padmala Date: Tue Nov 6 19:08:03 2012 +0530 Moving exceptions to stem's toplevel module Fixing #6357 commit a5c5ae8c3af4409b8fcac10a38b21a37723237c9 Author: Damian Johnson Date: Tue Nov 6 08:09:06 2012 -0800 Reverting comment change about tor error messages The prior spelling fixes condensed a comment that we had about the error message tor will provide when we use the wrong controller password. The change makes sense, but these lines are here to tell us the exact error message tor provides (after which it's hopefully reasonably obvious why we check for the "Password did not match HashedControlPassword" part of it). commit bb0a3ace259e4f23eb53837908a3163c1c19e305 Author: Eoin o Fearghail Date: Tue Nov 6 00:00:51 2012 +0000 Changing spelling of parseable to parsable, also one tiny whitespace fix. commit 2a597415c0a8ca8203ee0282103ef0c606f1d887 Author: Eoin o Fearghail Date: Mon Nov 5 22:54:02 2012 +0000 Squashed all my spelling changes into one commit commit c58514337ec02d17260c9468ba17b1a15f959c82 Author: Damian Johnson Date: Sun Nov 4 11:04:42 2012 -0800 stem.process integ test with RUN_SOCKET target broken Ok, this puzzled me for a while. Our stem.process integ tests make '/tmp/stem_integ' to be its temporary data directory for the run, then cleans it up afterward. This is great, except that the RUN_SOCKET causes us to run with the following in its torrc... ControlSocket /tmp/stem_integ/socket It's a coincidence that both use the same tmp directory, but it's not necessarily bad. What was bad, though, is that... * the stem.process tests broke since the mkdir call for stem_integ failed * if that hadn't failed our tearDown would have deleted stem_integ, and with it our control socket causing all further tests to fail We should probably use the tempfile module to make our stem.process data directory, but for now just being lazy and having it account for the RUN_SOCKET target. Caught by Ravi on... https://trac.torproject.org/7284 commit e6f95bb7d34d308b3cfdfe7c66c4d10ddb5756c2 Author: Damian Johnson Date: Sat Nov 3 21:52:05 2012 -0700 Adding a logo for stem Logo for the stem project, curtsey of the WP Clipart project. commit 3581e963953805c9156589255b534d73e45e2156 Author: Damian Johnson Date: Sat Nov 3 21:18:47 2012 -0700 Adding contact link Adding a link to my contact page. I'd prefer for people to contact tor-dev@, but I've learned from experience that I'll get quite a bit more input by providing a text field that doesn't require subscription. commit 0df574edf8bf004e55acef809772c5e0735616ce Author: Damian Johnson Date: Sat Nov 3 20:26:39 2012 -0700 Image credits and licenses Noting where the site images come from. Almost all of them are from the collection I've gathered of free image resources (http://www.atagar.com/freeMedia/). I've tried pretty hard to avoid images with unknown licenses but there is one exception: the anvil schematic for the API link. This is a fantastic image and I've searched quite a bit to figure out where it originally came from (found it on a blog but doubt that's the source). Pity... commit e095e101160a0a739909143ae22996fb5a376e64 Author: Damian Johnson Date: Sat Nov 3 19:46:27 2012 -0700 Link for the bug tracker Adding a link to our front page for a bug tracker page on trac that's specific to our project. commit 9fa93cb5cb84a73defd7ccfcdffe82896684d03a Author: Damian Johnson Date: Sat Nov 3 17:12:52 2012 -0700 Error when trying to parse the last line in router_status_entry.parse_file() When calling the Controller's get_network_statuses() method the last line would cause an exception because we tried to parse an empty string as a router status entry. Aborting when we get to the end instead. commit 86d6b6114ad0929ee787cf454c9d93f1af3f9033 Author: Ravi Chandra Padmala Date: Thu Nov 1 23:20:25 2012 +0530 Rewrite Controller.repurpose_circuit test to use a newly created circuit. Attempting to fix https://trac.torproject.org/projects/tor/ticket/7259 by using a newly created circuit instead of parsing the GETINFO circuit-status output. commit 54e3970b674665d58b786b52af02b18bdb686136 Author: Damian Johnson Date: Wed Oct 31 08:46:55 2012 -0700 Supporting files in whitespace checks We called os.walk() when determining the files for which we want to check whitespace. However, when presented with a file rather than a directory this causes us to not check anything... >>> list(os.walk("/tmp/foo")) [] This broke our attempts to check 'run_tests.py', and let a couple whitespace issues slip in. Fixing get_issues()'s handling for individual files. Issue caught by Eoin on... https://trac.torproject.org/7263 commit d46292f0dce4a189457b02a1ea951e845094c509 Author: Damian Johnson Date: Wed Oct 31 08:40:21 2012 -0700 Using absolute paths for whitespace checks We were using relative paths for our whitespace checks, which caused varying behavior based on our cwd... atagar@morrigan:~/Desktop/stem$ ./run_tests.py --unit ... TESTING PASSED (7 seconds) atagar@morrigan:~/Desktop/stem$ cd .. atagar@morrigan:~/Desktop$ stem/run_tests.py --unit ... WHITESPACE ISSUES * stem/example.py line 18 - indentation should match surrounding content (2 spaces) line 19 - missing 'with' import (from __future__ import with_statement) line 23 - indentation should match surrounding content (2 or 8 spaces) line 35 - indentation should match surrounding content (4 spaces) line 72 - line has trailing whitespace line 76 - indentation should match surrounding content (0 spaces) line 77 - indentation should match surrounding content (0 spaces) * stem/run_tests.py line 289 - indentation should match surrounding content (2 spaces) line 486 - line has trailing whitespace TESTING PASSED (19 seconds) Note that 'example.py' isn't part of stem. It's an untracked file that I have in the stem directory. The reason that it's being included in the whitespace check is that we're grabbing all python files under 'stem' which, now that we're one level up, is the whole project. Using absolute paths that are relative of run_tests.py so we get consistent results. commit 56aac96d6213f28a6b597c640affc7a5a963bf75 Author: Damian Johnson Date: Mon Oct 29 19:26:42 2012 -0700 Conficting DataDirectory for process tests The process module's integ tests start and stop tor instances. However, if you're already running tor then the DataDirectory of these instances will conflict with what you're already running... Oct 29 19:18:11.532 [notice] Tor v0.2.1.30. This is experimental software. Do not rely on it for strong anonymity. (Running on Linux i686) Oct 29 19:18:11.540 [warn] ControlPort is open, but no authentication method has been configured. This means that any program on your computer can reconfigure your Tor. That's bad! You should upgrade your Tor controller as soon as possible. Oct 29 19:18:11.541 [notice] Initialized libevent version 1.4.13-stable using method epoll. Good. Oct 29 19:18:11.542 [notice] Opening Socks listener on 127.0.0.1:2777 Oct 29 19:18:11.543 [notice] Opening Control listener on 127.0.0.1:2778 Oct 29 19:18:11.543 [warn] It looks like another Tor process is running with the same data directory. Waiting 5 seconds to see if it goes away. Oct 29 19:18:16.546 [err] No, it's still there. Exiting. Issue spotted by gsathya on... https://trac.torproject.org/7251 commit 89ec87cfa3c119d96ee32d7debec0dde9e97cb51 Author: Damian Johnson Date: Mon Oct 29 19:07:03 2012 -0700 Not properly aborting test when skip() is unavailable The test_cached_microdesc_consensus test is missing a return after its skip, causing us to fail if... 1. We don't have a microdescriptor consensus. 2. We're running python 2.5 or 2.6 Spotted by gsathya on... https://trac.torproject.org/7250 commit 081228149f80448dc9097d7e260b2738792250ca Author: Damian Johnson Date: Mon Oct 29 18:59:23 2012 -0700 Helper function for descriptor tutorial The 'Mirror Mirror on the Wall' tutorial had some repetition due to having both an example for getting descriptors from the disk and through the control port. Moving that code to a helper function so I can drop the common parts from the second example. commit aac44fe0a23f409e61484f8ce6c1b5c79a312714 Author: Damian Johnson Date: Mon Oct 29 09:03:52 2012 -0700 DictWriter.writeheader() introduced in python 2.7 The writeheader() method of the DictWriter class was added in python 2.7, breaking our 2.5 and 2.6 compatability... ====================================================================== ERROR: test_multiple_descriptor_types ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/unit/descriptor/export.py", line 91, in test_multiple_descriptor_types self.assertRaises(ValueError, export_csv,) File "/usr/lib/python2.6/unittest.py", line 336, in failUnlessRaises callableObj(*args, **kwargs) File "/home/atagar/Desktop/stem/stem/descriptor/export.py", line 39, in export_csv export_csv_file(output_buffer, descriptors, included_fields, excluded_fields, header) File "/home/atagar/Desktop/stem/stem/descriptor/export.py", line 91, in export_csv_file writer.writeheader() AttributeError: DictWriter instance has no attribute 'writeheader' Noting the prereq in our pydocs and ignoring the 'header' flag if we can't support it. commit b09de1093e7a7237d3825d5728126cbf2f15842b Author: Damian Johnson Date: Mon Oct 29 08:47:26 2012 -0700 OrderedDict substitute for older python versions In d30a628e I simplified the _get_descriptor_components() by using the collections.OrderedDict builtin. Unforutunately OrderedDict was introduced in python 2.7, breaking our 2.5 and 2.6 compatability... ====================================================================== ERROR: test_voting_delay ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/unit/descriptor/networkstatus/document_v3.py", line 359, in test_voting_delay document = get_network_status_document_v3({"voting-delay": "12 345"}) File "/home/atagar/Desktop/stem/test/mocking.py", line 782, in get_network_status_document_v3 return stem.descriptor.networkstatus.NetworkStatusDocumentV3(desc_content, validate = True) File "/home/atagar/Desktop/stem/stem/descriptor/networkstatus.py", line 458, in __init__ self._header = _DocumentHeader(document_file, validate, default_params) File "/home/atagar/Desktop/stem/stem/descriptor/networkstatus.py", line 541, in __init__ entries = stem.descriptor._get_descriptor_components(content, validate) File "/home/atagar/Desktop/stem/stem/descriptor/__init__.py", line 289, in _get_descriptor_components entries = collections.OrderedDict() AttributeError: 'module' object has no attribute 'OrderedDict' Caught by eoinof on... https://trac.torproject.org/7244 I'm fixing this by adding an OrderedDict substitute that's under the MIT license... http://pypi.python.org/pypi/ordereddict commit 0afe836c94f83f3e11435cdb9f8a3f33dbb9f106 Author: Damian Johnson Date: Sun Oct 28 14:37:39 2012 -0700 Fixing spelling of check_requriements() Spelling mistake caught by eoinof... https://trac.torproject.org/7245 commit 1d8cdb206e54354fa563f578615b3079da8c7abb Author: Eoin o Fearghail Date: Sun Oct 28 20:45:23 2012 +0000 Added a count of how many, if any, tests were skipped. This count is then displayed at the end of the test run as skipped tests may not otherwise be noticed! commit 39d90df0f9f48c318a9e1ba4ca32836b40a6eb81 Merge: a00fb8a f31b19b Author: Damian Johnson Date: Sun Oct 28 13:54:05 2012 -0700 Revised API documentation Stem's API docs were a monolithic dump of our pydocs. This was utterly unreadable so breaking it down into individual module pages. I also applied some uniform formatting changes to all of our pydocs to make them more reader friendly. commit f31b19b966bc2f1243f58eebf8a1db0ce2876ff0 Author: Damian Johnson Date: Sun Oct 28 13:49:17 2012 -0700 Making the logging NullHandler private We have a null logging handler to avoid having the logging module give a warning about no handlers being present. There's no reason for stem's users to care about it so making it private. commit 07c75fa254926b2e14ef91d8edc9e8e20170ce81 Author: Damian Johnson Date: Sun Oct 28 13:48:04 2012 -0700 Spelling corrections Spell checked all of the stem/* files and the test utilites. Quite a few mistakes... commit 1e1d84143b26212d3158f30555145687d3c653d3 Author: Damian Johnson Date: Sun Oct 28 12:31:04 2012 -0700 Moving AuthMethod docs to the module overview We were using definition entries for the AuthMethod enums which was nice, but gave them more emphasis than they deserved. Users won't usually touch a PROTOCOLINFO response directly so the enumeration really isn't that important. commit a78ea67a521197b8c57c8556df4250ec5161b6b0 Author: Damian Johnson Date: Sun Oct 28 12:17:20 2012 -0700 Linking image and clone command to gitweb I disliked having a '(browse)' link for a couple reasons... 1. people might copy it as part of the clone command if they're new to git 2. it was pretty small considering its relevant importance Instead linking the clone command itself and the git image. commit ebb3de86c9af92001845f3560f857615e1f633eb Author: Damian Johnson Date: Sat Oct 27 19:32:43 2012 -0700 Revised API docs for stem.response This is a little different from the other modules in that we're including stem.response.__init__ as an automodule then autoclasses for all of the contents. The response classes each have precious little documentation to having them each on their own page would be pointless. Also moving the AuthMethod enum to the connection so we can drop the protocolinfo module documentation (it wouldn't be included since we're using autoclass). commit d2cd6ccb0c9df470478278e570f732beec973df6 Author: Damian Johnson Date: Sat Oct 27 18:42:32 2012 -0700 Revised API docs for stem.process commit 04ea42cba22be4defeddf0d11803c23476612bdd Author: Damian Johnson Date: Sat Oct 27 18:16:34 2012 -0700 Revised API docs for stem.control commit 08fa765d219b24a8c0bb0142b71a940cd6fd7bf2 Author: Damian Johnson Date: Sat Oct 27 17:42:38 2012 -0700 Revised API docs for stem.socket commit ec185b901de8829f12c7e0c1778f079e444ce14a Author: Damian Johnson Date: Sat Oct 27 17:26:35 2012 -0700 Revised API docs for stem.connection commit 4167d59d8fa9d72656381dec39c6a7315176ebed Author: Damian Johnson Date: Sat Oct 27 16:57:34 2012 -0700 Collecting API stubs under a directory Moving all the autodoc stubs under an api directory to keep things tidy. commit 8459d01c2ce4feb9c661e9bf958da5169eae58a3 Author: Damian Johnson Date: Sat Oct 27 15:53:41 2012 -0700 Revised API docs for stem.util.tor_tools commit 6de23cdc3fb45eb9493c9d3819585f414fecb9cf Author: Damian Johnson Date: Sat Oct 27 15:49:06 2012 -0700 Revised API docs for stem.util.term commit d2a3acff92383ea95602acd4382cf648b5693da4 Author: Damian Johnson Date: Sat Oct 27 15:42:02 2012 -0700 Revised API docs for stem.util.system commit cc37ede99703c0db0a34a3b3125339af473fe810 Author: Damian Johnson Date: Sat Oct 27 15:35:37 2012 -0700 Revised API docs for stem.util.proc commit 9217bb1b4bfe07235604cd2ac94d41995b8cbd33 Author: Damian Johnson Date: Sat Oct 27 15:28:28 2012 -0700 Revised API docs for stem.util.log commit e4014bd0e70fc3af567f6e762d585127141279e6 Author: Damian Johnson Date: Sat Oct 27 15:21:30 2012 -0700 Revised API docs for stem.util.str_tools commit 15ff02a5d2d654fb6afd6791d635ee2421f1b604 Author: Damian Johnson Date: Sat Oct 27 15:14:26 2012 -0700 Revised API docs for stem.util.connection commit a30d89064f857e355019dbd70a37997abe475699 Author: Damian Johnson Date: Sat Oct 27 14:02:09 2012 -0700 Revised API docs for stem.util.conf The utility's header was pretty sucky so rewrote it to better exemplify how this module should be used. commit 907fb84cbc11d054718ba045ac7028676f4c1746 Author: Damian Johnson Date: Sat Oct 27 10:37:21 2012 -0700 Revised API docs for stem.util.enum commit 8ee4c14fe0784c82637f7d3bb0b6d496c44717c9 Author: Damian Johnson Date: Sat Oct 27 10:23:51 2012 -0700 Revised API docs for stem.descriptor.router_status_entry commit 670334b61dddec3b53ab5ea47d80149a81795e54 Author: Damian Johnson Date: Sat Oct 27 10:08:44 2012 -0700 Line wrapping pydoc clauses I had been avoiding line wraps in :var:, :param:, and :return: entries to avoid confusing the spinx compiler but on reflection it's output is uneffected when we indent the following line. Just wrapping the lines that were really unweildy rather than going for a strict eighty character width. Sorry lines-of-code-must-be-shorter-than-my-iphone-screen zelots. commit 29f500874cb90aa6d9c3626a609dfc957950590f Author: Damian Johnson Date: Sat Oct 27 09:42:03 2012 -0700 Revised API docs for stem.descriptor.networkstatus commit 934a475cb1e07126847acdbe1166abf1d48128c1 Author: Damian Johnson Date: Sat Oct 27 08:19:06 2012 -0700 Revised API docs for stem.descriptor.extrainfo_descriptor commit 2c4e518adf545633e05fc97bba4718f9aebe3396 Author: Damian Johnson Date: Fri Oct 26 09:05:00 2012 -0700 Revised API docs for stem.descriptor.server_descriptor commit 6925f6147120b645338f20abb8597d7b7787cf31 Author: Damian Johnson Date: Thu Oct 25 18:31:02 2012 -0700 Revised API docs for stem.descriptor commit 4b528c5e75bea5892a63ed5d63d3135d46971503 Author: Damian Johnson Date: Thu Oct 25 09:37:01 2012 -0700 Revised API docs for stem.descriptor.export commit 3c04eb6092371526126ef745903bb3e96782511a Author: Damian Johnson Date: Thu Oct 25 09:17:35 2012 -0700 Adding toctree declarations Sphinx gives a warning saying "document isn't included in any toctree" if all *.rst documents aren't included in a toctree somewhere. I'm not using a toctree as our primary form of navigation so it doesn't really make sense, but it's easy enough to make a contents page off to the side that satisfies sphinx. commit 31cc49aa7606ce0f2b18a18fcdb0823a8d4448ec Author: Damian Johnson Date: Tue Oct 23 09:44:51 2012 -0700 Revised API docs for stem.descriptor.reader commit d00d2ac5a4679a428358f06242cb0980660bb100 Author: Damian Johnson Date: Sun Oct 21 21:26:26 2012 -0700 Revised API docs for stem.exit_policy commit 0cc97d2dcc15ce227983cfc72c234bf995b4857a Author: Damian Johnson Date: Sun Oct 21 20:04:30 2012 -0700 Only including prepared API docs Our sphinx-apidoc made it so all our pydocs were included in our built documentation. This is great, except that those documentation dumps are completely unreadable. Dropping the sphinx-apidoc call in favor of including specific autodoc pages we've prepared. Made some revisions to stem.version's documentation and including that to start. commit a00fb8ae0e2f7e9fe4aac4f28f87198a412feb52 Merge: fc7c5cb be5af88 Author: Damian Johnson Date: Sat Oct 20 16:00:03 2012 -0700 String utility toolkit Few utility functions for string generation (mostly for making human readable labels of size or time metrics). cr: neena (https://trac.torproject.org/7158) commit be5af88894666350f74a1944f9d3a37c4c827cfe Author: Damian Johnson Date: Fri Oct 19 13:50:33 2012 -0700 Using get_size_label() in the tutorial Making the "Mirror Mirror on the Wall" tutorial examples a little nicer by using our new str_tools.get_size_label() helper. commit e06b76b8ae68446edc3c3123412a514a00e2fba7 Author: Damian Johnson Date: Fri Oct 19 13:33:19 2012 -0700 Expanding str_tools unit tests Testing the common edge cases that come to mind, and making a few small fixes. I was also missing examples for a couple functions. commit 70227efa08d77f65b1eea069ae93bd0ba0bf1993 Author: Damian Johnson Date: Fri Oct 19 09:41:34 2012 -0700 Utils for human readable time and size metrics Snagging the utilities from arm's src/util/uiTools.py for making human readable labels of time and size metrics. They'll not only be handy for stem's users, but also our exception messages and something I want to use in our tutorials. I've simplified these functions a bit and added unit tests for the pydoc examples. More tests are needed, though... commit fc7c5cb0e0b06f26c7d8a1e0af348453127e3d4c Author: Damian Johnson Date: Fri Oct 19 14:00:17 2012 -0700 Parsing or-address in all server descriptors As mentioned in 19ba050 the 'or-address' lines initially just applied to bridges, but would later be relevant to all relay types. Looks like that day has come since 'or-address' now appears in my cached consensus (indicated via integ testing failures). commit e33f55d57b100d9f2e6f697e849811b28cd3bcfb Author: Sathyanarayanan Gunasekaran Date: Wed Oct 17 17:54:35 2012 +0530 Fix missing paranthesis commit 9ee778c6e723962ca8e3aa8470153a75a650d97b Merge: f717e88 a0470a6 Author: Damian Johnson Date: Tue Oct 16 09:05:20 2012 -0700 Controller methods for querying descriptors Adding convenience methods for querying descriptor information (except microdescriptors, for which we don't yet have a parser). commit a0470a6285c2462535f53536e7ed969725356049 Author: Damian Johnson Date: Tue Oct 16 09:03:27 2012 -0700 Using get_server_descriptors() in the tutorial Using the controller's new get_server_descriptors() method to exemplify an alternate way of doing the 'Mirror Mirror on the Wall' tutorial without reading the cached files directly. Both methods have advantages so I definitely want to exemplify each. commit c52521efe273e8091ec95554d24ed172e755d29e Author: Damian Johnson Date: Tue Oct 16 08:47:03 2012 -0700 Controller methods for fetching all descriptors Adding a couple methods to iterate over the server descriptors or router status entries. This definitely isn't what I want in the long run - we should iterate over the contents as they're being read from the socket rather than reading the whole thing into memory. Meh. Might as well get the functionality that we want now. The fix will be transparent for users later anyway. commit 79612002788e1f9f80733974b17eefa976055d06 Author: Damian Johnson Date: Mon Oct 15 19:12:15 2012 -0700 Controller methods for querying descriptor info Adding a get_server_descriptor() and get_network_status() method for querying server descriptors and router status entries. commit f717e8801374939ea969ef4c73a556f15db02dab Merge: 2a9d3c8 f996e24 Author: Damian Johnson Date: Mon Oct 15 09:14:57 2012 -0700 Basic stem website Making a front page and pages for our three most important sections... * Tutorial * API * Download All three are pretty minimal, but definitely better than what we had. commit f996e24b794c0f96e8e8221cdde5f21d8c2cf900 Author: Damian Johnson Date: Mon Oct 15 09:13:58 2012 -0700 Adding stub for API page Moving the old API docs to its own page. It's presently pretty minimal and sucky, but at least moving it to the right spot for now. commit 031ba36e8d67e13e0562b280ad5a89ca42c70236 Author: Damian Johnson Date: Mon Oct 15 09:07:01 2012 -0700 Adding a download page Page that'll later have our download options. For now it just has git instructions and a 'coming soon' filler. commit 8c767d35514ca345d051c94246668e97151d1c73 Author: Damian Johnson Date: Sun Oct 14 18:09:09 2012 -0700 Moving tutorial to dedicated page Beginning of my grand scheme to make stem's front page a simple listing of its sections (tutorial, API, release notes, download, etc). Moved the tutorial to its own page with a summary of the tutorials listed at the top. The tutorial is linked from the main page by an image link. commit eb28305879647bd628794674517da4377c127819 Author: Damian Johnson Date: Sun Oct 14 13:20:16 2012 -0700 Moving tutorial tests to be unit tests Both of our integ tests for the tutorial examples were a bit clunky, the controller test because it could only run with newish tor versions and the descriptor test because it had to make use of metrics descriptors. Oh, and they didn't actually assert anything. Moving the tests to be unit tests instead. This included greatly expanding our mocking capabilities, supporting arbitrary mock objects. I'm not sure if the end result is any better than our prior integ tests since we're mocking pretty much everything (and in the case of the descriptor test it's pretty ugly). On the other hand the tutorial tests are primarily for basic syntax and compatability with our current version of stem so guess it's ok... All this said, I *really* like our arbitrary object mocking capability... commit 11edc074fcd9f5d178899d3f4d8b6545bcd5d429 Author: Damian Johnson Date: Sun Oct 14 11:30:01 2012 -0700 Replacing reversed() in tutorial Dropping the reversed() call in the tutorial in favor of providing the 'reverse = True' argument to sorted(). Thanks to gsathya for the suggestion. commit 2a9d3c856c5dfe4310a836318ae45069a64fde3b Author: Damian Johnson Date: Sat Oct 13 19:27:17 2012 -0700 Clarifying what the tutorial's reader provides Just realized that newcomers wouldn't know that the DescriptorReader provided RelayDescriptor instances in the example (and hence where the methods that we use come from). Clarifying that detail. commit 23513eb220ed195ee73fcadd6fb826c3581810fc Author: Damian Johnson Date: Sat Oct 13 19:09:06 2012 -0700 Adding a tutorial to our sphinx front page Tutorial for basic stem usage, with tests for the examples we provide. Our documentation is still incredibly beginner unfriendly, but at least this gives them a place to start. On a side note one of our integ tests kinda sorta killed our test instance by calling... controller.signal("INT") We didn't notice this because it happened in our very last test. commit 5e29e28d16dc050d4cc5dcac9ae92d20935651d7 Author: Damian Johnson Date: Sat Oct 13 16:34:47 2012 -0700 Merging testrc.sample with settings.cfg We run the tests all the time but never with a configuration file so deemphasising the capability by merging the sample rc file with the settings.cfg (which is actually... well, used). If someone really wants to configure the tests this way then they might as well look at the settings.cfg anyway since it'll provide the full range of options. commit ea356b738fff3bab4a9e468298ea7807aff37960 Merge: 5cd457b ec335d1 Author: Damian Johnson Date: Sat Oct 13 11:33:47 2012 -0700 Network status document handling Parsing and lots 'o test for handling network status documents. This includes both v2 and v3 documents (votes, consensus, and microdescriptors). https://trac.torproject.org/6569 commit ec335d1b5bf4368982bc9736a271f675a6a7c60d Author: Damian Johnson Date: Thu Oct 11 19:54:05 2012 -0700 Supporting v2 documents in the descriptor reader The v2 descriptor's integ data broke the descriptor reader tests. Turned that I had forgotten to add v2 documents to the types that it recognizes. commit 372ee9836b98af582eecf3d35844397d8935bd9b Author: Damian Johnson Date: Thu Oct 11 19:27:36 2012 -0700 Parsing and tests for network status document v2 Parser, unit, and integ test for version 2 network status documents. These documents are deprecated and no longer generated, however we still need a parser to read older consensuses. Unlike the v3 parser I'm cutting a few corners... - not validating parameter ordering - no validation that header/footer parameters haven't swapped places - only the bare minimum unit test, no tests for invalid content We can remedy these if necessary but with the growing irrelevance of v2 consensus parsing I doubt we ever will. Plenty of more important things to do. commit 0793113b37afa737862440363983105e820640bd Author: Damian Johnson Date: Thu Oct 11 07:53:41 2012 -0700 Renaming NetworkStatusDocument to NetworkStatusDocumentV3 Changing the name to make room for v2 documents and future versions. commit 925e8a3e0232f734c1067e88ac8898e446dee35e Author: Damian Johnson Date: Wed Oct 10 19:15:29 2012 -0700 Reordering document's integ test attribute checks Changing the order of the attribute assertions to match the classes, and adding a few checks that were missing. commit 9170c0ff411bca266bd90af4bf423c5c0decb44e Author: Damian Johnson Date: Wed Oct 10 09:35:59 2012 -0700 Replacing _strptime() helper with normal datetimes I kinda liked the idea of the _strptime() helper and briefly moved it to the runner so I could use it in other tests, but on reflection it doesn't really help readability nor is it appreciably shorter. Swapping its usage to normal datetimes for uniformity. commit b0d24bee1d2a00f20aba8953a15df1e006860b33 Author: Damian Johnson Date: Wed Oct 10 09:15:13 2012 -0700 Refactoring network status integ tests General refactoring for the network integ tests, mostly moving things around and renaming to make things more readable. The only functional difference is that a couple tests were using the 'assert' keyword rather than testing assertions. This was a bug - it meant that we'd skip those tests unless running in assert mode and even then it would cause exceptions rather than testing failures. commit c20cfdcd6d87a04d23c508b8b333086c77000d77 Author: Damian Johnson Date: Tue Oct 9 09:15:41 2012 -0700 Addressing issues spotted by integ tests Enough of this unit testing, time to run our new parser against actual network status content. Unsurprisingly this ran into a couple issues... - Microdescriptors have an extra field on their 'directory-signature' lines. This is undocumented so it'll also need a spec fix... https://trac.torproject.org/7072 - Our parser for 'directory-signature' was only reading the first one, rather than iterating over all entries. Most of the rest of the changes are just revising the integ tests that Ravi wrote to accomidate changes I've made to the classes. commit 78e09a971b7c242dc1d0a6b95718b78edd5cb32b Author: Damian Johnson Date: Tue Oct 9 07:56:33 2012 -0700 Adding tests for the network status document examples Including unit tests for the header pydoc examples to check that they're runnable. I also changed the first example to show that the consensus file doesn't need to remain open when using that method. commit 3ddd3c55da9b859dcce218283ee91aab251bf230 Author: Damian Johnson Date: Mon Oct 8 09:27:57 2012 -0700 Rewriting network status module pydocs Revising the header documentation for the network status document module. The bit about v2 support is presently a lie, and I need to include the examples in the tests. Otherwise, this module is almost done! commit 27356d41908c70fdec23d26d4f9933f8fefb71f9 Author: Damian Johnson Date: Mon Oct 8 08:50:13 2012 -0700 Simplifying _get_descriptor_components() usage The _get_descriptor_components() provided the first and last keywords since those are often needed for validation. However, this is pointless now that we're using an ordered dictionary (we can simply check the key listing). commit 98d8e4685bab0dd107f5dd6f185492d6f3a5c136 Author: Damian Johnson Date: Mon Oct 8 08:36:47 2012 -0700 Dropping test for empty dir-source fields Nick replied on 'https://trac.torproject.org/7055' saying that the fields can't be empty strings, so dropping the test. Also adding a validation check that the hostname isn't an empty string. commit 60868f5f161dc0db59f4c19019b75c29f417b37c Author: Damian Johnson Date: Sun Oct 7 18:53:18 2012 -0700 Dropping unused descriptor helpers Removing the helper functions for the prior parsers of network status documents. commit 4216b5f1d5762d229945306508ea078c9fd1902c Author: Damian Johnson Date: Sun Oct 7 18:47:27 2012 -0700 Supporting microdescriptor flavored consensuses Adding support for microdescriptor flavored consensuses into the NetworkStatusDocument class. It made sense to have a separate class for it, but on the other hand it *is* still a v3 consensus and the only impact the flavor has is alternate router status entries so just blending a 'flavor' and 'is_microdescriptor' attribute in. commit a1d2a0ffffd5aeec96c37fbbf1b4134c9b34b631 Author: Damian Johnson Date: Sun Oct 7 18:18:14 2012 -0700 Unit tests for minimal v2 and microdescriptor v3 router status entries Tests to exercise basic parsing for the v2 and microdescriptor v3 RouterStatusEntry subclasses. They're both largely a subset of the v3 RouterStatusEntry so I don't plan to add additional tests for the moment. commit b8ca825a7f6dcde9f7689bca41a0938598749b4f Author: Damian Johnson Date: Sun Oct 7 17:59:08 2012 -0700 Module for V2, V3 and Microdescriptor router status entries Moving the router status entries to their own module and adding classes for V2 entries and microdescriptors (both still completely untested). The abstraction is based on the ServerDescriptor breakdown but with more liberal use of helpers for the actual parsing. commit 72cb24dde90c928135a686707db6baa5a88d5147 Author: Damian Johnson Date: Sat Oct 6 14:09:34 2012 -0700 Unit test to include directory authorities in document Adding a document test that includes authority entries. The especially interesting bit of this is checking that validation propagates to the KeyCertificate constructor (ie, it's checking that the 'validate' flag is being passed from the document to authority and authority to cert classes). commit dc89b293bb3f8a282766971597ada90723f17fa7 Author: Damian Johnson Date: Sat Oct 6 13:51:29 2012 -0700 Unit tests for DirectoryAuthority fields Tests for the DirectoryAuthority's individual fields, and fixes for a couple issues they uncovered. commit 5ac628e7c8bd95c6c2cade6477aceef51a941b2e Author: Damian Johnson Date: Sat Oct 6 12:26:10 2012 -0700 Adding unit test for empty dir-source values Just realized that the nickname and hostname fields, which lack any constraints, might also allow for empty values. I'm not really clear about this so asking for clarification... https://trac.torproject.org/7055 commit 347d4c03a63b8a3f81bee83ee990e5ea4ada9125 Author: Damian Johnson Date: Fri Oct 5 08:53:17 2012 -0700 Boilerplate descriptor tests for DirectoryAuthority Copying a few general tests from the other descriptors. I should move these into a descriptor test superclass so I don't need to keep copying them. That said, there are some minor tweaking that might prevent that... commit a09b1e038fc97dada370e01ad8fd0c4f81b90c12 Author: Damian Johnson Date: Thu Oct 4 09:39:27 2012 -0700 Fix when router status entries have duplicate lines Spotted a bug with the RouterStatusEntry where we have an undefined 'key' variable when validating that we don't have any duplicate entries. Turns out that this use case wasn't being tested so added a unit test to exercise the bug first. commit bf192cb70a770fe2551add747bbdb6aed2f94d9d Author: Damian Johnson Date: Thu Oct 4 09:04:53 2012 -0700 Parsing for DirectoryAuthority Rewriting the parser for the DirectoryAuthority class. Still passes the minimal descriptor tests so next gonna add the field specific unit tests. commit 11070d9a91e0c00b4b5854fc935c5fc7a871e97f Author: Damian Johnson Date: Sat Sep 29 12:41:24 2012 -0700 Unit tests for minimal DirectoryAuthority instances Adding unit tests for the minimal vote and consensus directory authority entry. Presently this is just exercising the old parsing code (which I broke in a few places during some of my earlier refactoring). commit 17ce11ac3822ec811fc9b25ed28f24391ae50137 Author: Damian Johnson Date: Sat Sep 29 11:19:59 2012 -0700 Dropping _read_keyword_line_str() and line_matches_keyword() Dropping a couple of the descriptor helper functions that are no longer used. There's a couple more that I'd like to get rid of but they're not yet fully replaced. commit e52f0ec96b01db7da8002479b9a864a6f18f6db8 Author: Damian Johnson Date: Sat Sep 29 11:17:09 2012 -0700 Dropping the old KeyCertificate class We have a new shiny and tested KeyCertificate class so dropping the old one. commit 7986afa40937657391346dea5b4aa6aeb561c870 Author: Damian Johnson Date: Sat Sep 29 11:13:34 2012 -0700 Unit tests for KeyCertificate fields Remaining unit tests that I'm planning to do for the KeyCertificate. Oddly this is the first time that I've added a unit test for malformed content in a key block field (oops). commit 92b691c5e9ace6803a5ed24d42e7ec41665595c8 Author: Damian Johnson Date: Sat Sep 29 10:19:16 2012 -0700 Boilerplate KeyCertificate unit tests General unit tests that I've been including with most descriptor types. I should probably include these in a more systematic fasion... commit 50cc258286cd31da9571f8010f5f24b2e56bf386 Author: Damian Johnson Date: Fri Sep 28 09:32:21 2012 -0700 Dropping a coupel unused imports Ran pylint over the recently refactored files. It caught a couple of unused imports. commit 8f89166499dafcd155022e07bbf39eafd7cccf4e Author: Damian Johnson Date: Fri Sep 28 09:05:35 2012 -0700 Moving network status document mocking into util The network status document has a few oddities compared to the other descriptor types. It includes... - ordering constraints outside of the mandatory fields - keyword-only lines - router status entries - different mandatory fields based on if it's a vote or consensus Imho this new version handles these a lot more elegantly than the prior helper that was embedded in the network status document unit tests. commit 6f5d91666562e562fd773636ffcd292abff0d939 Author: Damian Johnson Date: Fri Sep 28 08:03:56 2012 -0700 Moving key certificate mocking into util commit d1ed7c2f9be307d8968ba56fd1906fcd44c1ab25 Author: Damian Johnson Date: Thu Sep 27 09:40:06 2012 -0700 Moving router status entry mocking into util commit 3923916993535c87d95c223bbf9fd0479fc0dc68 Author: Damian Johnson Date: Thu Sep 27 09:10:27 2012 -0700 Moving extrainfo descriptor mocking into util The extrainfo descriptor unit tests had its own helper function for getting mock descriptor content. This is the same thing as what the server descriptor unit tests use so moving it to the mocking module. commit a5e8d26381fc9787a8cbc147497e4fa397d70fc5 Author: Damian Johnson Date: Thu Sep 27 08:22:12 2012 -0700 Generalizing mocking.get_server_descriptor() All of the descriptor unit tests (server, extrainfo, and document components) need a similar type of mock data. I've been doing this in an adhoc fashion but this can stand for a bit of uniformity. Starting off by cleaning up the server descriptor mocking. As per the convention for the mocking module, making it provide a mock object by default with an option to get the raw content. commit 5d1a11fa4f479cac4d4987d9841889ea251b2eba Author: Damian Johnson Date: Wed Sep 26 09:00:22 2012 -0700 Unit test for parsing a minimal key certificate Test for minimal key certificate parsing, and related fixes for the KeyCertificate class. The main gotcha in this is that KeyCertificates don't need to have a prescribed order (unlike other network status document fields). commit 6ee5abbe5c1137b4a311d5b7ef5003874982008e Author: Damian Johnson Date: Sun Sep 23 14:20:49 2012 -0700 Initial KeyCertificate implementation First stab at parsing the authority section's key certificates. This is completely untested, next step is to write some unit tests for it. commit 2e65e58568469f460fedf965164b1355fae61204 Author: Damian Johnson Date: Sat Sep 22 19:01:16 2012 -0700 Begin revising the DirectoryAuthority God damnit. I thought I was almost done but, on reading the spec some more, the key certs and authority entries deserve their own descriptor type just like the router status entries. Does this document have no end? Saving the top-down approach I was taking but putting it on ice for now. I should work at this from a bottom-up fashion instead, starting with the key certs and starting a new batch of tests for it. Fun fun. :( commit 7a13a14c337ecf30162e82d75c904910af884b11 Author: Damian Johnson Date: Sat Sep 22 18:06:20 2012 -0700 Unit test for misordered content I added a test a while back to check that we detect misorderd document attributes, but it didn't pass at the time so I skipped it. The document parsing rewrite is almost done so enabling the test. commit 72c561b9adb7e3963aa14da98b8d9bc77df533dd Author: Damian Johnson Date: Sat Sep 22 17:57:49 2012 -0700 Unit tests for parse_file() and router entries On reflection the reason that the prior changes passed the unit tests so easily was because the parse_file() function and inclusion of router status entries was completely untested by my unit tests. The RouterStatusEntry class itself it tested, but not its inclusion in a document. Integ tests would certainly cover this, but I want the unit tests to exercise everything too. Adding the missing tests and some fixes for issues that they revealed. commit a5babce203784eb45f7c585edbcee4caf2362212 Author: Damian Johnson Date: Sat Sep 22 14:12:21 2012 -0700 Generalizing how router entries and authorities are parsed The _get_routers() and _get_authorities() were essentially doing the same thing. Replacing both with a more general _get_entries() helper that reads a range of the document and constructs instances for it. Taking advantage of this nicer helper's keyword arguments to make the code more readable (functions that take a ton of positional args are is a pita). I'm a bit surprised (and concerned) at how easily this passed unit tests. No doubt I've broken the integ tests but I'm not putting any effort there until I've finished the document parser rewrite. commit e4185194801b6afd10671e6e7a1a53977a598c29 Author: Damian Johnson Date: Sat Sep 22 13:18:16 2012 -0700 Getting rid of the _get_document_content() helper Too many helper functions and the code becomes fragmented. The _get_document_content() was only used a couple places, and both of those were actually better with their own slightly different implementations. commit 71240065e2a89ab1c8eeb9f865fddc294742cb8a Author: Damian Johnson Date: Sat Sep 22 12:46:03 2012 -0700 Refactoring how authority information is fetched Authorities are another section, like the header, entries, and footer so fetching it from the _get_document_content() helper. I'm not really happy with this code yet, but it's a step in the right direction and lets us finally get rid of _parse_old(). commit 641bed527c019a92e47dd769bca23fa04411d9c3 Author: Damian Johnson Date: Fri Sep 21 09:00:19 2012 -0700 Breaking up the header and footer from NetworkStatusDocument The NetworkStatusDocument class was starting to get monsterous, and it was just gonna get worse. A network status document consists of four sections... - header - authorities - router status entries - footer Making the NetworkStatusDocument a thin container for these four, and making separate classes for them. This has made the code much nicer. The only disadvantage that I've seen is that validation is done in pieces so if, for instance, you're missing mandatory fields from both the header and footer you now won't be told about both in a single error message. Instead the header will be parsed first, fail, and just tell you about those. That said, this is a pretty minor regression and well worth the improved maintainability. commit 1f868090e2d641ddcb49d02bd15b5894f5bf6923 Author: Damian Johnson Date: Tue Sep 18 09:36:56 2012 -0700 Parsing the directory-signature and unrecognized lines Finishing up with the footer. It doesn't make sense for the DirectorySignature or DirectoryAuthority to be Descriptor subclasses (cuz... well, they aren't descriptors). However, I like having this struct class rather than providing our callers with a tuple list. I should probably do this for other descriptor documents too... commit ea3102387729daa17587f12b47f249ff49baf1bd Author: Damian Johnson Date: Tue Sep 18 08:03:12 2012 -0700 Adding bwweightscale parameter constraints My spec fix to clarify the parameter was merged [1] so enforcing the default and constraints in our parser. [1] https://trac.torproject.org/6872 commit 802f96e94c81dc7270d1c8311485e3cd1bf2e10d Author: Damian Johnson Date: Mon Sep 17 08:55:53 2012 -0700 Parsing the bandwidth-wights attribute The 'bandwidth-wights' line is pretty similar to the 'params', so sharing most of the parsing code between them. Testing for the following... - negative and zero values - malformed entries - ordering - that this can't appear in a vote - missing values or empty content commit 53f16d8546302621f786c30ef0fb1b69786444fe Author: Damian Johnson Date: Sun Sep 16 18:10:08 2012 -0700 Noting that we might want to add 'bandwidth-weights' later The "bandwidth-weights" field mentions a 'bandwidth-weights' parameter and a default value. This isn't listed in the params section so getting confirmation of what I should be making of this param first. commit b3444ace427dca9a66a1afed1d24a09971612810 Author: Damian Johnson Date: Sun Sep 16 17:43:46 2012 -0700 Rejecting all footer fields in outdated consensus-method We were only checking that we lacked the mandatory footer fields when our consensus-method indicated that we shouldn't have a footer. Instead checking that we have no footer at all. This also makes the code a little nicer... commit 5c4a3ec4cb22fe0fff6c44f3eecf2a2639788ed6 Author: Damian Johnson Date: Sun Sep 16 17:07:03 2012 -0700 Parsing the directory-footer attribute For being an empty attribute this sure is a strangely big change. Checking the following... - that footers don't appear prior to consensus-method 9 - that the directory-footer lacks any content - that prior to consensus-method 9 we're happy to not have the line (bug I introduced because the footer has mandatory fields) commit 84fe0b46adc684a663af8536da35648adfd841e2 Author: Damian Johnson Date: Fri Sep 14 09:35:03 2012 -0700 Tidying up NetworkStatusDocument._parse() start Much of the _parse() method's start was unnecessary. Shortening it to improve readability. commit f60e60006a54ba2fc1eba0cb2fa5fade55b670ff Author: Damian Johnson Date: Wed Sep 12 09:14:30 2012 -0700 Validating params values and including defaults The 'params' line of a network status document has several known entries, with their own constraints on the value. Validating that the document obeys those constraints. Also, the path-spec has default values for a handfull of params so optionally defaulting our params attribute to that. commit 355e474b3d62dceb09699f1be27b5b27b925e7fa Author: Damian Johnson Date: Mon Sep 10 09:12:19 2012 -0700 Parsing the params attribute Being a fair bit more anal about field validation, checking the following... * bounds on the value to be an 32 bit int * order of the values (keys should be ascending) * reject values like '+11' (the int() function accepts them) This also includes unit tests for these use cases and better error messaging. commit 634c8f1d32dab7717b518593225b43b9e0349285 Author: Damian Johnson Date: Sun Sep 9 16:44:37 2012 -0700 Basic validation attribute testing The network status document's unit tests were mostly just checking invalid inputs. Changing the attribute unit tests to test some valid input too. commit 7218845d7bfce16b9ae4a0812cc74cd54b07f7c2 Author: Damian Johnson Date: Sun Sep 9 16:02:58 2012 -0700 Defaulting to consensus-method 1 As per the spec clarification in commit b829224, when the consensus-method or consensus-methods fields are missing we should default to method 1 (since in practice that's the only time the field doesn't appear). commit 0e4f8bfbdcf1bc9e427748dd37b53ad6b4a32d93 Author: Damian Johnson Date: Sun Sep 9 15:13:32 2012 -0700 Parsing the known-flags attribute Only changes are... * Moving the Flag enum to stem.descriptor.Flag since it's something that users will commonly use. As a rule of thumb I'd like users to only need 'stem.descriptor' unless they're doing something fancy. However, this is just a temporary move - I plan to move Flag to is own module, like Version. * Excluding empty strings from our known_flags attribute (our prior behavior was to parse things like " " into ['', '', '', '']). The spec doesn't set any constraints on what a flag can be so I suppose technically one could be the empty string, though this'll never be the case. Still waffling back and forth about if this should cause a validation error or not... commit fd0deb96f30fc691019e0d2a09068a4787d80e97 Author: Damian Johnson Date: Sat Sep 8 19:11:13 2012 -0700 Parsing tor version list attributes Handling the 'client-versions' and 'server-versions' lines, comma separated lists of tor versions. commit fb808fde025a8c4f7f04d243f6e679f2f8b5e255 Author: Damian Johnson Date: Sat Sep 8 18:52:58 2012 -0700 Parsing the voting-delay attribute Nothing special here. Simply a pair of integer fields. commit cb96c3e094719b096e5425faeadc01a15dfcb388 Author: Damian Johnson Date: Sat Sep 8 18:42:29 2012 -0700 Parsing consensus-method and datetime attributes Handling the published, valid-after, fresh-until, and valid-until fields. Doing them as a batch since they're all simply datetime fields without any other constraints. commit 65e92e80711999591441abbdf23a21321e4d9271 Author: Damian Johnson Date: Sat Sep 8 18:18:38 2012 -0700 Parsing consensus-methods attribute Only noteworthy bit is that the spec says 'Method "1" MUST be included.' so checking for that. commit b931e980b62600d5cc9cd60069102c2ffa26cb84 Author: Damian Johnson Date: Sat Sep 8 12:35:01 2012 -0700 Parsing vote-status attribute Changing our 'vote_status' string attribute to 'is_vote' and 'is_consensus' boolean attributes. The spec specifically says that anything else is invalid so there's little reason to allow arbitrary content in the field. commit 5e12da0a50ee60577f3a571eea0ef3350e146d10 Author: Damian Johnson Date: Sat Sep 8 12:15:34 2012 -0700 Parsing network-status-version attribute I don't like our assumption that this is a v3 network status document, but I'll need to think more about how to handle other versions later. For now keeping the validation assertion that parsed documents are v3. commit db87c88965806ec13f9888c220879af45fc65e44 Author: Damian Johnson Date: Sat Sep 8 11:54:33 2012 -0700 Validating that fields aren't duplicated Almost all of the fields in a network status document can only appear once. Checking that this is a case. This also includes a unit test for the validation. I'm a little surprised that the old parser doesn't choke on duplicated content, but I'm about to replace it so probably not worth investigating. commit f23c5312b671dd2bd41baa5a0c24568592ac3241 Author: Damian Johnson Date: Sat Sep 8 11:23:08 2012 -0700 Validating the order of fields in documents As mentioned in the prior commit, to be valid network status fields need to appear in a particular order. Checking for this as part of document validation. commit bc09f1e697f4a34475b4b9b55bcb91c14494427b Author: Damian Johnson Date: Sat Sep 8 10:51:51 2012 -0700 Unit test for misordered fields To be valid a network status document's fiends need to appear in a particular order. I'm about to add vaidation for this, but the parser is being finicky. I'm tired of trying to fix incrementally fix it, so skipping this test for now and moving on to the parser rewrite. commit 725e2f9deb1a22b5cfca0242ea79d5c0548a40e3 Author: Damian Johnson Date: Sat Sep 8 10:26:48 2012 -0700 Unit tests for minimal vote and missing fields Unit tests for a couple important use cases and lots 'o fixes for the issues they uncovered. As mentioned earlier the 'validate' attribute took the wrong meaning in this parser so valid content errors and invalid content triggers stacktraces. commit 7b38323ee41aa890f5b777eff4bd365d5a6c1fb1 Author: Damian Johnson Date: Thu Sep 6 09:29:25 2012 -0700 Splitting up entry and document unit tests Splitting up unit tests for the RouterStatusEntry and NetworkStatusDocuemnt. Both will be pretty lengthy and there's little reason to try to mash them together. commit cf475d58dfa4d42e982eca6307e2a61e7545147e Author: Damian Johnson Date: Thu Sep 6 09:14:09 2012 -0700 Minimal unit test for network status documents Adding a unit test for the minimal valid network status document (plus a consensus-method field since that influences validation). This uncovered some bugs with the NetworkStatusDocument class... * The network_status_version field misdocumented as being an int (it was actually a str). We need it to be a str for microdescriptors so simply changed the pydoc. * The consensus-method and bandwidth-weights are documented in the spec as being optional fields. The parser errored with a stacktrace when consensus-method was missing, and gave a validation error if there isn't a bandwidth-weights. * Inappropriate validation error if there was unrecognized content. * The get_unrecognized_lines() method is documented as providing a list of lines. The NetworkStatusDocument returned a string instead. * Off-by-one error that caused consensus-method 9 documents to skip parsing footers. commit d30a628e5da06699d8533f92c4b1c496510213b8 Author: Damian Johnson Date: Sun Sep 2 17:06:20 2012 -0700 Checking for mandatory and disallowed fields There's several restrictions on a valid network status document, some of which are which fields it does and does not contain. Validating that manitory fields apprear, and that fields which shouldn't appear don't. This also drops my _get_entries() helper function in favor of _get_descriptor_components() since we're now dealing with documents which can contain key blocks. There wasn't really enough of an advantage to _get_entries() to justify the duplicate code anyway. Also fixing the _get_descriptor_components() I wrote so it return footer content in the header when the content doens't contain any 'r' lines. This currently keeps the old parser (renamed to "_parse_old()") since I haven't addressed any of the actual parsing yet. commit 239d9642bfc800b4f720880f359cdc92a713e63f Author: Damian Johnson Date: Tue Aug 21 16:52:56 2012 -0700 RouterStatusEntry unit tests Unit tests for the RouterStatusEntry use cases that come to mind. As normal they uncovered some bugs with the class. commit dc15e682be37ea450a9b5e45f05958f676eabded Author: Damian Johnson Date: Mon Aug 20 22:58:11 2012 -0700 Setting zero ports to None in server descriptors When writing the server descriptor parser I faithfully translated zero ports as zero because I wanted users to be able to differentiate between 'malformed' and 'zero' when validation was disabled. However, on reflection I don't really care about catoring to non-validation use cases. Ravi made the router status entries interpret a zero port as being None which makes perfect sense - zero means that the port doesn't exist. Doing this for the server descriptors too. commit be0c484de0be3cfa98a32c9ef5210b3e1d6eb6ee Author: Damian Johnson Date: Mon Aug 20 22:53:17 2012 -0700 Unit test for minimal RouterStatusEntry As usual, starting the class unit testing by parsing a minimal example. Next gonna go through field by field to exercise interesting use cases. commit a33297c8f65552c1ef721277e056a89f2b2727b7 Author: Damian Johnson Date: Mon Aug 20 22:20:44 2012 -0700 Rewriting RouterStatusEntry parser The networkstatus module has spotty error checking, for instance... * the unrecognized_lines attribute is never initialized, so anything touching it will trigger a NameError * missing values on an 'r' line will result in an IndexError, and extra values are ignored * malformed nicknames, ip addresses, and ports go undetected * non-integer ports or bandwidth/measured values give an 'invalid literal' error, which isn't terribly helpful * extra entries on a 'w' line inappropriately cause an error * malformed exit policies can cause an error when validation is disabled * we error if there's new keywords (the spec doesn't seem to say they're disallowed...) From the look of this code the 'validate' flag seems to have been misinterpreted as meaning "raise an exception if we run into something unexpected, like a new keyword or flag". This is wrong. Stem should *never* error when processing spec conformant content. Rather, the 'validate' flag means "raise an exception when we parse something that violates the spec". When that flag is False the parser should *never* raise an exception, making a best-effort attempt to parse even malformed content. I'm a bit sad to see that there are absolutely no unit tests for this module. That was the bulk of the testing for the server_descriptor and extrainfo_descriptor modules. Guess I'll write that next. commit 0752e699fcfb657eb27ed97e0b188376cd9368a1 Author: Damian Johnson Date: Sun Aug 19 21:47:06 2012 -0700 Dropping unknown_flags attribute I suggested an unknown_flags attribute that would have flags which stem didn't recognize but, on reflection, this wasn't a very good idea. Splitting the attributes mean that users have multiple places to look for the relay's flags. We were also failing validation if there were any unknown flags and that is definitely wrong. New flags are perfectly valid and will happen in the future - those new flags should not break stem's parser. This was probably confusion from when I asked for the cached-consensus test to check for new flags. This is something separate and added it with this change. commit b1a1674cd5df2b5443699738b6880c6a0077cf0f Author: Damian Johnson Date: Sun Aug 19 21:31:49 2012 -0700 Dropping known_flags arg from RouterStatusEntry The RouterStatusEntry accepts a known_flags argument and I'm not sure why. It's not used, and the known/unknown flag distinction is simply based on if stem knows about it (and hence is in the Flag enum) so I'm not sure how or why users would utilize it. commit dd071270d705507f0131bfceac7358f7223ec32d Author: Damian Johnson Date: Sun Aug 19 21:25:59 2012 -0700 Asserting that _decode_fingerprint() output is a fingerprint Checking that the _decode_fingerprint() function returns a fingerprint, throwing a ValueError if it doesn't. Also adding some tests for this. commit 2b4ec7c1c6e1e1848ca3080d9da41cb02d51b0ea Author: Damian Johnson Date: Sun Aug 19 18:11:49 2012 -0700 Error due to KEYWORD_LINE regex change The KEYWORD_LINE regex fix a few commits back broke one of the server descriptor unit tests because it caused lines with only a keyword (ex. "platform") to have a None value. Before the regex change they had an empty string value. I'm not really sure which is proper, but for now keeping the prior behavior. commit fd58bc5a565bca11d372188a4add167775f091c1 Author: Damian Johnson Date: Sun Aug 19 17:55:25 2012 -0700 Replacing RouterStatusEntry's identity with fingerprint Truncated base64 encoding of a relay's fingerprint? Not useful. Common hex encoded fingerprint that's used everywhere else? Very useful. Decoding the RouterStatusEntry's identity value and adding a unit test for it. Unfortunately the networkstatus module didn't have any unit tests at all so this is gonna take a while to remedy... commit 393c4de2817a27c1cbdaa7bdf881e9cd08e61a76 Author: Damian Johnson Date: Sun Aug 19 16:35:42 2012 -0700 Renaming RouterDescriptor to RouterStatusEntry The dir-spec calls these "router status entries", so using that name in stem too. commit f9a2ef8d0dd60a73c7cfe5a329b20329c4b4218c Author: Damian Johnson Date: Sun Aug 19 11:54:29 2012 -0700 Hybrid approach between iterator and attribute Ok, changed my mind. At least partly. If the user is using parse_file() then we want to provide them with router entries and a bare document, like I was. However, if they're calling the document constructors then they've already read the complete document into memory so there's little reason not to parse it. In short, documents now have a 'routers' attribute. That attribute is filled with the parsed routers if we use the constructor directly, and left empty if we used parse_file(). commit 35c5eaf2933dad64bab61cc65657d60db276a54a Author: Damian Johnson Date: Sun Aug 19 00:21:08 2012 -0700 Inverting network status' document and router relationship This code was trying very hard to produce a network status document that contained router entries. This is right and proper from an object oriented standpoint, but not from a practical one. The document cannot contain the router entries unless we keep them all in memory, and for the vast majority of users that is not desirable. In later revisions Ravi addressed the memory concerns by turning the router_descriptors attribute into an iterator. This works, but it's confusing and the router_descriptors' itertor only works as long as we keep the file open (or buffer the content). Instead, inverting the relationship so this module provides an iterator of routers and those routers have a reference to the document they came from. As a side benefit this lets us tidy up the code a bit. There like are some use cases where we would prefer a document object that contains routers, so I'll probably add that option too later. This also gets rid of the "Flavour" enum. I'm not entirely sure what it was for - it wasn't documented and its NS value was unused. Maybe it was vestigial from some prior changes. commit e1ce38ebd4e49faff993e212e6959f9a46e93e50 Author: Damian Johnson Date: Sat Aug 18 19:08:15 2012 -0700 Removing _skip_until_keywords() The _skip_until_keywords() is almost entirely a duplicate of _read_until_keywords(), its only difference being that it doesn't return the content. Adding an argument to _read_until_keywords() to do the same. commit 756625453f5ac0ff9415d4cb04bc373c70bf79ef Author: Damian Johnson Date: Sat Aug 18 18:53:07 2012 -0700 Keyword parsing broken for tab dividers Most descriptor types (server/extrainfo descriptors, V2 network status documents, etc) have their divider whitespace defined as one or more spaces or tabs. God knows why they chose that - there's no good reason for outputing that and it makes parsing a pita. An exception is V3 network status documents, which redefine SP as being a single space. I've never seen this in the wild so this is mostly an academic concern, but I'm trying very, very hard to make stem spec conformant. The _read_until_keywords() and _skip_until_keywords() functions only worked if the keyword was divided by a space, not a tab. I sunk two hours into fixing a bug with the KEYWORD_LINE regex which caused it to match against signature content. Long story short, we were accidently fuzzing our parser by letting there be a 'keyword' and 'value' when there was no whitespace between the two. commit 7927579282e6b6a3071dc1901c0a725a94fa7563 Author: Damian Johnson Date: Sat Aug 18 16:45:43 2012 -0700 Iterating over routers with parse_file() Having parse_file() iterate over the RouterDescriptors in a consensus rather than providing the document itself. 99.9% of the time our callers want to do something with the routers, and providing them a document instead just forces them to unwrap it. I'm thinking about changing the networkstatus module to provide RouterDescriptors that reference the NetworkStatusDocument they came from rather than the other way around. We'll see how it works. commit 21e5a4cd578dc81496d8355e377048c3707488b0 Author: Damian Johnson Date: Sat Aug 18 16:30:03 2012 -0700 Combining TestNetworkStatusDocument and TestMicrodescriptorConsensus Having separate test classes was a fine idea, but resulted in our tests claiming that 'test.integ.descriptor.networkstatus' ran twice. That's confusing, so we either need to put them in separate files or merge the classes. Opting for the later for now since TestMicrodescriptorConsensus only had one test. commit a85e6f77ee1b4ba40e86bd148062bf8e479484e1 Author: Damian Johnson Date: Sat Aug 18 16:12:35 2012 -0700 Raising TypeError when metrics header is unrecognized When we get a metrics header for a format we don't recognize the _parse_metrics_file() function is supposed to throw a TypeError (and says it does in a comment). However, it didn't. This caused unrecognized metrics types to be parsed like a valid descriptor file containing zero descriptors. Fixing and adding a test for this use case. Also fixing a couple other minor bugs I ran into while fixing this: * Some tests had an assertTrue() rather than assertEqual(), causing the assertions to almost always evaluate to True. * The UnrecognizedType we threw for text files didn't have its mime type set. commit f1ecbde2b5f00019ec171c44628c03c2f09e1952 Author: Ravi Chandra Padmala Date: Thu Aug 16 18:18:09 2012 +0530 stop skipping unexpected "opt" lines commit 6c3717b65acc9d208ef3bf90b5b54f3983e507df Author: Ravi Chandra Padmala Date: Wed Aug 15 20:47:29 2012 +0530 Fixes to document parsing One major change is that stem.descriptor.networkstatus.parse_file now returns a NetworkStatusDocument object instead of iterating over the router descriptors in the document commit 8ad310114b1ea7b743a868a8b70832eea5b8f3e2 Author: Ravi Chandra Padmala Date: Fri Aug 10 17:49:23 2012 +0530 Add microdescriptor parsing commit dcf2092607c8be29d065def1cb104290709bef87 Author: Ravi Chandra Padmala Date: Thu Aug 9 17:21:09 2012 +0530 RouterDescriptor objects store a list of flags commit 9ad3395400cd21a26c0106e665aa6e1e2026a353 Author: Ravi Chandra Padmala Date: Thu Aug 9 13:01:44 2012 +0530 Removed DescriptorParser commit eb0e424ed9459188b70f33ff401e23e9fd89138b Author: Ravi Chandra Padmala Date: Wed Aug 8 12:39:55 2012 +0530 Implement lazy router descriptor reading commit 1b26a46ed3c2b26293474d9349efb247f5888a65 Author: Ravi Chandra Padmala Date: Wed Aug 8 08:33:48 2012 +0530 Extend _read_until_keyword to read until multiple keywords commit 5f61f4fe91014dbe44e5acc8facd69e0c3467797 Author: Ravi Chandra Padmala Date: Tue Aug 7 21:39:57 2012 +0530 Remove unnecessary imports commit cba98e26f16214925f7f5e6d61b51767da32b66e Author: Ravi Chandra Padmala Date: Tue Aug 7 21:24:30 2012 +0530 Fix incorrect stem.descriptor.networkstatus import commit e7cc83b4e382d209dda4660b5d30b9691f3fd713 Author: Ravi Chandra Padmala Date: Tue Aug 7 13:48:24 2012 +0530 More documentation commit 9e6fb6d9d99144832b78e5cf1b28fc4232d17c15 Author: Ravi Chandra Padmala Date: Tue Aug 7 13:25:43 2012 +0530 Fix incorrect docstring and reduce 1 LoC elsewhere commit 92f46907b3d84c53ab10c8f19851d65be86f7b9f Author: Ravi Chandra Padmala Date: Tue Aug 7 13:19:57 2012 +0530 stem.descriptor.networkstatus.KeyCertificate => stem.descriptor.KeyCertificate From section 4.7 the dir-spec, A concatenated set of all the current key certificates should be available at: http:///tor/keys/all.z The key certificate for this server (if it is an authority) should be available at: http:///tor/keys/authority.z ... we might want to parse these seperately at some point, so, moving it out of stem.descriptor.networkstatus commit a288e1e43f2cecc55b1b46cbfab6734aee1a5395 Author: Ravi Chandra Padmala Date: Tue Aug 7 13:10:49 2012 +0530 Update comments commit 9df15326fecd223bf7d4e5a40465a2fa80069025 Author: Ravi Chandra Padmala Date: Tue Aug 7 15:01:49 2012 +0530 Remove trivial comment commit 847d1caec38e110f7d75ae1db526a85803094d13 Author: Ravi Chandra Padmala Date: Tue Aug 7 14:43:42 2012 +0530 Fix cached-consensus parsing commit 21cfed42b859becabe71a803164004fea2372929 Author: Ravi Chandra Padmala Date: Tue Aug 7 14:14:51 2012 +0530 Add support for votes from Metrics commit 4e29ae8dfc69cb68a25c9cf159e56db4e943565b Author: Ravi Chandra Padmala Date: Tue Aug 7 12:09:25 2012 +0530 stem.descriptor.networkstatus_descriptor => stem.descriptor.networkstatus commit 1ed53535e19b1c065dfc9c4589c31283cdac7649 Author: Ravi Chandra Padmala Date: Tue Aug 7 12:03:05 2012 +0530 Initialize instance variables commit f955d14f25041846cdc82a8e59fe704378a6f6ac Author: Ravi Chandra Padmala Date: Tue Aug 7 12:02:03 2012 +0530 Fix version parsing in NetworkStatusDocument commit cc0c897908bb0d1bd97672675badb1286d71bfbc Author: Ravi Chandra Padmala Date: Tue Aug 7 11:58:33 2012 +0530 Fix DescriptorParser.remaining commit 789145ee6ca6dea2617cdd641771cbce1b50296a Author: Ravi Chandra Padmala Date: Tue Aug 7 10:16:43 2012 +0530 Add tests for vote and consensus documents commit 4d9a7a38040add8a3e48254ddffd9f32189ccb26 Author: Ravi Chandra Padmala Date: Mon Aug 6 14:08:26 2012 +0530 Add integ tests for networkstatus_descriptor commit 4a58b7a9fb30584f70ade4a6d75bc8f9821e57d0 Author: Ravi Chandra Padmala Date: Mon Aug 6 14:07:25 2012 +0530 networkstatus_descriptor.parse_file iterates over RouterDescriptors in a document commit 1cf9a62f46d9b9c4ffcbae90dbb59abaa66ff3c8 Author: Ravi Chandra Padmala Date: Mon Aug 6 14:05:29 2012 +0530 Fix DescriptorParser and make parse_file iterate over router descriptors commit fc1ad3f21f8d23233551514dc87524a40697621d Author: Ravi Chandra Padmala Date: Mon Aug 6 05:28:41 2012 +0530 match Measured instead of Measured= commit fb710843f6f21ec1779d80462a1a943594623281 Author: Ravi Chandra Padmala Date: Mon Aug 6 05:16:56 2012 +0530 Fix documentation formatting stem.descriptor.server_descriptor's module overview is formatted incorrectly. (see stem.socket) commit 8ed384d31aa446dd3c5954dd4353e059657ed7a3 Author: Ravi Chandra Padmala Date: Mon Aug 6 05:14:51 2012 +0530 Move DocumentParser to stem.descriptor.DescriptorParser commit a9f35d5e86bffc4b04cb525736a23a7e72009efc Author: Ravi Chandra Padmala Date: Thu Jul 19 16:46:09 2012 +0200 Add support for network status parsing commit 5cd457bebedb6e1582ce7d0d7782df80e51a7291 Author: Damian Johnson Date: Fri Oct 12 08:49:56 2012 -0700 Cleaning up orphaned *.pyc files A common gotcha when deleting or renaming files is that the python bytecode can still be imported and used without the source file. In other words if you delete 'foo.py' but forget to also delete 'foo.pyc' then your 'import foo' statements will still work. This has often bitten us because the tests still pass, so the problem goes undiscovered until someone new checks out the repository. Having stem's test runner check for orphaned pyc files and remove them prior to running its tests. Idea suggested by Ravi. https://trac.torproject.org/7103 commit 705b61674e8cec9e5608a32c6a50a708fee8c5f7 Author: Damian Johnson Date: Tue Sep 25 09:51:26 2012 -0700 Running apidoc when being built by read-the-docs The read-the-docs service runs sphinx-build, bypassing our custom Makefile. The only gotcha with this is that it means we don't call apidoc, causing our autogenerated documentation (which is... well, just about everything) to not exist. Making the apidoc in the conf.py file, using the READTHEDOCS environment variable to deterine if we're on that platform or not. Tested by first running a regular sphinx build to repro the apidoc not being called... sphinx-build -b html -d _build/doctrees . _build/html ... then did it again after setting READTHEDOCS to confirm that the autodocs now existed. commit 66dea047a651d595b755c754d84250cd1dd687ac Merge: 56a06b6 5a0da76 Author: Damian Johnson Date: Mon Sep 24 09:01:38 2012 -0700 Adding a method for MAPADDRESS Method to map address aliases. cr atagar (https://trac.torproject.org/6951) commit 5a0da767e086aab1f196f60ed9fd042eb6825db4 Author: Damian Johnson Date: Mon Sep 24 09:00:08 2012 -0700 Couple minor fixes for tests Correcting an ImportError due to test.util being moved, and preventing the 'is valid IP' check from being a no-op. commit ddf7aa7a21bfc8418488b27e05fa7e53ef4f514c Author: Ravi Chandra Padmala Date: Mon Sep 24 20:16:18 2012 +0530 Fixing a bunch of things after cr #6951 https://trac.torproject.org/projects/tor/ticket/6951 commit 060c82b3a874c88159ee4a20803c93f22512851a Author: Ravi Chandra Padmala Date: Sun Sep 23 11:45:33 2012 +0530 Fix a typo commit 8e54f37f64ea04a0772d825276f4e5e4a34d4df2 Author: Ravi Chandra Padmala Date: Wed Sep 5 18:29:10 2012 +0530 Implement Controller.map_address commit 56a06b6283d5bcd3c5eb6d6c85b98b2949858cf9 Author: Ramkumar Ramachandra Date: Thu Sep 13 12:27:52 2012 -0700 test/unit: correct a typo in rule.py A tuple was being initialized as a dictionary. Correct this. Signed-off-by: Ramkumar Ramachandra commit 223ebc5617ce46007dfd5a9210e4ff91fb9767ea Merge: 3df65c0 d97a0c3 Author: Damian Johnson Date: Mon Aug 27 11:12:59 2012 -0700 Adding a method for SETCIRCUITPURPOSE cr atagar (https://trac.torproject.org/6670) commit d97a0c30002062ac0fbc45b32c2be6061cc35594 Author: Damian Johnson Date: Mon Aug 27 11:08:10 2012 -0700 Skipping repurpose_circuit() test if offline The repurpose_circuit() test uses 'GETINFO circuit-status' results which are empty when offline, causing an error... ====================================================================== ERROR: test_repurpose_circuit ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/integ/control/controller.py", line 386, in test_repurpose_circuit first_circ = controller.get_info('circuit-status').splitlines()[0].split() IndexError: list index out of range ---------------------------------------------------------------------- Ran 14 tests in 1.390s commit d27fbbb06a247d3fb12684f76f2a7f5a51a36b04 Author: Ravi Chandra Padmala Date: Mon Aug 27 11:02:48 2012 -0700 Implement Controller.repurpose_circuit commit 3df65c080072d11d77664e2738652f3c0c08134e Merge: 4f6a16c 74ba2e9 Author: Damian Johnson Date: Fri Aug 24 12:51:00 2012 -0700 Adding methods for SIGNAL and EXTENDCIRCUIT Feature additions by Ravi. cr atagar (https://trac.torproject.org/6659 and https://trac.torproject.org/6666) commit 74ba2e9db42508a2bc2873dd078ae5bf2cbd140e Author: Damian Johnson Date: Fri Aug 24 12:38:20 2012 -0700 Revisions for EXTENDCIRCUIT addition Couple minor changes: * The EXTENDCIRCUIT integ tests fails if we don't have an internet connection. Making it dependent on the ONLINE testing target. * Revising the pydocs a bit. For instance, there's no point in saying "If the purpose isn't provided, "general" circuits are built." when we can simply provide a default for the argument. ;) However, there's still a couple mysteries... * When I was offline I got the following stacktrace... ====================================================================== ERROR: test_extendcircuit ---------------------------------------------------------------------- Traceback: File "/home/atagar/Desktop/stem/test/integ/control/controller.py", line 373, in test_extendcircuit circ_id = controller.extend_circuit(0) File "/home/atagar/Desktop/stem/stem/control.py", line 1109, in extend_circuit raise stem.socket.ProtocolError("EXTENDCIRCUIT returned unexpected response code: %s" % response.code) ProtocolError: EXTENDCIRCUIT returned unexpected response code: 512 ---------------------------------------------------------------------- However, according to the control-spec the 512 response code is for "Syntax error in command argument". That doesn't make sense if we're failing because we lack a connection. * Is the 'path' argument for EXTENDCIRCUIT the circuits that we build through, or the relays to be chosen from when building a single hop? The EXTENDCIRCUIT description doesn't say, which seems to me to be a weakness in the spec. commit fea0a43ded650b78d9453c1bc2ce15fdb99e394f Author: Ravi Chandra Padmala Date: Thu Aug 23 22:07:01 2012 -0700 Implement Controller.extend_circuit commit 4f8b48ba3a7ee34a5725c9df242a961da9e7da01 Author: Damian Johnson Date: Thu Aug 23 22:01:31 2012 -0700 Collapsing SIGNAL documentation Shortening the pydocs for the signal() method. Might as well list both signal aliases on the same line. commit aeb39b813961d02509a9c3e4e3db585b4190cac1 Author: Ravi Chandra Padmala Date: Thu Aug 23 21:52:01 2012 -0700 Implement wrapper method for Controller to handle SIGNAL requests commit 4f6a16c9526b45d3772abb797df2771bed6f884f Merge: 92a0bf0 6d1f362 Author: Damian Johnson Date: Thu Aug 23 21:44:33 2012 -0700 Adding a method for USEFEATURE Methods to enable a feature and check if a feature is on. cr atagar (https://trac.torproject.org/6417) commit 6d1f3625710eb382a810907a49b481ca14762783 Author: Damian Johnson Date: Thu Aug 23 21:32:36 2012 -0700 Minor revisions for USEFEATURE Few very minor tweaks... * 'GETINFO orconn-status' provides empty results when we don't have a connection, causing the test to fail. * Making is_feature_enabled() case insensitive. * Short circuiting is_feature_enabled() checks if it's in enabled_features. * The enabled_features attribute is simply used for an existance check, so we don't need to be too careful about deduplicaion. That said, we *do* need to be careful about the case of its entries. * Renamed the version constants since I'm trying to list their category at the start, to make a sort of namespace. commit 51a00e9a3a2f2fa10277759616582d4acd2942a3 Author: Ravi Chandra Padmala Date: Thu Aug 23 00:05:10 2012 +0530 Fix documentation formatting commit 17d536657c43984665138ee131e77a6a291176d5 Author: Ravi Chandra Padmala Date: Wed Aug 22 23:45:49 2012 +0530 Add Controller.is_feature_enabled Also made some minor modifications based on CR in #6417 commit 9a70131cf2bf0b25f89faa89c62fb70d5e891ee1 Author: Ravi Chandra Padmala Date: Tue Jul 3 01:34:46 2012 +0530 Implement use_feature wrapper method commit 92a0bf0d6afe39db45108626d2e89cde70997a2a Merge: ac5be8c 683eeb2 Author: Damian Johnson Date: Tue Aug 21 18:11:11 2012 -0700 Caching expansion and test prompt I was planning to simply add caching for GETCONF and SETCONF requests, but ended up making a few other changes on the way. The most prominent is a new 'prompt' script that you can use to quickly get python interpretor with a controller instance. Handy for testing. cr neena (https://trac.torproject.org/6631) commit 683eeb25e812346cfd01790fe7729b686b8a2c2c Author: Damian Johnson Date: Tue Aug 21 18:10:30 2012 -0700 Fixing issues spotted by Ravi Typos and whitespace issues spotted by Ravi. commit 75d7f077e4b23361bc163171a4ec38b137f019ee Author: Damian Johnson Date: Sat Aug 18 11:36:38 2012 -0700 Setting string attribute of OperationFailed All OperationFailed subclasses had an empty string attribute, making stack traces pretty unhelpful. Before: >>> control.get_info('blarg') Traceback (most recent call last): File "", line 1, in File "stem/control.py", line 622, in get_info if default == UNDEFINED: raise exc stem.socket.InvalidArguments After: >>> control.get_info('blarg') Traceback (most recent call last): File "", line 1, in File "stem/control.py", line 622, in get_info if default == UNDEFINED: raise exc stem.socket.InvalidArguments: GETINFO request contained unrecognized keywords: blarg commit f999b627dc3ab00cdd7c245d4811ad7b0a2cc20c Author: Damian Johnson Date: Sat Aug 18 11:21:08 2012 -0700 Adding an easy method for getting a controller instance Adding a 'prompt' script that kicks off a python interpretor with a readily available stem.control.Controller instance. This starts a tor test instance if it isn't already running and offers to shut it down when done. The purpose of this script is to make it easy to get a controller instance we can exercise. The sh shebang makes this *nix specific, though I suppose windows users can still use it by running 'bash prompt'. Example usage: atagar@morrigan:~/Desktop/stem$ ./prompt Welcome to stem's testing prompt. You currently have a controller available via the 'control' variable. >>> control.get_info('version') '0.2.1.30' >>> quit() Would you like to stop the tor instance we made? (y/n, default: n): y commit a6c5a063d1f963a53e012b4c31478f90dea0d795 Author: Damian Johnson Date: Fri Aug 17 10:24:16 2012 -0700 Better validation of controller's from_port() function I scratched my head due to an awefully misleading stacktrace when I passed a port value into the method's argument slot. Validating the inputs to avoid having this cause confusion in the future. Before: Traceback (most recent call last): File "", line 1, in File "test/prompt.py", line 67, in controller controller = stem.control.Controller.from_port(CONTROL_PORT) File "stem/control.py", line 457, in from_port control_port = stem.socket.ControlPort(control_addr, control_port) File "stem/socket.py", line 287, in __init__ if connect: self.connect() File "stem/socket.py", line 171, in connect self._socket = self._make_socket() File "stem/socket.py", line 311, in _make_socket control_socket.connect((self._control_addr, str(self._control_port))) File "/usr/lib/python2.7/socket.py", line 224, in meth return getattr(self._sock,name)(*args) TypeError: coercing to Unicode: need string or buffer, int found After: Traceback (most recent call last): File "", line 1, in File "test/prompt.py", line 67, in controller controller = stem.control.Controller.from_port(CONTROL_PORT) File "stem/control.py", line 459, in from_port raise ValueError("Invalid ip address: %s" % control_addr) ValueError: Invalid ip address: 2779 commit fc736ff07c2ded068ee3e5b9ad1028923ee89c47 Author: Damian Johnson Date: Fri Aug 17 10:07:55 2012 -0700 Fixing circular import Correcting a circular import between the connection and control modules... >>> import test.prompt Traceback (most recent call last): File "", line 1, in File "test/prompt.py", line 15, in import stem.control File "stem/control.py", line 48, in import stem.connection File "stem/connection.py", line 106, in def connect_port(control_addr = "127.0.0.1", control_port = 9051, password = None, chroot_path = None, controller = stem.control.Controller): AttributeError: 'module' object has no attribute 'control' commit c87b1005ad045b4554c3ed4c2ba6ce6e54aa81dc Author: Damian Johnson Date: Mon Aug 6 10:04:40 2012 -0700 Caching and logging for GETCONF/SETCONF Caching and logging in a similar fashion to GETINFO. Remaining todo items... * This is broken if another controller tampers with our configuration values. We should listen for configuration change events to catch that. * Calling 'GETCONF blarg' causes an exception without a message. I should figure out why. commit 2a34547cc6dc157617c4a485d7e9e41816d46e66 Author: Damian Johnson Date: Mon Aug 6 08:45:07 2012 -0700 Caching parsed controller version Our controller version is something that we'll frequently look up. There's no reason to leave it uncached. commit 9fe8da962aadec5d56bfeb6d6635a01228514ca2 Author: Damian Johnson Date: Mon Aug 6 08:24:16 2012 -0700 Pydoc mistake Actually, I'm not sure what I was trying to say there. Issue spotted by Karsten. commit ac5be8cee5f58517c09217fd6860f7ef483dacc2 Author: Damian Johnson Date: Sun Aug 5 20:14:18 2012 -0700 Fully parsing transport lines The extrainfo descriptor's new transport lines were a little funny in that they're only relevant to bridges, and when they apprear in bridges they're scrubbed of everything except the transport name. This in turn meant that they only appeared with transport names in the wild. However, now that transport lines can appear in non-bridge relays I'll sometimes see complete transport lines, so parsing and validating them properly. commit 3f380f95876baffe48725828c23706dd9b46b760 Author: Damian Johnson Date: Sat Aug 4 19:49:39 2012 -0700 Parsing transport lines in non-bridge relays Spotted a relay in the wild with a transport line and asn thinks that, though it's an oddity and doesn't make sense for non-bridges, this isn't a bug... extra-info toorvoid 0B37323298FF98CD86ED404895BB27B7426E8AE1 published 2012-08-04 12:19:13 transport obfs2 83.212.96.201:33570 Expanding the parser to allow for it in any extrainfo descriptor. commit 73fcfaf8ccbc2c1cb714f35fcc8e2823daf39808 Author: Damian Johnson Date: Sat Aug 4 18:52:33 2012 -0700 GETINFO caching and logging Caching static GETINFO results, short circuiting geoip lookups when the geoip db is unavailable, and logging the call runtimes. commit f7157ae4f576acb1b90e458f479e2a820aefbe86 Merge: 051ed27 7022021 Author: Damian Johnson Date: Sat Aug 4 12:18:56 2012 -0700 CSV export functionality for tor descriptors commit 7022021c4207a0066cb93261629f2ba020d307f6 Author: Damian Johnson Date: Sat Aug 4 12:13:47 2012 -0700 Revising descriptor exporter and associated tests Several changes for the prior csv export functionality, especially its tests. Most notably this includes... * rewrite of the tests to be simpler and break down by test cases * newline breaks rather than windows style '\r\n' * excluding private attributes (attributes that start with an underscore) * writing directly to a file in export_csv_file() rather than buffering to a string first * general refactoring to hopefully improve readability commit ff43d9e7ad306499dce6e20909d0ce2e842541b4 Author: Damian Johnson Date: Sat Aug 4 10:48:06 2012 -0700 Moving descriptor contructor to mocking module We'll need to construct server descriptors in both their unit tests and when testing the exporter. The mocking module is both for function mocking and creating mock objects so this is the perfect place for it. I might move the extrainfo descriptor constructer here later if it seems useful. Personally I'm finding the current exporter tests and mocking changes very confusing so reverting the mocking changes and writing some simpler tests. commit bb0dcfc7fdf9303251dbe7304005d349f9956fd6 Author: Erik Date: Wed Aug 1 16:40:56 2012 -0400 Submitting for code review. commit e94e00a61d262fd52706c0891276811ca0369e52 Author: Erik Date: Tue Jul 31 12:38:27 2012 -0400 minor documentation update commit aafb7b429e7220c938da7a591d17ca1b2463aa09 Author: Erik Date: Tue Jul 31 12:35:09 2012 -0400 First complete build of export.py tests. Everything should pass, and testing export_csv_file() promted some changes, which include the change from specifying a file for writing to providing a document object. This code is now ready for review. commit 9440f4ffa20aa7ba2a0e3297cbf5b245bf2c1c91 Author: Erik Date: Mon Jul 30 18:09:13 2012 -0400 First complete build of unit tests for torexport. There is a bug in test_export_csv_file which will be fixed tomorrow morning, otherwise all tests are functional and should be passing. commit 4ea5cf2e2d8f7bb27db96d35d8fc0538b2745e70 Author: Erik Date: Tue Jul 17 17:46:16 2012 -0400 Early unit test builds with lots of holes to fill. Problems encountered include mocking functions that utilize keyword args, passing descriptor objects to functions being tested rather than just dictionaries, and determining if, in the case of the user defining both include_fields and exclude_fields, an error should be thrown or the functions should simply remove any overlap from the two lists. commit 305f7fdb30f420a751f2de29f250170c19569bb6 Author: Erik Date: Thu Jul 12 15:24:50 2012 -0400 Updated some of Damian's suggested revisions. Not sure of others. commit 508373f1a7824b1ff9b86a597a718918d4250e24 Author: Erik Date: Thu Jul 12 14:13:59 2012 -0400 First draft of torexport, 3 functions defined to handle different use cases. First draft of torexport, 3 functions defined to handle different use cases. csv_file_exp() takes a list of descriptors and exports their attributes as a csv file. descriptors_csv_exp() takes a list of descriptors and returns a generator. Each iteration, this function returns a line of csv representing the attributes of a single descriptor object. descriptor_csv_exp() returns a single line of csv based on the attributes of a sole descriptor object. commit 051ed27b99a5b8796777b86360a41bec52527b12 Author: Damian Johnson Date: Tue Jul 24 08:42:56 2012 -0700 Descriptor pydoc mistake The parse_file() helper no longer runs over bridge descriptors, just relays. The reason for this is that only relay descriptors exist in the cached-descriptors file so there's little need for this to include hacks to differentiate between the two. commit e2528b85f6b0bf1fb82c42599e71d1c925fcd6ae Author: Damian Johnson Date: Mon Jul 23 10:09:38 2012 -0700 Calling QUIT prior to disconnecting socket Making the Controller's close() method try to be a nice controller by issuing a QUIT call before detaching the socket. This is a best-effort call (errors ignored) since the caller just wants the socket shut down and we don't need the QUIT to succeed to do that. Original change by gsathya and discussed on... https://trac.torproject.org/6234 commit e4699ba9545a99ec58b7d211dc7bff8d236fbb8e Author: Damian Johnson Date: Mon Jul 23 10:04:23 2012 -0700 Including common tag comparisons in version unit tests commit 63519a16c31b6478e912c41cd7d47e1fcb634750 Author: Damian Johnson Date: Sat Jul 21 22:28:41 2012 -0700 Including tor's git commit in Version Expanding the Version class to include an 'extra' and 'git_commit' attribute that reflected a proposed expansion of the spec... https://trac.torproject.org/6445 This is already the de-facto form of tor versions so moving ahead with including it in the Version class. commit 43db1dea2487ed7517c8be3f44a315fdeac731a8 Author: Damian Johnson Date: Sat Jul 21 19:55:51 2012 -0700 Sorting tor versions lexically According to the version spec we should "compare them lexically as ASCII byte strings". This seems a little weird since it means a reverse alphabetical order (ie 'Z' comes before 'A') but oh well. commit fd2da106d0f90061532f8880702138b297ca4230 Author: Damian Johnson Date: Fri Jul 20 18:19:29 2012 -0700 Pylint fixes Variety of issues, most of them caught by pylint. commit d7ca32639205ca99ca053270b98b90dee4cc7986 Author: Damian Johnson Date: Thu Jul 19 09:08:25 2012 -0700 Adding exit_policy to stem's __init__.py Forgot to add the new exit_policy.py module. commit 78a997ea34aaee8cc4515382fb0f0ec4814c32a4 Merge: e68adb7 e670090 Author: Damian Johnson Date: Thu Jul 19 08:58:07 2012 -0700 Adding an ExitPolicy class First draft was ported from arm by gsathya in... https://trac.torproject.org/5454 I've since rewritten most of it to support IPv6, masks, and properly comply with the Tor spec for an exitpattern (arm's class was written to support torrc policies, which are different). commit e6700902cd914922515f17179ee1e8e97a5af507 Author: Damian Johnson Date: Thu Jul 19 08:55:30 2012 -0700 Minor post-rebase revisions Just a couple things I spotted after rebasing my exit_policy branch onto the current master. commit 59ed7fdd4f90fe6ac804b7b24dfdc0c3f2bc68dd Author: Damian Johnson Date: Thu Jul 19 08:47:43 2012 -0700 Using the ExitPolicy class for ServerDescriptors The ServerDescriptor class had a todo item to replace the lists of strings that it was using for an exit policy with the ExitPolicy class when we had one. Making it so... commit af9e5b4e9f4dcb149d37c7f7ad0249abaa56e689 Author: Damian Johnson Date: Wed Jul 18 09:55:06 2012 -0700 Revised MicrodescriptorExitPolicy Rewrite the MicrodescriptorExitPolicy and expanded its tests. commit 6d8dcbec4aacfc27ea43e28deea19675c715a28d Author: Damian Johnson Date: Tue Jul 17 09:53:51 2012 -0700 Tests for the ExitPolicy's __str__ and __iter__ methods commit c1ced39c89092a81dad3bd3d8ad3545296a64190 Author: Damian Johnson Date: Tue Jul 17 09:47:21 2012 -0700 Test and fix for the ExitPolicy's summary() method Ok, I was being stupid. The performance issue for the summary() method was due to an O(n) lookup when checking if we had seen a port (which made, for instance, to full-port-range policies choke on a 65535^2 operation). Swapped to using a set instead to make it constant time. commit 22ec4c9226da797778f56d8d1303b61c4e3bf710 Author: Damian Johnson Date: Tue Jul 17 09:29:32 2012 -0700 Dropping the ExitPolicy's test_parsing() The test_parsing() test is redundant with the ExitPolicyRule tests, so dropping it. commit ac5b8e74f4b07ee0e2ab25745519417711bdf576 Author: Damian Johnson Date: Tue Jul 17 09:27:31 2012 -0700 Tests for is_exiting_allowed() and can_exit_to() Tests for a couple more ExitPolicy methods, and fixes for issues with is_exiting_allowed() that they revealed. commit 1936c4d5bdb861c34409a3e366e63035cce3a4a3 Author: Damian Johnson Date: Tue Jul 17 08:29:56 2012 -0700 Test for the set_default_allowed() method commit 6f36f95711d54fee1bdf7391842f5e76dde8538b Author: Damian Johnson Date: Tue Jul 17 08:10:17 2012 -0700 Adding a test for pydoc header examples Little test for our pydoc example, and fixing a couple little bugs. There's a larger bug due to our naive summary() implementation though a better solution isn't immediately coming to mind. commit 384e78df0594c7d60af4b4a0096b12d4419db7f8 Author: Damian Johnson Date: Tue Jul 17 07:58:14 2012 -0700 Correcting and expanding summary() example The ExitPolicy class no longer has a from_str() function. Also adding an example that better exemplifies how ranges work. commit 5ccc2ca7b4d7ab1974871400e0b8fb3ff1478df4 Author: Damian Johnson Date: Mon Jul 16 09:55:06 2012 -0700 Test for the ExitPolicy's constructor Test to make sure that we can handle both string and ExitPolicy lists. This also checks that we can easily handle the split() output when breaking up a csv. Added __eq__() methods for the ExitPolicy and ExitPolicyRule to make policy comparisons easier. commit 363e87f5bd55b4ecac9514d6f423fba52d4f55ec Author: Damian Johnson Date: Mon Jul 16 09:05:25 2012 -0700 Revised ExitPolicy class Revising the ExitPolicy class to something resembling its final incarnation. Next gonna revise the tests (they pass, but could use some love). commit 004650ae2b09ec295f6e7ce4d70993c1764761d9 Author: Damian Johnson Date: Sun Jul 15 14:53:53 2012 -0700 Splitting up ExitPolicyRule parsing The ExitPolicyRule constructor was unpleasantly huge (and by extension, hard to read). Moving most of it to helper funcitons for parsing the addrspec and portspec. commit 77c3a5ffa871accde7cb62fb3f9c40b30051c9c2 Author: Damian Johnson Date: Sun Jul 15 14:06:16 2012 -0700 Removing ExitPolicyLine and ExitPolicyError The ExitPolicyRule is a drop-in replacement for ExitPolicyLine, and there's little reason to introduce a custom ExitPolicyError exception when a ValueError effectively describes its use cases. commit 0c069727eb5337cc7c3f422387e3f4d0ad5caa9c Author: Damian Johnson Date: Sat Jul 14 18:32:50 2012 -0700 ExitPolicyRule class Adding a class that conforms with the dir-spec's exitpattern entity. Plus lots 'o unit tests that told me over and over and over again that I was being stupid. Hopefully this version is at least kinda sorta close to being right... commit ebec5fbd4a8080b1003ae87681cd5b804f5a81d7 Author: Damian Johnson Date: Fri Jul 13 09:54:42 2012 -0700 Utilities for IPv4 and IPv6 addresses Adding the utilities that I need for the ExitPolicy to handle and translate addresses and their masks. Pity that these aren't provided by the python builtins. The IPy package seems to do it but this isn't worth adding a new dependency. commit 30ba1019fcbeca9220e85fa5216b6bac1b5db16f Author: Sathyanarayanan Gunasekaran Date: Tue Jul 3 02:03:12 2012 +0530 Add microdescriptor exit policy The microdescriptor policy class can now parse exit policy summaries. Add tests for this commit fdb140763097c67c9612c5e7090534330e89a68f Author: Sathyanarayanan Gunasekaran Date: Mon Jul 2 19:19:56 2012 +0530 Add exit policy test to run_tests commit 03f625b328ba9148909dc523196a16f4356b9dc2 Author: Sathyanarayanan Gunasekaran Date: Tue Jun 26 08:51:24 2012 +0530 Add unit test for exit_policy Test the validation and parsing of the exit policy commit 289063fefe8857c6ca10e95d2a3a90fcbd925f51 Author: Sathyanarayanan Gunasekaran Date: Tue Jun 26 08:49:53 2012 +0530 Remove MicrodescriptorExitPolicy, add validation and exception The micro desc exit policy is currently useless until we figure out what it's used for. Validate all the ip address and ports. Raise exception if wrong commit b51d8da9bd2a6d4377019b37bbad768ad8ba17ca Author: Sathyanarayanan Gunasekaran Date: Tue Jun 26 08:48:23 2012 +0530 Make stem.util.connection.is_valid_port accept a list Iterate through the list and check if each item is a valid port or not. commit a8b654e046e143bb89c40021c0602dd9510912fe Author: Sathyanarayanan Gunasekaran Date: Mon Jun 11 23:27:47 2012 +0530 Add MicrodescriptorExitPolicy This class has four methods - add() - which is the public method that the user calls to add exit rules or policy. If a policy is added, then this instance can't be changed further. Rules can be added till we we get a wildcard('*') as a rule. add_rule() - parses the rule add_policy() - parses the policy __str__() - creates a string representation of the policy commit 0a1c6238cc95f37a1b26369247d5d9876ba2b804 Author: Sathyanarayanan Gunasekaran Date: Thu May 17 22:19:10 2012 +0530 Add ExitPolicy.get_summary() This provides a summary description of the policy chain similar to the consensus. commit 4d88884b0cfdc98ca4c4741d4e1073267418038c Author: Sathyanarayanan Gunasekaran Date: Fri Mar 23 15:18:44 2012 +0530 Add doc and fix whitespace commit 607e32a3cf3d69ed185ecee4e62f55e7101d587a Author: Sathyanarayanan Gunasekaran Date: Fri Mar 23 14:10:54 2012 +0530 Parse "private" alias better commit 80228a52e3f9507547cf924156a1bcafdea6540b Author: Sathyanarayanan Gunasekaran Date: Fri Mar 23 12:21:01 2012 +0530 Overhaul exit_policy.py - Fix Indentation issues - Remove ExitPolicyIterator class - Add ExitPolicyLine class - This class represent a single line from the Exit Policy. (provides much better abstraction IMHO) - __init__(), __str__(), and check() are copied from ExitPolicy - Changes in ExitPolicy - - Now, it acts a wrapper class to ExitPolicyLine - It contains a list ExitPolicy._policies which stores all the policies and each item in _policies is an object of ExitPolicyLine - check(), isExitingAllowed(), __str__(), and __iter__() are changed to provide a wrapper of sorts for ExitPolicyLine - add() is now used to add an exit policy line - This creates an object of ExitPolicyLine and adds it to ExitPolicy._policies commit 88c436276f1aa0d20206499099b689eb50ace573 Author: Sathyanarayanan Gunasekaran Date: Thu Mar 22 23:20:43 2012 +0530 Add ExitPolicy, ExitPolicyIterator from arm/torTools commit e68adb7fdd465e1780bc4ee1243e0c226add77d9 Author: Ravi Chandra Padmala Date: Wed Jun 27 18:34:15 2012 +0530 Implement convenience method for handling SAVECONF requests This also changes the Exception structure a bit. I have added an InsatisfiableRequest class which will be raised when a valid request couldn't be satisfied by Tor. I've also added a super class for non-socket non-protocol errors called OperationFailed. Any error that was raised by an error response returned by Tor (i.e., one with an error code & a message) should subclass this. The exceptions now look like ControllerError - Base exception raised when using the controller. |- ProtocolError - Malformed socket data. |- OperationFailed - Tor was unable to successfully complete the operation. | |- UnsatisfiableRequest - Tor was unable to satisfy a valid request. | +- InvalidRequest - Invalid request. | +- InvalidArguments - Invalid request parameters. +- SocketError - Communication with the socket failed. +- SocketClosed - Socket has been shut down. commit 611e75dbeb5a81a655ca0f8ec1b4495850c3459a Author: Damian Johnson Date: Tue Jul 10 08:22:55 2012 -0700 Allowing extra-info dirreq-v*-share to be above 100% Spotted an extra-info descriptor with dirreq-v2-share and dirreq-v3-share values over 100%... https://lists.torproject.org/pipermail/tor-dev/2012-June/003679.html Karsten suggested simply removing the restriction on this field's upper bound since it's soon going away. commit a0f980c996587ef112379891339d463bda3d487a Author: Damian Johnson Date: Mon Jul 9 12:00:08 2012 -0700 Supporting multiple extra-info transport lines The 'transport' lines being added to extra-info descriptors can appear any number of times, so storing them as a list. These options are weird in that they have an "address:port [arglist]" which never appears in the wild since they both only appear on bridge descriptors and are scrubbed from bridge descripotrs. commit d52e934c24c9d327e277eb60bfee060697cb43ec Author: Damian Johnson Date: Mon Jul 9 11:52:56 2012 -0700 Only consulting major descriptor versions Minor descriptor version bumps are backward compatable changes. We might not recognize new additions, but it won't break us either (they'll just be unrecognized). Pointed out by Karsten on... https://trac.torproject.org/6257 commit ab553f54e70ea21d8bd117cf912e1b208417e5d5 Author: Damian Johnson Date: Mon Jul 9 11:34:33 2012 -0700 Minor revisions for LOADCONF and prior cleanup commit 6cc4e195452bc573e34f0b2de295b1028a20eb4a Author: Ravi Chandra Padmala Date: Fri Jun 29 00:25:17 2012 +0530 Implement Controller.load_conf commit c5472856c9895bd118af79297ccdce59fcba2e13 Author: Ravi Chandra Padmala Date: Mon Jul 9 10:12:23 2012 +0530 Remove ugly if-elif tree in favour of a dict commit ac0d2183f04838790c64ff3194ae29f90fd4ed93 Author: Ravi Chandra Padmala Date: Mon Jul 9 09:55:38 2012 +0530 Aesthetic changes to __init__.py files Making __all__ lists multiline when they're long and awkward. commit 6a91cda3db2a1ab1c4bf485fad03fd5cea24657a Merge: a325a40 3d15387 Author: Damian Johnson Date: Sun Jul 8 13:13:17 2012 -0700 Controller methods for GETCONF and RESETCONF Feature addition by Ravi, addressing... https://trac.torproject.org/6239 commit 3d1538746adad85bb7235d8bc7f4b3d93b1f3d50 Author: Damian Johnson Date: Sun Jul 8 12:39:29 2012 -0700 Splitting set_conf/reset_conf back up Ravi made a good point that the set_option() syntax for resetting config options is now worse, so trying a compromise of our approaches. This reintroduces set_conf() and reset_conf() methods, but also keeping my set_options() method to handle the more complicated use cases (batch setconf requests, resetconf with values, and context sensitive options). Personally I found the old set_conf() method confusingly overloaded (python does not handle overloading use cases very well). This solution lets us keep the set_conf() and reset_conf() methods simple, while also providing something that's reasonably intuitive (imho) for the complicated use cases. commit e2057f1beb158ce111ddc735c8b1a58fab79f6d1 Author: Damian Johnson Date: Sat Jul 7 19:26:56 2012 -0700 Reintroducing tuple support to get_options Damn I hate it when I'm right. The reason that arm supported tuple lists is the fucking HiddenService* options. Adding support for them back in, though I gotta admit that I flirted with the idea of just saying "screw hidden services". These things are such a hack... commit f9b370c455d4128aabe73702a7a445dfe17d4a2f Author: Damian Johnson Date: Sat Jul 7 18:50:17 2012 -0700 Revisions for prior SETCONF and RESETCONF additions Several revisions to the prior changes, the most prominent of which is to use an API more similar to arm's. The SETCONF and RESETCONF methods are functionally identical except how they handle undefined values, so there's little reason to provide separate methods in our controller. Just using an optional 'reset' arg instead. One thing that arm got wrong (and so did stem) is that our method to change multiple options should accept a dict rather than a tuple list. Order *might* matter for the accursed HiddenService options, but for the 99.9% use case a tuple list is a very strange argument to accept for this. This also includes some testing fixes, for instance... * Testing when we have a list argument (for instance, setting our ExitPolicy). * Failing our test if one of the calls that should fail actually succeeds. * Reverting the configuration options that we change after the test. * An identical test for context sensitive options was added to both the getconf and setconf test. Definitely a good use case to check and it could belong with either setconf or getconf, but doing the same test twice is pointless. commit da66a5dfa3818de5376e252b667a649abccf9702 Author: Ravi Chandra Padmala Date: Fri Jul 6 21:04:53 2012 +0530 Make get_conf_map work properly with HiddenService* options commit 9af2616aabb3bd515d695f3ab4ed1c839428bd4b Author: Ravi Chandra Padmala Date: Tue Jul 3 00:07:36 2012 +0530 Fix documentation formatting commit 7d8b1fdef6e6b2cd38a7c2f3aa09a957b03af1a5 Author: Ravi Chandra Padmala Date: Wed Jun 27 01:28:39 2012 +0530 Make tests cross-platform & fix things that broke during a rebase commit ea57e936e08400d63a423c669311643afd4dd84d Author: Ravi Chandra Padmala Date: Tue Jun 26 10:04:52 2012 +0530 Add Controller.reset_conf and fix Controller.set_conf commit 85c923bfc784aaf8080e08053ef6b577e8c23944 Author: Ravi Chandra Padmala Date: Thu Jun 21 22:08:30 2012 +0530 Add SETCONF integration tests, SingleLineResponse unit tests and the bugfixes that come with them commit 51025794e68b1607e6a15741ebdab45650df0c25 Author: Ravi Chandra Padmala Date: Thu Jun 21 20:30:04 2012 +0530 Add unit tests for SingleLineResponse and add missing imports commit 4ed9a1fba65257ccd0caba41da79ac6b879737f2 Author: Ravi Chandra Padmala Date: Thu Jun 21 13:58:23 2012 +0530 Add set_conf wrapper method commit ba53da1772007aca88bc75f1fed60d49170eb964 Author: Ravi Chandra Padmala Date: Fri Jun 15 18:21:47 2012 +0530 Add a generic SingleLineResponse class commit 0bfa815f85337cbd216d0fbb2965b6ab305d3874 Author: Ravi Chandra Padmala Date: Mon Jul 2 22:48:09 2012 +0530 Fix Controller.get_conf_map's documentation formatting commit a325a40ff04df1a83614c529995749f2da8e79f8 Author: Damian Johnson Date: Sat Jul 7 20:42:45 2012 -0700 Replacing platform specific %D date formatting Python's strftime() method has a set of platform independent flags, but also accepts anything that our platform's libc does... http://stackoverflow.com/questions/10807164/python-time-formatting-different-in-windows I was using '%D' date formatting for our logging, but that doesn't work on Windows... https://trac.torproject.org/6206 Replacing with the equivilant platform independent flags. Issue caught by Beck. commit 74597a78159acf20146c72d0392e74f07d2bf9e3 Author: Damian Johnson Date: Sat Jul 7 20:23:27 2012 -0700 Skipping proc tests broken by tor's disabledebugger features Stem's proc integ tests exercise that utility by running it against our tor instance. However, as discussed in ticket #3313 the DisableDebuggerAttachment feature screws up our proc permissions. Skipping those tests if they would fail due to this. Caught by Ravi in... https://trac.torproject.org/6286 commit 6aa207434661434984a7ea5b8cf3d2ab013dea69 Author: Damian Johnson Date: Sat Jul 7 20:14:19 2012 -0700 Processing proc stat files with more than 44 entries A long-standing arm and stem bug is that it checks if '/proc//stat' contains exactly 44 elements and, if not, refuses to parse it. This is a mistake, stat files may have more entries depending on the platform. Changing the check to account for this. commit bbcff94dee48e96db3e7c9b396926a9fac7f64b9 Author: Damian Johnson Date: Sat Jul 7 20:07:18 2012 -0700 Accounting for logging.Handler bug in python 2.5 In python 2.5 logging.Handler doesn't extend object, causing our super() call to fail with a stacktrace. Hacking around it. commit 48404ed766a775b2d958d9b4b686e96cc88914c0 Author: Damian Johnson Date: Sat Jul 7 19:55:24 2012 -0700 Skipping tests that make repeated connections on OSX Stem's tests currently hang on Mac OSX, and that doesn't look to be getting better any time soon... https://trac.torproject.org/5917 I've narrowd this down to being either a python, tor, or mac bug with how repeated socket connection/closures are handled on that platform. For now skipping the tests that encounter this issue if we're running on a mac. commit 4c7831fee70848b002b97d8d8dd4b30519ec9eb9 Author: Damian Johnson Date: Sat Jul 7 13:15:52 2012 -0700 Clarifying comments for the get_pid_by_open_file test The prior fix is better for the test (working == better!), but makes it a bit less self-descriptive. Adding a couple comments. commit bdc71d9c70e6184f7450e1fecef2cb2ce7bc4ac5 Author: Ravi Chandra Padmala Date: Wed Jun 27 16:44:01 2012 +0530 Fix #6248 and make test_get_pid_by_open_file cross-platform while doing so commit 6631bf536adb73f7499bde68a0f1289a51e8f2aa Author: Damian Johnson Date: Fri Jul 6 22:33:15 2012 -0700 Removing broken stem.util.control import I deleted stem/util/control.py but forgot to rm the pyc afterward so stem continued to work for me despite having a broken import. Caught by Beck. commit c9e2f7be38e7dede69a7898c7627294f857ea395 Merge: af27984 c09b80e Author: Damian Johnson Date: Wed Jul 4 14:32:25 2012 -0700 GETCONF handling commit c09b80ef912bb590a37e0d4533a34bbba14cabf9 Author: Damian Johnson Date: Wed Jul 4 14:25:32 2012 -0700 Revisions for GETCONF handling These changes have gone through several review iterations so there wasn't much to tweak with it. Only substantial twiddling on my part was... * The super() method is new to me, and looks like it's the preferred method (especially in Python 3.x when it's syntactically much nicer). Swapped all of our constructors to use super() rather than Parent.__init__(self, args). This also forced me to fix some base classes that weren't inheriting from object. * Moved _case_insensitive_lookup() helper into controller module rather than having a separate util. * Removed the "GETCONF reply doesn't match the parameters that we requested." check since this'll always produce an error if tor adds new context sensitive options (my bad). * The get_conf_map() method wouldn't accept context sensitive options like HiddenServiceDir. * The get_conf_map()'s default value is what we should return if the lookup fails, not the mapped values (ie, get_conf_map("blarg", {}) should give an empty map). * Integ test was missing a requirement check that we could make a control socket, causing the RUN_NONE target to fail. commit ffd68ccc660a92cc3cff61ae8baafebb6799e52e Author: Ravi Chandra Padmala Date: Mon Jul 2 09:39:35 2012 +0530 Fix a bug because of which get_conf wouldn't work for HiddenService* commit 765c497f824fde380d8d1b864e3dec36fcce3b11 Author: Ravi Chandra Padmala Date: Fri Jun 29 23:12:34 2012 +0530 Changes for review commit a035e89b522950a0bef54840ace887d2af31095c Author: Ravi Chandra Padmala Date: Thu Jun 21 03:39:34 2012 +0530 Documentation fixes & some code refactoring commit 9e40210c83c2af32fecb08e4ba4ea71833ce8050 Author: Ravi Chandra Padmala Date: Thu Jun 21 03:39:34 2012 +0530 Behave smartly with context-sensitive config keys commit 0b83af2922b5c377161626c7aef2af14a0152463 Author: Ravi Chandra Padmala Date: Sun Jun 17 22:27:55 2012 +0530 Fix a bug where getconf parses didn't set the default correctly for multiple keys and write a test to check for this commit f6231a9dd9ed52dae4e77e153cae13e344a4e094 Author: Ravi Chandra Padmala Date: Sun Jun 17 20:29:58 2012 +0530 Add getconf test to check for multiline configuration keys commit b8959dfe154180018071a07d7ad8eceb68a4c516 Author: Ravi Chandra Padmala Date: Sun Jun 17 18:05:54 2012 +0530 Various modifications to GETCONF parsing * Change the way entries are stored. Now only stores one value unless the caller explicitly mentions that he want to retrieve multiple values using the 'multiple' argument. * Stop checking if the received configuration options are the ones that were requested. (The HiddenService options do this) * Minor documentation fixes. commit a80a784b29a91fb3ee9a6ec8a30b829b68648e41 Author: Ravi Chandra Padmala Date: Thu Jun 14 12:16:44 2012 +0530 Add attributes to InvalidRequest and InvalidArguments Adds attributes to store the error code and error message to InvalidRequest and InvalidArguments commit 42ae08cfdda0ea2f90bccb2b3621b182ddf99e0a Author: Ravi Chandra Padmala Date: Wed Jun 13 07:38:13 2012 +0530 Make the GETINFO parser raise InvalidArguments instead of ProtocolError when appropriate commit ece7f85dd675633de3b866c868bba1560fd22060 Author: Ravi Chandra Padmala Date: Tue Jun 12 21:07:13 2012 +0530 Minor change to GetConfResponse documentation commit 3b25147b7fcaadbc7e2aec6ec89158ed1d738687 Author: Ravi Chandra Padmala Date: Tue Jun 12 19:59:30 2012 +0530 Multiple fixes to GETCONF parsing * Add stem.socket.InvalidArguments and make the GETCONF parser raise it instead of stem.socket.InvalidRequest. * Fix the parser to parser to correctly parse multivalue configuration keys (such as ExitPolicy). Add tests for the same. * Fix tests to run against a Tor client using a control socket file. * Minor changes to documentation. commit 16d392ac74996ae38a8f3a33191ebc1899a5cf8f Author: Ravi Chandra Padmala Date: Sat Jun 9 09:56:17 2012 +0530 Add integration tests for testing Controller.get_conf commit f01a50e5bd1b24cb30676d869057fea3c8e04af9 Author: Ravi Chandra Padmala Date: Sat Jun 9 09:55:01 2012 +0530 Add get_conf convenience method to Controller commit 0db367a151dc75fde8a3a90fd273bda7ad9227ab Author: Ravi Chandra Padmala Date: Sat Jun 9 09:49:07 2012 +0530 InvalidRequest now inherits from ControllerError Moved stem.response.InvalidRequest -> stem.socket.InvalidRequest and made it a subclass of stem.socket.ControllerError commit f908ce9c44ffad9237d0f571351adb2143001f52 Author: Ravi Chandra Padmala Date: Fri Jun 8 17:50:06 2012 +0530 Add unit tests for GETCONF parsing commit 5701329478dfc363da17d80cafca26f07a9a9def Author: Ravi Chandra Padmala Date: Fri Jun 8 12:41:09 2012 +0530 Add GetConfResponse class for parsing GETCONF responses commit af27984ffcaa418fb71e68992976967a06c4ba56 Merge: 2fe0f2c c043464 Author: Damian Johnson Date: Sat Jun 30 15:51:59 2012 -0700 Testing fixes for Windows commit c0434646354a6031fd8fb0750af4700f1a450f1a Author: Damian Johnson Date: Sat Jun 30 15:50:14 2012 -0700 Minor formatting tweaks commit 489c7bab48001e8af82b897f2d7ab5a487169643 Author: Beck Date: Fri Jun 29 23:22:36 2012 +0800 Skipped some unrelevant integ test cases on windows commit 7515471fa13c92873307054ea9e40ef5eea64c4a Author: Beck Date: Wed Jun 27 00:31:55 2012 +0800 Replaced a hardcoded path in test_relative_cookie commit c001394928e01eb9807c747fed0283c7748578f9 Author: Beck Date: Tue Jun 26 23:58:32 2012 +0800 Fixed test_expand_path_unix, and added test_expand_path_windows test_expand_path_unix was changing os.path.sep in order to change the path seperator used by os.path.join. However, this did not work. The path seperator was hardcoded in os.path.join, which is posixpath.join on *unix systems and ntpath.join on windows. We now mock os.path.join to use posixpath.join or ntpath.join in different test cases. commit 8edfe7123997e99885ae514bbc1ae76d4c9bc1f8 Author: Beck Date: Fri Jun 22 23:31:39 2012 +0800 Fixed bugs in calls to test.runner.skip() commit 62e51e9e98ef2d2f309eece027d7d2417e18dedd Author: Beck Date: Fri Jun 22 23:22:42 2012 +0800 Fixed bug that util.system.expand_path does not expand tildas on windows commit 9724963081f269eb68b17cf116c950dbd6591474 Author: Beck Date: Fri Jun 22 19:51:59 2012 +0800 Replace calls to os.uname() by platform.uname() os.uname() is unix-specific, while platform.uname() is a more portable uname interface that would also work on windows. And platform.system() == platform.uname()[0]. commit 2fe0f2c84f0d41111571f2ae02575124da008743 Merge: c35c8a6 14087a7 Author: Damian Johnson Date: Sat Jun 30 15:41:40 2012 -0700 Unit and integration tests for stem.util.proc commit 14087a70e2c0e40541d610ce41c4b08c8fab6b3e Author: Damian Johnson Date: Fri Jun 29 08:42:08 2012 -0700 Revisions for proc unit and integ tests Lots 'o changes, most just nitpicks to satisfy my OCD. The main functional changes are just... * The mock_fn() and find_subsets() helpers are useful for other tests so moving them to the mocking module. The authentication unit tests, for instance, had a helper that this has replaced. * The default target for the mock_fn() calls doesn't seem to be needed, and could mask issues that prevents proper system abstraction (ie, if we're passing through to os.readlink() then we probably won't notice that the test only works on Linux). * The stem.util.proc.is_available() check was broken since we were using the function as a boolean, which is always True... >>> def foo(): ... return False ... >>> bool(foo) True * We created a controller to spawn a control connection, but forgot to close the controller afterward. * Test broken if we were running without a control port (for instance, with the RUN_SOCKET target). commit af9d226e851b3ca1ab552a3fbd249a61602b8fd3 Author: Megan Date: Thu Jun 28 16:51:48 2012 -0400 Removed commented out code that was forgotten. commit 830bc5fa28f9683109a9ed7bee2980312f550910 Author: Megan Date: Thu Jun 28 16:31:59 2012 -0400 Added integration tests for proc.py. In the cases of test_get_memory_usage(), test_get_stats(), test_get_connections(), there were issues of doing direct tests. First, the only way to have values to check against is with code written in proc.py, which would mean we are checking the code against itself. Second, the results of these functions change frequently, so by the time the functions are called, their results will differ from the expected values. To get around this, we simply checked that these values were nonzero. commit 10ae1304d02c390237d865c5fc439992249bc9d0 Author: Megan Date: Tue Jun 26 17:20:37 2012 -0400 Made additional changes regarding sphinx documentation. This completes Code Review 1. commit 48fcbc69e4ed13a44ff160ba35997a46487f997f Author: Megan Date: Tue Jun 26 17:05:35 2012 -0400 First code review changes. commit e94e48ecf3209273b6de13bd405fa87d3f1203b4 Author: Megan Date: Mon Jun 25 18:05:44 2012 -0400 First complete draft of proc unit testing code. Most mocked functions utilize mock_fn(), which checks for expected input and either returns the appropriate output given the mocking or lets the original function be called. This is necessary in cases where outside code calls these functions and the mocked version is not able to supply them with the needed output. Private methods in the stem/util/proc.py file are not tested, nor is is_available(). This decision was based on the conventions set in system.py testing code. commit 3fd005a27ca35bf802d3917c992d2626534e87c9 Author: Megan Date: Thu Jun 14 16:17:32 2012 -0400 Created proc.py testing code and imported unit tests to run_tests.py. Wrote mock_get_lines to mock the proc module's _get_lines function. Completed unit testing code for proc utilities such as get_system_start_time, get_physical_memory, and get_memory_usage. The test_get_memory_usage function was written to test the functionality of mock_get_lines as it is the only function that takes in multiple line prefixes. commit c35c8a6cbb3659841f79159512c24a08a5e1d622 Author: Damian Johnson Date: Fri Jun 29 09:59:16 2012 -0700 Handling version 1.1 of bridge extrainfo descriptors Version 1.1 of the bridge extrainfo descriptors adds a new 'transport' field. I'm not entirely sure yet of its attributes, but recognizing the new descriptor version and parsing the field. I'm also expanding the tests a bit so unit tests cover bridge extrainfo descriptors, and parsing the metrics header line rather than just doing string matches. This is to address... https://trac.torproject.org/6257 commit 86baf8e562c0486d2ac2e2be893480eb149742b4 Author: Sathyanarayanan Gunasekaran Date: Sat Jun 23 19:16:38 2012 +0530 Remove single letter variable According to the guidelines of our benevolent dictator we must abolish the use of single letter variables. The sole infiltrator has been abolished. Once again, there is peace in the land of stem. commit 54a013552266540d5029af2ff08db2fed1c1dc92 Author: Damian Johnson Date: Sun Jun 24 14:14:49 2012 -0700 Stopping tor initialization timeout when done Ok, I'm kinda surprised that we haven't tripped over this before. We never terminate the tor initialization timeout on either or success or failure, causing us to arbitrarily kill our process after 90 seconds. This bug probably survived so long becuase our integ tests take less time than that to complete, and then spawns a new tor instance resetting the timer again. commit 4b79530c47abdeebb33bc7c06a6a6bda4e2815b6 Merge: 09acb5e 1c01521 Author: Damian Johnson Date: Sun Jun 24 18:20:59 2012 -0700 Python 2.5 and OSX compatability fixes commit 1c0152185a9ff27a49f6491bcd2ea06f64158820 Author: Damian Johnson Date: Sat Jun 23 14:36:08 2012 -0700 OSX sucks at terminating processes Our os.kill() call evidently doesn't kill tor right away on OSX, causing tests to disrupt each other (port conflicts). commit 04a6225ffdccc81a431044d05f4f411c7e5a88f5 Author: Damian Johnson Date: Sat Jun 23 14:18:32 2012 -0700 Inverted check when falling back to kill a process Oops, we're supposed to use os.kill() as a fallback if we *aren't* on windows. commit e625e84b003420908d410ab10f51fdd78a1ccf17 Author: Damian Johnson Date: Sat Jun 23 12:16:53 2012 -0700 Respecting reader stop() calls in mid-directory If we, say, attempted to read a directory with a million files then the reader's stop() method won't be respected until we've finished with that directory. That's stupid. Checking the _is_stopped Event after processing each file. Python lacks targeted breaks, so I needed to refactor the code a bit to accomidate this. commit 4e02850db89006f256290e8bc7bea7169c6cd57c Author: Damian Johnson Date: Sat Jun 23 11:55:57 2012 -0700 Missing kill() fallback when ending integ tests As an earlier commit mentioned, Popen.kill() is unavailable on python 2.5, and its fallback (os.kill()) is unavailable on Windows. Accounting for this in the integ test runner. commit 2fc1c420818bff19437a2b74713223a410d9898e Author: Damian Johnson Date: Sat Jun 23 11:47:24 2012 -0700 get_pid_by_port() unavailable on Mac OSX Tests for get_pid_by_port() are failing on Mac OSX 10.5.8 because sockstat is unavailable and lsof neither provides ports nor accepts the flags we need. Noting this in the pydocs/comments, and skipping its integ tests when on a mac. commit a5f6006b3e1d072e1cd9524aa57408d707ff3ba0 Author: Damian Johnson Date: Fri Jun 22 09:27:03 2012 -0700 Respecting custom tor commands for process integ tests The process integ tests were attempting to run 'tor', which naturally doesn't work if it isn't in your path. Using the command that we're using for our main test instance instead since that is sure to work. commit ac34b8151c5b46369578922ba4f90fdce15128b3 Author: Damian Johnson Date: Fri Jun 22 09:18:39 2012 -0700 Missed another tarfile 'with' declaration Missed that we were using the 'with' keyword for a tarfile in the integ tests. Using a try/catch instead so this'll work on python 2.5. commit 9d54db62a7b918c3bc6a852275adaf70dfdfef44 Author: Damian Johnson Date: Fri Jun 22 09:14:20 2012 -0700 Subprocess lacks a kill() method on python 2.5 The Popen.kill() ad Popen.terminate() methods were first added in python 2.6... http://docs.python.org/library/subprocess.html#subprocess.Popen.kill We can work around this with os.kill() but that only works on *nix... http://stackoverflow.com/questions/552423/use-python-2-6-subprocess-module-in-python-2-5/552510#552510 commit e51fea598dced5e3c4a05c4cd0ad509beed530fe Author: Damian Johnson Date: Thu Jun 21 09:45:14 2012 -0700 Avoiding the 'with' keyword for tarfile The tarfile's __exit__() method was added sometime after python 2.5, causing it to produce a stacktrace if used via the 'with' keyword. Pity, yet another hack to accomidate a six-year-old version of python... commit 60b0e8ba06a03e5ae1664959c2f9e1552f224b39 Author: Damian Johnson Date: Thu Jun 21 09:40:00 2012 -0700 Using theading's currrentThead() rather than current_thread() The threading module's current_thead() function isn't availalbe in python 2.5. It isn't documented, but there seems to have been a mass aliasing in 2.6 to the underscore convention so using that instead. commit 1098d9ad35dc0c04dcced034b1e564706c543c96 Author: Damian Johnson Date: Thu Jun 21 09:31:50 2012 -0700 Python 2.5 doesn't have is_alive() method for Threads Though it's not documented [1], the Thread class' isAlive() method was first aliased to is_alive() in python 2.6. Also filed a ticket for the documentation bug [2]. [1] http://docs.python.org/library/threading.html#threading.Thread.is_alive [2] http://bugs.python.org/issue15126 commit 9a8ac67f57fc203eb20dddea3aa09ba49bfe7a97 Author: Damian Johnson Date: Wed Jun 20 10:35:17 2012 -0700 Python 2.5 doesn't have is_set() method for Events In python 2.5 the threading module used camel case method names. They also accepted the underscore convention (the 'official' style for python) in python 2.6, but until we drop 2.5 compatability we need to use the camel case versions. commit cec8ecfcf175af55d9503d45e169cd26f9664279 Author: Damian Johnson Date: Tue Jun 19 09:47:33 2012 -0700 Integ test relied on tor being in path I thought that it was a safe assumption that tor was in our path since the integ tests will only run if it can start tor. However, the tests have a '--tor ' argument which means that the tests may be running without our path including the tor binary (or even having the binary called 'tor'). Changing the test to look for 'ls' or 'dir' instead since every platform that I can think of has that. Caught by Karsten. commit 09acb5e43ba6afd265bcf79e7534dbf369fa227b Author: Damian Johnson Date: Fri Jun 22 09:56:17 2012 -0700 Checking for windows newlines A while back I added windows/old-style-mac newline checking to the whitespace checker, but got distracted before committing. Reviving it from the stash. This should do the trick, but haven't checked it against windows files (guess we'll get confirmation that it works or doesn't when we have one submitted). commit f724692fe512a2eaf2b2c4bfd491513f48bfca23 Merge: a4243fd 6f626f3 Author: Damian Johnson Date: Thu Jun 21 08:45:09 2012 -0700 Merging Beck's windows test compatibility fixes commit 6f626f3f1a43368f13b2679f3eab6322f26788c0 Author: Damian Johnson Date: Thu Jun 21 08:35:32 2012 -0700 Minor changes for windows test compatibility fixes Just a few minor tweaks... * The 'path' and 'relative_path' were starting to get confused, so making the former the pristine input argument and having the later be what we modify. * A recent change removed direct usage of 'skipTest' since it breaks us on python 2.5 and 2.6... https://gitweb.torproject.org/stem.git/commitdiff/ed0db8f * Skipping the 'is multiple tor instances' check rather than always skipping the system integ tests. * run_tests.py somehow lost its executable permissions in 2b96648 (...?) commit 03780a98ab131fb5ac6cf1bac192d1d7ec9f238e Author: Beck Date: Wed Jun 20 01:41:37 2012 +0800 Fixed a bug in stem.util.system.expand_path that may introduce a trailing slash. commit be611a6ad6c7054c35e02571087950e133839592 Author: Beck Chen Date: Wed Jun 20 00:24:15 2012 +0800 Fixed test_skip_listener_unrecognized_type on windows. This test case is included in stem.test.integ.descriptor.reader. Now it allows both "image/png" and "image/x-png" types. commit 73277fc0dc9709b56fd69fa0d0c0dec2effcb2c9 Author: Beck Chen Date: Fri Jun 15 12:18:03 2012 +0800 Skip two test cases in test.integ.descriptor.reader on windows. The two test cases are test_load_processed_files_permissions and test_skip_listener_read_failure. They both call os.chmod() to change permissions of the test file, but on windows, you can only set the file's read-only flag with it and all other bits are ignored. This would cause tearDown() to fail since test files are read-only and can't be removed, thus fail all the test cases. commit 232858d8607270e00d42316e8655c70e0221c87e Author: Beck Chen Date: Fri Jun 15 02:17:00 2012 +0800 Implement stem.util.system.expand_path() on windows. commit 67be8c6435e6d08b71327687ffa1097d01345bf2 Author: Beck Chen Date: Fri Jun 15 01:26:22 2012 +0800 Skip test_stop() in test.integ.descriptor.reader on Windows. commit 5c18901aa94ada77f43c1f67d91f4ee300ccea67 Author: Beck Chen Date: Thu Jun 14 19:32:36 2012 +0800 Skip integ tests for util.system if pgrep is unavailable. Since pgrep is used in setUp() of test.integ.util.system.TestSystem, all test cases would fail if pgrep is unavailable in the current platform (e.g. windows). We'll simply skip them for now. commit 57904d4540dc9e988a431986695e6e1f2f5cf8fa Author: Beck Chen Date: Thu Jun 14 19:11:09 2012 +0800 Uses stem.util.system.is_windows() for platform checking in run_tests.py commit 417d7f8117cf40378901876d0e7b324f3f694a18 Author: Beck Chen Date: Wed Jun 13 01:17:10 2012 +0800 Disable ANSI escape sequences in windows commit a4243fd09384fcbe37f076cf4ec6163c80f00839 Author: Damian Johnson Date: Wed Jun 20 09:37:40 2012 -0700 Adding pydocs for prior mocking change commit 66376f5a6ebc35fd553506c226342ec8777fc9b5 Author: Erik Date: Mon Jun 18 17:01:53 2012 -0400 Changed mocking.mock() to accomodate modules that link to other modules. We ran into a problem mocking os.readlink as in certain unix environments os links to an outside module, posix. This resulted in the __dict__ of readlink remaining unchanged, and thus not mocking the realink function (which in our environment resides in the posix module). To patch this, we added a third argument to mock() that can be used to explicitly specify the target_module. A default value of None is provided so all prior calls on this method are compatible. commit ed0db8f0defa6d94e62b20c7a2cc335f3801d91a Author: Damian Johnson Date: Mon Jun 18 09:52:51 2012 -0700 TestCase's skipTest method unavailable before python 2.7 Blarg! The unit testing's skipTest method was a new addition in python 2.7. This is especially pesky since skipTest alters the control flow of the tests (raising an exception so we can place it in setUp methods). Options are to either monkey patch support into our testing module (ick!) or add a fallback everywhere we use skipTest. This change adds a best-effort skip method to the runner, then leaves it to the caller to return out of the method (reporting success for skipped tests). Sucks, but best that we can do until we drop *both* python 2.5 and 2.6 compatability, which probably won't happen until we finally move to the 3.x series. commit 385b05b3df8e07b4cd589a49c0f348a9541726bd Author: Damian Johnson Date: Thu Jun 14 09:17:25 2012 -0700 Patched mocking.mock() to handle standard library functions Although standard library functions have the same type as built-in functions, they cannot be mocked using the same setattr approach. To fix this, we patched mocking.mock() to first look for built-in functions, then to handle non-built-in functions with the same type as built-in functions. This will catch standard library functions and mock them appropriately. This change is an adaptation of one submitted by Erik Islo and Megan Chang. commit 122d0a6aa57f21fe2ef7d41c6fd14736074345c6 Author: Damian Johnson Date: Wed Jun 13 09:33:20 2012 -0700 Tuples lack an index method in python 2.5 Yet another python 2.5 compatability hack. commit 4581c355d64f3241d649256d4e14d839aedc9ea5 Author: Damian Johnson Date: Wed Jun 13 09:27:47 2012 -0700 Adding todo notes around the stupid concessions to 2.5 I hate dirtying our code for ye old python 2.5. It's ancient and that fact isn't getting rosier very fast. However, a lot of people still use it so we should still support it for now. This won't always be the case so including TODO notes to remove these icky bits when we drop compatability for it. commit d2eef839544eaf1ed55a85df2e4fd466a175fc40 Author: Damian Johnson Date: Wed Jun 13 09:24:34 2012 -0700 The followlinks arg of os.walk needs python 2.6 We were using the 'followlinks' argument two places: in the descriptor reader and whitespace checker. For the former I'm noting that 2.6 is a requirement for the argument and coding around it to work on 2.5. For the later I don't really care if the whitespace checker follows links so I'm simply dropping the arg. commit b84a75638942e61a58b2af8edd97448ea72857d7 Author: Damian Johnson Date: Wed Jun 13 09:20:34 2012 -0700 Adding module to check for requirements Stem requires at least python 2.5, and beyond that our python version and modules does impact what functionality we have available (yay). Adding a module for tracking what we have available. commit e09e6b00a4b603db6342d27a11e3073acf8c74c6 Author: Damian Johnson Date: Wed Jun 13 08:46:21 2012 -0700 Tor executable shouldn't be needed for unit tests We only need a tor variable when running the integ tests. Dropping the check that it exists in our path if we're just running the unit tests. commit 98b3a8dfb10944ff3ec7b41a3b1f39297173bf26 Author: Damian Johnson Date: Wed Jun 13 08:39:57 2012 -0700 Dumbing down syntax for python 2.5 Evidently even with the 'with' keyword's future import it still chokes on comma delimited entries (instead you need to do nested 'with' blocks). Ick. Oh well, if that's all we need to maintain 2.5 compatability then I'll be happy. It's a pity we can't tell python 'run in 2.5 compatability mode' to check for these things. I doubt that we'll remember to run against a 2.5 instance to keep stem working there. Oh well, guess this is a problem that'll need to be addressed when figuring out a release procedure. commit 408dbe96e88767ca55688d196287e41c57345219 Author: Damian Johnson Date: Tue Jun 12 09:58:08 2012 -0700 Adding --test argument to selectively run tests Adding an argument that'll only run tests matching the given module prefix. This is to make it faster to write new tests (so we don't need to wait a minute for the full integ suite to run each time). Also adding examples to the '--help' output and rearranging the options so '--config' is at the end (it isn't especially useful and may be dropped in the future. commit b8f4d3c75f0d4c085dcba02f47f4808c301e235a Author: Damian Johnson Date: Tue Jun 12 08:49:01 2012 -0700 Couple fixes for prior controller changes Two minor bugs... - our version parsing was broken for versions without a space in them - we were quoting the runner password constant rather than using it, breaking the RUN_PASSWORD target Also some minor whitespace tweaks and moving imports to the start of the file (we're only doing method level imports if doing otherwise would break things). We might change our import style later to resolve our circular imports, but that'll be a project-wide change. commit 843f5568d42d4568459d0a4597e078380d5fa76a Author: Beck Date: Tue Jun 12 16:41:12 2012 +0800 Fixed indentation errors. commit c4c242f589b39519ec7d71841fd4679ecac498af Author: Beck Date: Tue Jun 12 16:03:46 2012 +0800 Added integ tests for convenience methods. commit 1d8ee15ef127073c384021c81e0ea07951d2c2fe Author: Beck Date: Fri Jun 8 00:31:56 2012 +0800 Move all convenience functions to Controller class, and minor changes on them commit c4b5917be15fdfce5921b2dd704f025be59ae97d Author: Beck Date: Thu Jun 7 10:45:57 2012 +0800 Added convenient methods in BaseController and Controller. commit fc327f29f66a84850ab89a32f3dd358756797029 Author: Beck Date: Fri Jun 8 00:31:56 2012 +0800 Move all convenience functions to Controller class, and minor changes on them commit de09fd34f4246b3b0323998de3829962bcad0dc7 Author: Beck Date: Thu Jun 7 10:45:57 2012 +0800 Added convenient methods in BaseController and Controller. commit c1450e440e388295a329e84b3e935bf86f88e673 Author: Damian Johnson Date: Mon Jun 11 19:01:34 2012 -0700 Whitespace checker wasn't checking non-tests The whitespace checker was being ran with its default target, which is the directory of that file (in this case, test). Explicitely running over everything else, and fixing the issues that it found. commit c52042c44b5a157dceb6013743fdea40bbc3f52c Author: Damian Johnson Date: Mon Jun 11 10:01:17 2012 -0700 Adding future 'with' import and checking for it Stem aims for python 2.5 compatability, but in that version the 'with' keyword wasn't available by default and would only function with a 'future' import. This has the possability of being a continuing pain in our ass so adding this as a check to the check_whitespace module. Admittedly this isn't a whitespace issue, but that module is just so damn conveninent for lintian style issues too... While doing this I discovered that we aren't running the checker over the stem module (just the tests, it seems), so something else to fix... commit 65351e9eea2a30e81632e045c0b2825d0573719c Author: Damian Johnson Date: Sun Jun 10 15:16:57 2012 -0700 Converting test.* contents to reStructuredText The test contents aren't currently included in the stem documentation, since they aren't generally of interest to users. However, might as well do its documentation as reStructuredText for consistency's sake. I'll probably include just the framework utilities with our documenntation later to help people hacking on stem. commit 9654beb5acb7ad0e8e90bbdf7b9e186ccd946654 Author: Damian Johnson Date: Sun Jun 10 14:24:33 2012 -0700 Adding extra-info digest() method Adding digest() methods for extra-info descriptors. I'm also swapping the server descriptor counterparts to provide a hex encoded digest as per Karsten's suggestion. commit 860cd87212d6caca7f23423940ed1807cf3f0f84 Author: Damian Johnson Date: Sun Jun 10 13:58:23 2012 -0700 Integ test with bridge extra-info content Adding a bridge extra-info descriptor from metrics. I'm currently being a bit lazy and only parsing some of the mapping fields. The rest should be covered by unit tests, so I'll only add the rest if this ends up bitting us. commit 737018b7ee3da5b457689404376a7937de5a016f Author: Damian Johnson Date: Sun Jun 10 13:33:21 2012 -0700 Making relay-signature field only apply to relay extra-info Parsing and handling the relay-signature field as only an attribute of extra-info descriptors for relays. commit 7947d69d4e20ca402e9d1d35fe6e62012d5e61fe Author: Damian Johnson Date: Sun Jun 10 12:31:14 2012 -0700 Separating relay and bridge extra-info descriptors Creating separate classes for relay and bridge extra-info descriptors. At present they're identical to their common parent (ExtraInfoDescriptor), adding their differences next. commit 560923cb7b572d02046c6ca2bd5eb4502fa591b3 Author: Damian Johnson Date: Sat Jun 9 18:18:00 2012 -0700 Revisions to change for SAFECOOKIE support The SAFECOOKIE change is pretty big so, unsurprisingly, it has needed quite a few changes. Many are stylistic nitpicks, though this also includes several more substantial changes including... * The SAFECOOKIE AuthMethod was added, but the ProtocolInfoResponse's _parse_message didn't recognize it causing it to still be listed as an unknown authentication method. Also, we called send/recv on the controller in authenticate_safecookie() at one point rather than using _msg, causing it to break when called with a Controller. I'm a tad puzzled by these... were the integ tests ever ran with the RUN_ALL target? * Slightly reducing the exhaustiveness of the authentication unit test in the name of runtime. The test exercises each combination of exceptions for each exception type which led to a combination explosion when we added SAFECOOKIE to the mix (raising the unit testing runtime from around two to twelve seconds). Made some minor tweaks to bring the runtime back down without really impacting what we test much. * Changing the CookieAuthFailed attribute that indicates if it arose from SAFECOOKIE or COOKIE auth to be a boolean. Providing the AuthMethod would only make sense if we added even more variants of cookie authentication (possible, but not anywhere on the radar). * Added AUTH_SAFECOOKIE to 'stem.version.Requirement'. This turned out to need some fairly big changes to how we check for version requirements (see prior commit). * Few issues with the integ tests, including calling 'remove' on a tuple and a bug where it would accidently remove all auth methods from the protocolinfo response before calling authenticate(). * We only removed AuthMethod.COOKIE form a protocolinfo response's auth_methods when the cookie file was missing. * Few unused imports and lots 'o whitespace issues. I should probably improve the whitespace checker a bit. * Toned down the 'warning, we're under attack!' errors. The whole point of safecookie auth is simply to avoid providing the raw cookie contents to the thing we think is tor. Nothing about this handshake should be consitered to be an 'attack', nonce mismatches are more likely to just indicate some sort of bug. * No tor reply can be empty (if it was empty then we... well, wouldn't have read anything from the socket). Hence the ControlMessage can't be empty. It'll be a pita to check for this in every reply, so checking that as part of the ControlMessage constructor. * Added helper function to check if a value is hex digits. We do this a lot, so no need to repeat the regex everywhere. * Looks like we could misidentify IncorrectCookieValue verses CookieAuthRejected exceptions when we had both cookie and password auth. * I was a bit confused about what the AUTHCHALLENGE unit tests for an invalid response were attempting to exercise, so replaced with a shorter and simpler test for missing components. commit 7ecd7c6a63baf08b638a378a7f4c237ec7ceeb84 Author: Damian Johnson Date: Sat Jun 9 14:20:56 2012 -0700 Expanding capabilites of tor version comparison Requirements were simple tor versions which was fine for simple 'greater than' comparisons, but broke down for slightly more complicated requirements such as 'at least version X in the 0.2.2 series or version Y in the 0.2.3'. Introducing a 'meets_requirements' that will allow us to compare with arbitrary rules. For now we're just comparing with Versions or VersionRequirements, but could be expanded later if we need to. commit 682cbb4a2b36b0d226ea2a7e8d9bb527fdb2e28e Author: Ravi Chandra Padmala Date: Tue May 8 01:05:20 2012 +0530 Implement Safecookie support in Stem commit 63c2ddb95e4c96e509e030240df12266012ce844 Author: Damian Johnson Date: Wed Jun 6 10:09:21 2012 -0700 Descriptor typo corrections Fixes by Karsten commit ee6e475a347277ff6a387107219fa5ba79a7ae99 Author: Damian Johnson Date: Wed Jun 6 09:53:03 2012 -0700 Targeting reader test at /usr rather than root dir I've been running the reader's test_stop() integ test by running over the root direcctory because I simply needed something 'big' to keep the test occupied for a second or two. This turned out to inadvertantly fuzz the reader which is sorta a good thing since it discovered that block devices and encrypted partitions can be slow to read, causing stop() to block for a bit. However, we don't want to run over people's private stuff, and slow reads on encrypted partitions probably isn't a problem that we want to solve anyway, so running over '/usr' instead. commit da1e0a75082efef9e24f86fbba68e6c47116619c Author: Damian Johnson Date: Wed Jun 6 09:23:11 2012 -0700 Correcting incorrect variable in logging Issue caught by Karsten. commit ebed653be563f2e16c384d9d3b235ad2ecb11566 Author: Damian Johnson Date: Wed Jun 6 08:47:54 2012 -0700 Skipping non-regular files when the reader crawls When the reader encounters block devices it can get stuck, causing the integ test for stop() to fail. Caught by nickm. commit 38adeda918ffc1367ec8eb8dc0002bd737ad95c6 Merge: 2a21ec5 8badba1 Author: Damian Johnson Date: Tue Jun 5 21:28:53 2012 -0700 Merging changes for sphinx documentation commit 8badba152f5d9b2a83a9b9893c328cd23dd219a2 Author: Damian Johnson Date: Tue Jun 5 21:22:49 2012 -0700 Targeting contents at top of modules For modules like stem.response and stem.util I was linking to the first anchor, which isn't the behavor I wanted (skips the title at the top, which is a nicer landing). I resisted making this change earlier because it means that those links have different styling from those above, but now that there's more of them I'm happy with it this way. commit 4819e2fc97f76424a3dead741d327be711f53a8d Author: Damian Johnson Date: Tue Jun 5 21:15:37 2012 -0700 Converting stem.descriptor.* to reStructuredText Fingers so sore... commit e13ea63c09e766d15c36983c9b37f6adde1a2bde Author: Damian Johnson Date: Tue Jun 5 20:14:39 2012 -0700 Removing double backticks from None documentation Double backticks should be monospaced, but with the haiku theme it's an inverted background with is far more noticeable. I didn't mean to accent 'None' this much so dropping the formatting from it. commit 12af869b1f669117de736c86f3e094ad2e467be4 Author: Damian Johnson Date: Tue Jun 5 20:08:57 2012 -0700 Converting stem.util.tor_tools to reStructuredText commit fa60694958ed41a323a099e844ce92fb48faa373 Author: Damian Johnson Date: Tue Jun 5 20:06:31 2012 -0700 Converting stem.util.term to reStructuredText commit 3319642a609cd4995724877b90af76cd44067085 Author: Damian Johnson Date: Tue Jun 5 20:02:20 2012 -0700 Converting stem.util.system to reStructuredText commit 57e11a1345ed5547424b03fc5f7d9e663c4661ea Author: Damian Johnson Date: Tue Jun 5 19:53:00 2012 -0700 Converting stem.util.proc to reStructuredText commit 1e04a08e75ae73f204c6ba3d4f8eef33104973f5 Author: Damian Johnson Date: Tue Jun 5 19:40:57 2012 -0700 Converting stem.util.log to reStructuredText commit d5ef02b04eccebef5d5bbf89003431c0d41f1401 Author: Damian Johnson Date: Tue Jun 5 19:33:10 2012 -0700 Converting stem.util.enum to reStructuredText commit c1392cd8b3322be4b39980dea54d5b224ed7062f Author: Damian Johnson Date: Tue Jun 5 19:26:54 2012 -0700 Converting stem.util.connection to reStructuredText commit 1b23a2da711b202a9fba24f47c173f09024efa34 Author: Damian Johnson Date: Tue Jun 5 19:22:52 2012 -0700 Converting stem.util.conf to reStructuredText commit 68dc6af5c74cd8b733b07542c88f5af96bb146be Author: Damian Johnson Date: Tue Jun 5 09:31:55 2012 -0700 Converting stem.response to reStructuredText commit c350257b05455509ab99c9019747e63553a668d5 Author: Damian Johnson Date: Tue Jun 5 08:25:48 2012 -0700 Converting stem.socket to reStructuredText commit cbcf257ee91f78a58984a2335814a589c51f14e9 Author: Damian Johnson Date: Tue Jun 5 07:54:20 2012 -0700 Converting stem.control to reStructuredText commit 1cfad693de89bc4769605f21d52ef9ba4f38205b Author: Damian Johnson Date: Mon Jun 4 10:04:45 2012 -0700 Adding module overviews back to docs One bit of documentation that I've found very useful, and sphinx lacks, is a brief summary of a module's contents. Hence adding that back to the headers. commit c03ebb23ae39f9c9b68a1471dbf3debb30c399a7 Author: Damian Johnson Date: Sun Jun 3 21:26:21 2012 -0700 Defaulting connect_* to provide a Controller As a TODO comment mentioned, the connect_* convenience functions were supposed to provide a higher level controller class when one was available. It now does. commit f46b5c7c0e543d90f26155e472a1658827c5f8fe Author: Damian Johnson Date: Sun Jun 3 21:23:49 2012 -0700 Converting stem.connection to reStructuredText commit 1ccf6be7dec076afdf4c4a4150c38b5065a8f17e Author: Damian Johnson Date: Sun Jun 3 16:28:07 2012 -0700 Switching to the haiku theme The haiku theme is lighter, and lacks a left-hand bar which makes the content far less cramped. All in all, looks far nicer for the content that we have so far. commit 438dc79a512879473499211988a86e08099720de Author: Damian Johnson Date: Sun Jun 3 16:21:26 2012 -0700 Missed linking a funcion commit ecc0e86d46ac2db77d45d46af5ac50168befa6c9 Author: Damian Johnson Date: Sun Jun 3 16:18:00 2012 -0700 Converting stem.version to reStructuredText commit 6651e8304a3cee2ffed344371c830a6014853201 Author: Damian Johnson Date: Sun Jun 3 15:38:38 2012 -0700 Converting stem.process to reStructuredText commit 241359c043c3ffb2d2d15d24cacad6c9d7fd0ae7 Author: Damian Johnson Date: Sun Jun 3 13:13:49 2012 -0700 Removing autogenerated rst files via clean target The html target creates some rst files, so removing them when 'make clean' is ran. commit a3a7b2055144ae4ccb06e464510df211fc4fb43a Author: Damian Johnson Date: Sat Jun 2 19:19:24 2012 -0700 Skip recreating unchanged docs Greatly decreasing the time it takes to generate documentation when pydocs haven't changed. Sphinx is smart enough to avoid running against files whos last-modified timestamp hasn't changed, but not smart enough to check the hash of the content. Hence using rsync to avoid modifying our pydoc derived content unless the pydocs have changed. commit 58566d6b2e20899c82206dba50d2aebfa008a96c Author: Damian Johnson Date: Sat Jun 2 18:55:25 2012 -0700 Generating sphinx docs for all stem modules Using the sphinx-apidoc command to generate documentation for all stem modules. I'm doing this via the make file to simplify the documentation creation process (you still just need to run 'make html'). Also removing the windows make.bat file since I'm not gonna be keeping it up to date. We can resurrect it if someone volunteers to maintain it. commit af83350dcf62f819d8e84c6e34051dc3ff509add Author: Damian Johnson Date: Sat Jun 2 13:35:12 2012 -0700 Adding basic stem intro to sphinx index commit bad7ae65902f648c3127de66a35a95223b6582a6 Author: Damian Johnson Date: Sat Jun 2 11:41:54 2012 -0700 Configuring sphinx documentation attributes Setting attributes and toggling attributes for the autogenerated sphinx documentation. This seems like a pretty slick tool, though I'm still puzzling out how to make it index all of stem. It seems to only register modules that I explicitely tell it about - I'm probably missing something... Also still deciding between the default and haiku themes. Ruled out the others. commit 5c9579626a131e8bd17c283e0904525e30fe4488 Author: Damian Johnson Date: Sat Jun 2 11:06:42 2012 -0700 Adding module metadata (author, version, etc) commit ff5257f2e848ed84fa3e8e23b60149d464262c0e Author: Damian Johnson Date: Thu May 31 10:01:30 2012 -0700 Ignoring sphinx build output Build artifacts from sphinx are placed in 'docs/_build/*', so having git ignore it. commit feeac963d1383b9eb5a08251dbd329ff64f1b846 Author: Damian Johnson Date: Thu May 31 09:57:51 2012 -0700 Initial sphinx autogenerated content Results right after running 'sphinx-quickstart' commit 2a21ec5848df9d1d5a520abbb6d5927a804acd09 Author: Damian Johnson Date: Mon May 28 17:07:57 2012 -0700 Moving ControlMessage and ControlLine into stem.response The ControlMessage and the ControlLine instances in it are the... well, messages that we get from tor. It belongs in stem.response so moving it there. Functionally this is fine, and I'm happy with the tests in a functional fashion as well. However, all this splitting and refactoring has made the tests a mess in terms of what the test functions belong to (some test.unit.response tests are checking stem.connection functionality, for instance). More cleanup work to do there... commit d3fe6a1ed751a60b395f80aa7b4663ab6f78d230 Author: Damian Johnson Date: Mon May 28 16:38:30 2012 -0700 Cleaning up response classes General refactoring of the response classes. commit be06640765510b42912f074ebad6acb7cf4d0bac Author: Damian Johnson Date: Mon May 28 15:52:18 2012 -0700 Renaming getinfo 'values' attribute to 'entries' The name 'values' has bugged me for a while because it's a dictionary, so users might end up typing things like "my_getinfo_response.values.values()". Renaming it to something that's still generic but without these issues. commit 6015799baf9c0e6d3a618c948c32c27b2d7e1fc3 Author: Damian Johnson Date: Mon May 28 15:48:32 2012 -0700 Being more anal about 'OK' status checking Refusing to parse messages that lack a 250 response on all lines, and checking for a 'OK' status on the last line (and only the last line). commit 152059f7d687c93326e039c954bbfa72a9356323 Author: Damian Johnson Date: Mon May 28 15:43:35 2012 -0700 Adding an is_ok() method to controller responses Method added by Ravi during his safecookie work. Stealing it a little early since I want it for refactoring that I'm doing. commit 72aed261dceb560d05563fab1af4e6f549e2bdfd Author: Damian Johnson Date: Mon May 28 15:23:52 2012 -0700 Single retry for socket connection I just had integ tests fail due to an interrupt while trying to connect to the tor control socket. This is the first time that I've seen it, so this isn't much of a concern, but connecting to a socket is idempotent so we can do with retrying it once if we fail. commit 1d7ab654ec473a5e54ea3253f82abdcadc93158f Author: Damian Johnson Date: Mon May 28 15:12:16 2012 -0700 Moving tor response classes into their own module As we add more response classes it'll be messy to sprinkle them all about the codebase. Making a single 'stem.response' module that'll contain them all. These could probably do with some more love so I'll next see if I can make them any tidier. commit 717c5aaecc8e61cee5ca8f1bae80b3eade5f6985 Author: Damian Johnson Date: Sun May 27 20:29:51 2012 -0700 Implementing Controller.get_info Implementation and testing (both unit and a little integ) for GETINFO queries. There's still several todo notes to clean up, but the method itself is done. commit 045a5ec8774074eb92066571dfc661c4098bd77a Author: Damian Johnson Date: Sun May 27 14:35:15 2012 -0700 Adding Controller class Making a class for the general controller. As a todo comment mentiond, moving the from_port/from_socket_file helper functions there and revising the tests to reflect that. commit 683078f749ca6627f8ff65c4562697b6e1034689 Author: Damian Johnson Date: Fri May 25 09:27:07 2012 -0700 Minor error in logged message s/server/descriptor commit 40a971105ff162d0caa9e0a1c5fa3c0f7d3c2b72 Author: Damian Johnson Date: Fri May 25 08:59:03 2012 -0700 Validate that signing key hash matches fingerprint When parsing server descriptors checking that their fingerprints match a hash of their signing key as part of validation. This requires the rsa module which is neither built in, nor is it packaged for debian distros. Installation of it is easy, but requires pip or easy-install. Instructions are available at... http://stuvel.eu/files/python-rsa-doc/installation.html It looks like python's builtin crypto module might be capable of doing this as well... http://stackoverflow.com/questions/5000434/python-rsa-library https://www.dlitz.net/software/pycrypto/api/current/Crypto.PublicKey.RSA._RSAobj-class.html#exportKey However, those instructions include usage of a private member and the function that they suggest dosn't exist on my system (python 2.7.1), so I feel pretty justified in saying "the pycrypto builtin is crap for this use case, and patches welcome if someone can figure out how to make it work". All credit for this patch goes to Beck, who's been diving into the descriptor crypto on... https://trac.torproject.org/projects/tor/ticket/5810 commit f7fb726cc3dea8bfd294833b151117858490802d Author: Damian Johnson Date: Wed May 23 10:00:04 2012 -0700 Removing expectation that bridge nicknames are scrubbed Karsten plans to stop scrubbing bridge descriptor nicknames, so removing that expectation from the 'is_scrubbed()' method. commit e7e03d2f61d6dcc7bc5e5ad4dee91c37a814ee16 Author: Damian Johnson Date: Wed May 23 09:52:23 2012 -0700 Supporting bridge descriptor's router-digest field Scrubbed bridge server and extra-info descriptors will soon have a 'router-digest' line which contains the server digest. This is a value that we can't calculate for ourselves because, of course, the descriptor's contents have been modified. I'm flagging this as being a required field. This does not yet parse it in extra-info descriptors since it neither has a digest() method nor did I realize that bridge extra-info descriptors were special. commit f1d58b21fd5fe7dcd8b5330627f7521ac86ffbc0 Author: Damian Johnson Date: Sun May 20 15:24:22 2012 -0700 Citing ticket 4817 in related pydocs The pydocs warn users of authenticate_cookie and authenticate_password that we can't reliably differentiate certain failures. Citing the related ticket. commit ce07df9962bc464c338cbee0753e40169f801505 Author: Damian Johnson Date: Sun May 20 15:11:55 2012 -0700 Stating runtime when launch_tor timeout test fails Just had a transient testing failure from test_launch_tor_with_timeout. There's nothing to go on since we don't even know how long the test took, so providing that information on failure. Might need to mess with the bounds a bit if this happens again. commit 688f321287f5e9c53451dd91608dc3aa5088a621 Author: Damian Johnson Date: Sun May 20 15:05:10 2012 -0700 Disabling launch_tor timeout when on windows On windows the python signal module lacks a SIGALRM attribute, breaking launch_tor. Disabling timeouts on windows to avoid this issue - ideally we should find an alternative method for timeing out the function on windows but that can wait. Caught by pythonirc101. Issue is discussed on... https://trac.torproject.org/projects/tor/ticket/5783 commit 4d6c8c8943d9540d5fcc6c39c503ca523e4683d3 Author: Damian Johnson Date: Sun May 20 14:49:31 2012 -0700 Adding separate funtion for launching tor with tmp torrc Like Vidalia, adding a function that creates a torrc, launches tor with it, then removes it from disk. This is preferable to using the commandline arguments since the those shouldn't be used for substantial data nor the hashed password. This replaces launch_tor()'s options argument, though I added a more genral args replacement that can be used to do the same if the user wants. commit 301401360337e4c02a5fd5e4e8520cc1ecf88633 Author: Damian Johnson Date: Fri May 18 10:25:49 2012 -0700 Utilising metrics descriptor type annotations Using the @type annotations added by Karsten to the metrics descriptors as per ticket 5651. commit aed91aa634b9f960e87296b9e631a2c6aa05f2c0 Author: Damian Johnson Date: Fri May 18 08:33:13 2012 -0700 Removing GETINFO version queries from integ tests Sathyanarayanan spotted a couple points in our integ tests where we make 'GETINFO version' queries, then compare against the parsed tor version. This es no bueno because the parsed version just contains what, according to the version-spec, is the version and the getinfo query includes extra data like the git commit id it came from. This is actually an issue we encountered before and was fixed. However, since then these queries slipped back into new tests since they seem like an easy way of exercising a socket. Oops. :) Tested by checking out the git commit that gsathya is using (ef0bc7f), then running stem's integ tests against that. This fixes ticket 5918 and might address 5917 too (I'm not sure why the later is hanging - that's not happening for me though the test did have an error). commit 47c9ddd8e460488abad58afb8893a1524e37835c Author: Damian Johnson Date: Thu May 17 09:59:24 2012 -0700 Included last tor warning or error in launch_tor exceptions Adding the last tor waring or error message that's probably useful for figuring out the reason why tor failed to launch. commit bca28c1e409a318bf9303a635530382c13c17115 Author: Damian Johnson Date: Thu May 17 09:29:04 2012 -0700 Replacing enums with class instances for connect_* functions The connect_port and connect_socket_file convenience functions accept a controller type that we would like to return. I had been using an enum for this argument, but this means that we both (a) need to manually update it when adding controllers and (b) can't recognize controller the user makes. Replacing this with a class instance, like what TorCtl does. Tested by running against a script for printing BW events. commit b69db7feba640967c2bb6f2740d0aac2bf32666f Author: Damian Johnson Date: Thu May 17 09:19:28 2012 -0700 Helper function to only run tests once Adding a helper function to skip tests if they've been run before. This is needed because some of our tests take a long time to run and are uneffected by testing targets, so there's no point in wasting time on running them repeatedly. commit dd597ba4855197653c1d9cd543c3bd6c611432a5 Author: Damian Johnson Date: Tue May 15 08:36:56 2012 -0700 Removing platform dependency note from process module Removing comment that sets the wrong expectations of the process module's capabilities. It needs to be platform independent since it's used to run our integ tests, and when it isn't that's a bug. That said, Windows is unfortunately having trouble with it at present... https://trac.torproject.org/5493 https://trac.torproject.org/5783 commit e2974e85a6560d204b674e7179a13074a3494536 Merge: bfb3973 7d96e78 Author: Damian Johnson Date: Sun May 13 17:12:27 2012 -0700 Merge extrainfo descriptor support commit 7d96e78e90e455019f674eff2a611449107eb109 Author: Damian Johnson Date: Sun May 13 17:02:28 2012 -0700 Unused constructor annotations arg Cached extrainfo descriptors don't have annotations, so excluding it from the constructor. Functionally it was already unused, just forgot to remove it from the args. Caught by pylint. commit 5bbf40dbc9a7374f555b4ec7aaf838be6156c9fc Author: Damian Johnson Date: Sun May 13 16:57:53 2012 -0700 Missing util import This should have caused an error, but for some mysterious reason imports throughout stem have a wider scope than I'm intending (causing pylint to compain about lots of re-imports). Definitely an error - we don't want to rely on whatever buggy magic made it work. commit 1b2ebb2d6624de2133f58386d0b66a0e0da6b962 Author: Damian Johnson Date: Sun May 13 16:53:01 2012 -0700 Checking that cached extrainfo is all recognized Asserting that there aren't any unrecognized extrainfo descriptor lines in the cached file to help catch new additions. commit 47acd940498dfe605324bd2e4f08826c7f8c5019 Author: Damian Johnson Date: Sun May 13 16:50:28 2012 -0700 Parsing 'exit-*' extrainfo lines Handling the last extrainfo attributes... * exit-stats-end * exit-kibibytes-written * exit-kibibytes-read * exit-streams-opened commit c7ea62037adef91fe11728caab75e19096c69d52 Author: Damian Johnson Date: Sun May 13 16:24:37 2012 -0700 Parsing conn-bi-direct extrainfo lines Another special snowflake extrainfo attribute. Parsing and added a test. commit 97fad7b6f7b6113fdb3e5ca8925224901748a98c Author: Damian Johnson Date: Sun May 13 15:46:58 2012 -0700 Citing ticket about negative cell-processed-cells values Comment to explain why we're accepting negative values, and citing the related ticket. commit 153a98eca741d3185e7d20c5e3566763a54384bd Author: Damian Johnson Date: Sun May 13 15:06:29 2012 -0700 Parsing 'cell-circuits-per-decile' extrainfo lines Handling the last of the 'cell-*' attributes. commit 8dd9d118977366ef8a17ad08a39809cbee735df4 Author: Damian Johnson Date: Sun May 13 14:47:40 2012 -0700 Parsing some more cell-* extrainfo lines Handling the cell-processed-cells, cell-queued-cells, and cell-time-in-queue entries which are all numeric lists. I was a little surprised to find negative cell-processed-cells values in the wild. commit 00e7ce52acc2ddb8f45ccae29469d3463ba39010 Author: Damian Johnson Date: Sun May 13 14:13:57 2012 -0700 Parsing cell-stats-end lines Not much to this attribute. The next few are slightly different, though. commit 86bce1e8db0799efbca86af72a4ef3c0a27f9674 Author: Damian Johnson Date: Sun May 13 13:46:49 2012 -0700 Parsing entry-stats-end and entry-ips lines Handling a couple more extrainfo attributes. These were trivial thanks to earlier refactoring to make common value types easy to parse. commit bbf0e80cf889cc4b0738acf93c698fc54720c767 Author: Damian Johnson Date: Sun May 13 13:35:33 2012 -0700 Parsing dirreq-v*-direct-dl and dirreq-v*-tunneled-dl lines Parsing four extrainfo descriptor fields related to directory mirroring stat collection. These are similar to dirreq-v*-resp in that they're key=value mappings, with largely known key sets. commit f7527756275e71b82d426394a74fc0f3ac9824b8 Author: Damian Johnson Date: Sat May 12 20:10:12 2012 -0700 Parsing 'dirreq-v2-resp' and 'dirreq-v3-resp' lines These lines were a bit more substantial to implement since callers would want enums for their statuses, but not all statuses are necessarily known. Looks like the 'dirreq-v*-direct-dl' and 'dirreq-v*-tunneled-dl' lines will be similar. commit 63acd8169345270df0f1f6b545f67b5453b555fb Author: Damian Johnson Date: Sat May 12 18:11:29 2012 -0700 Renaming dir_v*_reqs to dir_v*_requests It's a little understandable that we'd want to squeeze out a few characters in the descriptors (er, or it would if we didn't compress them), but there's no point for us to do the same. commit e0ad461440361736b8ce340786433fe6dfaf44f2 Author: Damian Johnson Date: Sat May 12 15:39:15 2012 -0700 Parsing dirreq-v2-share and dirreq-v3-share lines Couple more extrainfo descriptor fields. Ye gods there a lot. commit b82dca6e2e2543cfa19e9d15be1d8ba89227407d Author: Damian Johnson Date: Sat May 12 15:15:55 2012 -0700 Parsing 'dirreq-v2-reqs' and 'dirreq-v3-reqs' lines Adding support for a couple more extrainfo fields. commit b9cd295c09afe662247cc6a89442b40860b9157b Author: Damian Johnson Date: Sat May 12 15:07:02 2012 -0700 Reorganizing extrainfo fields by value type Grouping both parsing and unit tests by the value format to make it more maintainable and easier to add the dozens of other attributes. Also using a 'dir_' prefix for all directory mirroring attributes rather than 'dirreq_' (I was previously being inconsistent on this). This expands the testing to cover more attributes, along with adding support for the "dirreq-v2-ips" and "dirreq-v3-ips" lines. Those lines revealed some misses in how I was parsing the maxmind locales. In particular I was missing '??' and numeric locales like 'a1' which it uses for an 'Anonymous Proxy'. commit bff56e18ecde250eb9ad2eba7db5e29d1817bd8f Author: Damian Johnson Date: Sat May 12 13:57:53 2012 -0700 Revising descriptor pydocs Revisited the ordering, defaults, and descriptions of descriptor attributes to be shorter and more user friendly. commit 2ddffb3d961bcc01829ad58c96b8fa236a35cfff Author: Damian Johnson Date: Sat May 12 12:42:25 2012 -0700 Dropping measurement *_line attributes Removing attributes like read_history_line which contained the raw value for the read-history line. These are left over from when I made no effort to parse the line, and are useless since we have their parsed values. Minor changes to our pydocs so these related items are better labeled. commit 6f40a5e87228ea8f0af76d2aeba859025ce468b2 Author: Damian Johnson Date: Sat May 12 12:32:09 2012 -0700 Parsing dirreq-stats-end extrainfo lines Parsing and test. This is the same format as bridge-stats-end so combining those tests. commit c33c4a0f0e6b98898622d0929e531d12abdce72a Author: Damian Johnson Date: Thu May 10 09:29:04 2012 -0700 Parsing bridge specific extra-info params Parsing, validation, and tests for bridge specific lines... - geoip-start-time - geoip-client-origins - bridge-stats-end - bridge-stats-ips I haven't yet seen an actual bridge descriptor, and I'd like to add one as an integ test but that can come later. commit 15a25f0b05eacfff21309715a4838d4ccd6f11f9 Author: Damian Johnson Date: Thu May 10 08:31:26 2012 -0700 Parsing geoip-db-digest lines in extrainfo descriptors commit d3f7c00fa6cf2573f30551589338e4d42d61465a Author: Damian Johnson Date: Thu May 10 08:00:36 2012 -0700 Accepting additional args on extra-info line Forgot that the spec says that extra arguments should be accepted and ignored. commit 0737bada04f71434e9169bc964257dffce8f90c3 Author: Damian Johnson Date: Wed May 9 10:01:32 2012 -0700 Parsing extrainfo dirreq-read/write-history lines Handling the dirreq-read-history and dirreq-write-history. These are almost identical to the read-history and write-history lines so tieing it into that parser. This is the last field I need to parse the metrics descriptor I'm using for an example in the integ tests. commit 7178c5cb0a831d217c969ad38453dfdc85630983 Author: Damian Johnson Date: Wed May 9 09:27:16 2012 -0700 Renaming read/write_history to read/write_history_line The 'read_history' variable name was a bit vague, and generally the attribute isn't useful unlike its read_history_* counterparts. Renaming the variables to make it clear that read_history_line is the raw content for the read-history line. commit 9d24a9090a00a8686b6eae3258bb6af84cd06087 Author: Damian Johnson Date: Wed May 9 09:11:46 2012 -0700 Only checking cached descriptor existance when needed Maybe once upon a time we had multiple tests that ran over the cached descriptors? I don't recall it, and we definitely don't right now so only checking for the existance of the cached-descriptor and cached-extrainfo files when we want to test against them. commit 02451060cf8d5fcc8834d455e0c12a24b08b876b Author: Damian Johnson Date: Wed May 9 09:02:51 2012 -0700 Helper function for getting descriptor resource paths Making a common helper function for getting the paths of resources in the descriptor data directory rather than making everyone figure out a DESCRIPTOR_TEST_DATA constant. commit 46fb6b272e45ab8c569e1439a53a985c5a8d827d Author: Damian Johnson Date: Wed May 9 08:54:26 2012 -0700 Letting runner's get_test_dir() provide resources The most common use of the integ runner's get_test_dir() method is to get a path for resources in that directory. Providing an optional argument to get that path rather than making the caller use os.join(). commit 406205c2549e855140723bc42af874248a6b3026 Author: Damian Johnson Date: Wed May 9 08:42:48 2012 -0700 Extra-info integraion tests ExtraInfo counterparts for parsing a metrics descriptor and tor's cached file. There's some common bits with the server descriptor tests that I plan to move out. commit fa6d81cf69d5590f6bcb0ad94cf34a7b313f02c2 Author: Damian Johnson Date: Wed May 9 08:09:00 2012 -0700 Parsing read/write-history lines in extrainfo descriptors Copying the code for parsing from the server descriptors. I suspect that this isn't yet exercised by the tests - I'm next gonna improve our integ tests then check if this would be appropriate for a unit test too. commit b8e08ce69d30d10cfa6879b7205f62e988fea0e1 Author: Damian Johnson Date: Tue May 8 09:53:13 2012 -0700 Helper function for parsing timestamp/intervals Several extra-info descriptor lines are of the form... 'YYYY-MM-DD HH:MM:SS (NSEC s) *' So making a helper function to parse it. Using it for the server descriptor's read/write-history entries and it improves code readability quite a bit... commit 9042c172f8544912888d174ab18b4a19882db7e3 Author: Damian Johnson Date: Mon May 7 09:27:06 2012 -0700 Reverting change to recognize 127.0.0.1 as bridges Reverting change 477d448 since Karsten reports that bridges with that scrubbing scheme should no longer exist. commit 77a498ae31d0c95d1cdaa88752965fcea870990d Author: Damian Johnson Date: Mon May 7 09:23:09 2012 -0700 Unit tests for extrainfo descriptors Few basic unit tests for extrainfo descriptors. These are done in a similar fashion to the server descriptors. I'm highly tempted to refactor out some common bits, but at present that would probably hurt code readability more than it would help. This'll change if I keep using the same type of helpers for descriptor unit tests. commit 7a91e7d85bbc24df130147cc8789e87a5c0f8911 Author: Damian Johnson Date: Mon May 7 08:39:34 2012 -0700 Parsing extrainfo published and router-signature lines Parsing the remaining two mandatory fields for extrainfo descriptors. These fields are identical to what's in server descriptors. commit bd1d3f345f25835a52813449f4bb45bdddf32eaf Author: Damian Johnson Date: Mon May 7 08:37:34 2012 -0700 Bumping runtime of async event test by a second Testing for asynchronous events relies on BW events happening every second by checking that we get two of them. This is not reliable since we don't have a hard assurance about its rate. Giving the test an extra second to make it more reliable. commit 234a90a79624e5503c4bf65db01c14e54fb9b8de Author: Damian Johnson Date: Sun May 6 20:52:21 2012 -0700 Basic constraint checking for extra-info descriptors Parsing the 'extra-info' line and performing basic validation about extra-info descriptor fields (that we have required fields, most fields don't appear more than once, etc). I'll add some unit tests for this after I've added parsing for all of the mandatory fields. commit 13944b062ff26a5efa178ff8d5552ff1d049574b Author: Damian Johnson Date: Sun May 6 20:08:53 2012 -0700 Skeleton for extra-info descriptors Basic module for parsing extrainfo descriptors. This doesn't actually do any parsing yet, just turning the raw content into a ExtraInfoDescriptor instance. This abstracts code we'll need out of the server_descriptor module so we'll be able to use it. What code there is here is exercised via the runner tests. commit 7930b74b6586220cd02ee816107a85fc1ba7e4ae Author: Damian Johnson Date: Sun May 6 13:38:45 2012 -0700 Missed a few more 'version 3' labels commit bfb3973049c72a9bd6de951566417eca7e132e70 Author: Damian Johnson Date: Fri May 11 09:28:51 2012 -0700 Integ tests for launch_tor I've been leaving launch_tor untested because it was exercised in order to run the integ tests and any tests involving it would take a long time, on the order of a dozen seconds! Hey, it's a long time when you run this as often as I do... It's an important function and has more options than what's exercised for running integ tests so adding some tests for it. commit f3c5f334ce84621547139ae70eedb2c3fbfa2076 Author: Damian Johnson Date: Fri May 11 09:05:18 2012 -0700 Expanding launch_tor to support commandline options When writing small scripts we often want to start a tor instance for a special purpose, for instance to test a new feature. When doing this we want a bare-bones tor instance with just a few specific options. Adding better support for this by making the launch_tor function accept command-line options and the ability to use a blank torrc. commit 80809adcf7023d91c08f97ee9e7affcb3a3bc748 Author: Damian Johnson Date: Sat May 5 19:17:55 2012 -0700 Testing for the controller's asynchronous event handling Adding a test that hammers the control socket with queries while also listening for BW events. This also tests for and fixes a bug where listeners wouldn't get all of the enqueued events if the controller was closed. commit af691f5b54d1f571ee51d8c67010ea19e4c672fd Author: Damian Johnson Date: Sat May 5 17:48:29 2012 -0700 Supporting controllers in the connection module Users will be using a BaseController subclass unless they really need to work at a low level, in which case they'll be using a ControlSocket. Making the connection module (which does authentication) support both. commit 81f272b9e227b36fe93037cbd1d8ef2265403269 Author: Damian Johnson Date: Sat May 5 17:06:12 2012 -0700 Making decriptor reader accept path or list of paths More often than not people will want to read descriptors from a single path rather than a list of paths. There's no need to make them wrap that path in a list before calling us. commit 10cb30b546016f975559bdf1d8dd785ee516ae84 Author: Damian Johnson Date: Sat May 5 16:54:24 2012 -0700 Making descriptor reader persistence more convenient Our current functions for loading/saving processed file listings are fine if you want error handling and a great deal of control. However, I suspect that most callers would prefer for this to be an attribute of the reader itself. Adding an argument that performs best-effort persistance of our processed files listing. commit 477d448f5f19d9d5336e3297943b14567bb55c52 Author: Damian Johnson Date: Sat May 5 15:51:47 2012 -0700 Parsing descriptors with a 127.0.0.1 address as bridges A prior scrubbing scheme for bridge descriptors scrubbed their addresses to be 127.0.0.1 rather than 10.x.x.x. Parsing both as bridges so we're compatible with both schemes. Thanks to Karsten for pointing this out. commit cffc3c1be1c4c2a00ce41325a37038d83c8a80a1 Author: Damian Johnson Date: Sat May 5 15:47:06 2012 -0700 Mislabeled server descriptors as being version 3 We're using version 3 of the dir-spec, but server descriptors themselves are still using their initial version. Correcting the class names and documentation that labeled them as being version 3. Thanks to Karsten for the correction! commit 3a5e9979e40a400b8f4861406bc75af32578c30c Author: Damian Johnson Date: Sat May 5 15:27:59 2012 -0700 Separating bridge scrubbing validation into method Bridge descriptor scrubbing is a moving target and it's quite possible that we'll need to parse descriptors that conform to a newer or older scheme. Hence making scrubbing validation part of the constructor is inappropriate. Moving it to a separate method instead. This includes two methods, one to simply check if we think that the scrubbing is right, and another to get descriptions of the issues. This is of limited usefulness to callers, so we might need to switch to an exception hierchy later. However, we don't have any use cases that care to check the scrubbing yet so leaving that alone for now. commit fcbbc58d700bb6a22a4d1e7bd7d5fcb22999c9f2 Author: Damian Johnson Date: Sun Apr 22 21:14:33 2012 -0700 Ensuring that we're closed when msg() raises SocketClosed The ControlSocket's recv() method cannot assure that we're closed when it raises a SocketClosed exception (as explained in 4f8be72), but the BaseController's msg() method can. Integ tests were inconsistently failing because I expected to have this assurance. commit f921c569d3565a554f90a6ae72b52032eca86cb9 Author: Damian Johnson Date: Sun Apr 22 20:57:31 2012 -0700 Bug with socket tests when we had an emulated chroot Forgot to make one of the fixes for handling chroot tests in a socket use case. commit 4f8be72088e40bb4a6b62e9e39a4c3d9121ec472 Author: Damian Johnson Date: Sun Apr 22 20:40:53 2012 -0700 Concurrency is still hard It felt like the handle_close flag was hacky and now I know why. If recv() tried to call close() while a send() call was in flight and would also eventually call close() then we'd encounter a similer type of deadlock as commit d5162f4 tried to fix. This did not come up for control port connections, but it did frequently occure for control sockets (as seen by deadlock when running the RUN_SOCKET target). This did not always occure because the recv() sometimes acquired the send_lock before the next send() call, making everything proceed correctly. Fixing this by having recv() make a non-blocking attempt to get the send_lock. If it works then we know that we can safely close, and if we can't acquire the lock then we know that a send() or close() call is already in progress and can leave the closing to them. This does introduce a possible issue where the send() call succeeds, the recv() call fails with a SocketClosed, and we're left in a state of being alive. However, this is both a weird use case (how can the send() work if the socket is closed?) and also not strictly wrong (raising a SocketClosed does not mean that we've finished shutting ourselves down). It's a little tempting to add a dedicated close_lock to account for this, but after looking into that I realized that this would make the concurrency far more convoluted. commit 7271f04a9cdaf6bb6eb633424891c17654b2807f Author: Damian Johnson Date: Sun Apr 22 18:38:16 2012 -0700 Fixing testing thread leak and adding check for it Python does not gracefully handle lingering threads at shutdown, even if they're daemons (... which is dumb since that's the only point of the daemon thread). Having threads causes occasional nonsensical stacktraces with things like "'NoneType' object has no attribute 'socket'" for module references. Adding a check after running our integ tests that only the main thread remains, and erroring if that's not the case. This also fixes the thread leak that was probably the issue in... https://trac.torproject.org/projects/tor/ticket/5512 commit 636fd3704d6ddcba6a2a780119570fc2e25dd77f Author: Damian Johnson Date: Sun Apr 22 15:05:17 2012 -0700 Handling chroot in stem.connection and integ tests On ticket 4896 [1] I proposed a couple ideas for how we could handle chroot setups, but on reflection neither of them were very good. At a low level we can't reliably expand paths, nor should we try. If the user makes a raw 'GETINFO config-file' query then we should simply return what tor gives us, not try to 'fix' it by expanding the path. Rather, we should correct for chroot prefixes at a higher level like the controller. My current plan is... * The Controller class will have an optional chroot_path constructor argument, and a get_chroot() method. All of the Controller's methods and those of subclasses should take it into account for tor resource paths. * The stem.connection functions now accept a chroot_path argument. We need this since they will construct Controller instances, and also do the initial authentication (we need to know about chroots for cookie authentication). [1] https://trac.torproject.org/projects/tor/ticket/4896 commit 0f4fc2ee2a3e442aa3cad3dd5bb28580771570a2 Author: Damian Johnson Date: Sun Apr 22 13:55:04 2012 -0700 Providing cookie path with CookieAuthFailed exceptions A common attribute that callers want when authenticate() raises a CookieAuthFailed is the path that we tried to read the cookie from. Since the authentcate() call does its own PROTOCOLINFO query the caller never sees it (except for the exception message). Simple thing to fix. commit 6bb517d876243aafd5cf3ae0eef7ecbe45958389 Author: Damian Johnson Date: Fri Apr 20 12:25:18 2012 -0700 Catching unexpected exception from descriptor reader The os.stat() call can raise an OSError, so catching that and notifying the skip listener. commit 13f7ce34e735b2547f3ab669bed0ceb571624c0f Author: Damian Johnson Date: Thu Apr 19 19:14:42 2012 -0700 Testing option for chroot setups Adding a new target to simulate a chroot setup by stripping the test directory from any control socket responses. As expected this is causing integ tests that validate this path to fail. Sathyanarayanan took the first several stabs at this, and this is just another potential option for... https://trac.torproject.org/projects/tor/ticket/4896 commit e1a3aba733313c842c24c40ac235c073a548030d Author: Damian Johnson Date: Thu Apr 19 09:23:27 2012 -0700 Allowing for negative uptimes prior to tor 0.1.2.7 Tor 0.1.2.7 fixed a bug which could result in negative uptime values. Making our descriptor parser accept negative uptimes for tor versions with the bug, and fail validation for later versions. This includes a test with one of the problematic descriptors, checking that we can both parse it, and will fail validation if it was for a later version. Thanks to Karsten for figuring out where these values were coming from! commit 725beac5437d85e8da5baa8a6fce90ce76fa9f9f Author: Damian Johnson Date: Wed Apr 18 09:31:14 2012 -0700 Adding test for contact info with carriage return Interesting use case spotted by Karsten on... https://trac.torproject.org/projects/tor/ticket/5637 commit 21228b096617abbaabc1d53ca964b8cad591bf88 Author: Damian Johnson Date: Tue Apr 17 08:47:19 2012 -0700 Retaining padding in the server descriptor digest() funciton Network status entries exclude base64 padding to save on space but there's no reason for us to do the same. Adding a comment explaining why they might mismatch. Thanks to Karsten for explaining it! commit ebf4c0274f7e90da8946f2ba49c3e787fb367264 Author: Damian Johnson Date: Tue Apr 17 08:46:21 2012 -0700 Left out part of the whitespace rename change Oops! Forgot to do a 'git commit -a' to pick up the other changes from renaming to 'check_whitespace.py'. commit e69cb95f93d6646e809bf0143ff595dae32a2b5e Author: Damian Johnson Date: Tue Apr 17 08:42:40 2012 -0700 Checking if a file's a tarball could fail due to permissions I encountered a transient IOError from the tarfile.is_tarfile() call. I'm still not sure why it complained about '/vmlinuz.old' (and the test actually still passed), but it's inappropriate for the reader to raise an IOError there. Falling back to mime type. commit 1a2c7559e31fea6fba6856594f982f6f0959cd5d Author: Damian Johnson Date: Mon Apr 16 09:22:25 2012 -0700 Renaming whitespace_check.py to check_whitespace.py Previous naming was stupid, making it read better. commit 06aa889eddea7b608fd5d776ff2c1a1cd90d3ab4 Author: Damian Johnson Date: Sun Apr 15 18:40:01 2012 -0700 Only coloring test output if it's to a terminal Dropping the '--no-color' argument in favor of a builtin check for if our output is going to a tty teminal or not. This is the same trick that git does for its 'color.ui = true' setting. commit 22965e5c209ecc7b924093757e7e75767ea82379 Author: Damian Johnson Date: Sun Apr 15 18:36:35 2012 -0700 Correcting whitespace Fixing the mistakes caught by my neat new whitespace checker. commit 25f65d01a10ab5e4993f48317927774c956e73ae Author: Damian Johnson Date: Sun Apr 15 18:27:37 2012 -0700 Checking whitespace as part of the tests A common mistake made by new contributors to stem (... or any project, actually) is incorrect whitespace. Checking for tabs, trailing whitespace, and incorrect indentation levels after running tests to make it easier for patch contributors to get this right. It's already showing a few mistakes in code that I wrote... oops. commit 2237bce8cdf66f189bb6ab2e83435a9c3b06b694 Author: Damian Johnson Date: Sun Apr 15 16:47:25 2012 -0700 Filling in missing __init__.py entries commit a534c0607139ac9250243673d081c790866abc48 Author: Damian Johnson Date: Sun Apr 15 16:34:57 2012 -0700 Adding missing funtion to hearder doc The ControlSocket was missing an entry in its header docs for __enter__ and __exit__. commit a676e6f7d81700e670610db7e8769007d30284d3 Author: Damian Johnson Date: Sun Apr 15 16:29:03 2012 -0700 Removing incorrect comment about dead tor code Robert reports that getinfo_helper_listeners isn't dead code after all, so removing the incorrect comment. commit 67dc38b9c7d9304caa1bffde05e912849f7dc5f1 Author: Damian Johnson Date: Sun Apr 15 16:21:48 2012 -0700 Changing None checks to be by identity The 'is' keyword is almost completely useless, but the one place where identity checks are preferred is when checking for None. It's faster and won't be tricked by buggy equality methods... http://jaredgrubb.blogspot.com/2009/04/python-is-none-vs-none.html This is the suggestion as per PEP 8... http://www.python.org/dev/peps/pep-0008/#programming-recommendations commit ac3f6271e0896cbdb2e668576f43670b260bfa2b Merge: 809b341 9f76969 Author: Damian Johnson Date: Sat Apr 14 19:12:01 2012 -0700 Merge server descriptor changes suggested by Karsten commit 9f76969739eccf740da0f77378bcabc5672a85bb Author: Damian Johnson Date: Sat Apr 14 18:30:40 2012 -0700 Handling descriptor files in a depth first fashion Directories enqueued all of the files that it contained prior to processing them which has a couple obvious disadvantages... - huge targets like the root directory or years worth of descriptors can consume lots of memory with the paths alone - this could easily cause us to have a huge startup time before we provided the caller any descriptors This was stupid, depth first parsing makes much more sense. commit 707897bea765b3d403dea9eaf4a3a0788a455bd0 Author: Damian Johnson Date: Sat Apr 14 17:23:03 2012 -0700 Handling empty read/write-history value listings Bug where relay server descriptors with read/write-history lines but no values on them would fail validation. Added a unit test for this and fixed the bug. commit 96678f167f210d10f65c14d977e9f764c4d70dbd Author: Damian Johnson Date: Sat Apr 14 17:09:37 2012 -0700 Adding a test for descriptor content from 2005 Version 2 relay server descriptors were first introduced in 2005 so adding a descriptor from back then. This in theory doesn't exercise anything that the unit tests don't already do, but it still provides some good use cases (mostly around deprecated attributes). I ran over the full December 2005 metrics tarball and all descriptors were happily parsed. commit 04176384995a89555067ddecf4cf6a7fe882ce39 Author: Damian Johnson Date: Sat Apr 14 17:03:28 2012 -0700 Making descriptor reader skip tests more resilient Couple problems with the skip tests for the descriptor reader: - A bug in the test for skipping unmodified files caused the test to rely on the order in which descriptors were read. It had a check that should have caught this, but rather than "assertEqual(1, len(skipped_files))" it had "assertTrue(1, len(skipped_files))". Fixed the check and narrowed the test to just the single descriptor file that it was supposed to test against. - The test which checks that we properly skip and report non-descriptor content would fail if other files are added to the data directory. This is as it should be, but the error message wasn't helpful and I've encountered this a lot due to vim '.swp' files. commit 62f34c67bcaaa7205b09cc7cf6201d78447c0a39 Author: Damian Johnson Date: Sat Apr 14 15:51:16 2012 -0700 Expanding attributes for read/write-history Parsing and validating read-history and write-history attributes. These are depricated fields, but still found on archived and extra-info descriptors. This includes a unit test to exercise the additions. commit b8f6fd2c1a601676be30dd2850211f8fedfb3c14 Author: Damian Johnson Date: Fri Apr 13 22:57:33 2012 -0700 Removing integ test for descriptor digests Dropping the server descriptor integ test for digest(). It was a valiant effort, but is flaky because it'll always fail when the consensus or descriptor changes and are out of sync. commit c64f663c6f3215eb4e492cfe60bf7be46f50a1d0 Author: Damian Johnson Date: Fri Apr 13 22:53:36 2012 -0700 Removing restriction that descriptor socks port is zero The socks port of server descriptors is depricated and always zero, but this isn't necessarily true for archives. commit ed0e8ac4d0dca0c58cc17e2314839351590038b8 Author: Damian Johnson Date: Fri Apr 13 22:39:33 2012 -0700 Renaming calculate_digest() to just digest() commit 8c1b5b6053e08bce55b9bc935daa492f727c68e8 Author: Damian Johnson Date: Fri Apr 13 22:19:33 2012 -0700 Implementing calculate_digest() for server descriptors Implementation and test for a function to get the digest value for a relay server descriptor. This is the same value found in the network status entry. This was an addition suggested by Karsten. commit b56bb55187d6baa00510b98c6ae1c91739476acf Author: Damian Johnson Date: Fri Apr 13 20:36:33 2012 -0700 Removing timeout from descriptor reader enqueue calls The reader had a timeout for enqueue operations so it could periodically check if we were stopped while waiting to insert a descriptor that we've read. As Karsten thought we were able to do better. Dropped the timeout by clearing our queue on shutdown (to unblock any enqueue calls in process), and skip further enqueuing while we're shutting down. Both iterating and stopping are under a read lock so we don't need to worry about this changing the order in which descriptors are provided to callers. commit 0d478c57866c868c63be50714887c1b1535fbd0f Author: Damian Johnson Date: Fri Apr 13 20:11:37 2012 -0700 Removing deprecation warnings for server descriptor fields The read/write-history and eventdns server descriptor fields are part of the spec but no longer used. I was logging a notice that they're deprecated when we encountered them, but this is neither useful nor does it make sense. We might be parsing old metrics archives where those attributes are perfectly valid. Integ tests include a check that those aren't seen in the wild, which is what I really care about. commit 850b1cbbc07554ea369b44c89b3bbfdd1b99fa69 Author: Damian Johnson Date: Thu Apr 12 10:53:28 2012 -0700 Unit tests for bridge descriptors and related fixes Unit testing the new bridge descriptors. This mostly focused on parsing or-address entries and the validation that the class does for scrubbed values. Fixes included having string port values and not checking that or-address addresses were scrubbed. commit 785b22f274c30c2e52703c97bee3f2fb5a23e247 Author: Damian Johnson Date: Thu Apr 12 09:36:49 2012 -0700 Moving relay/bridge differentation into parse_file_v3() It was confusing for parse_file_v3() to only work for relay server descriptors so moving the hack to differentiate relay from bridge descriptors there. This is also nice because it moves server descriptor logic out of the general __init__ module and into the one that's specifically for handling server descriptors. commit c3fddad00ecad4ea5a94dafcfe6a0aaa10c4cfd0 Author: Damian Johnson Date: Wed Apr 11 21:12:26 2012 -0700 Differentiating read bridge and relay descriptors The last integ failure was due to our attempt to read a bridge descriptor file as a relay descriptor, which doesn't work because it is missing cryptographic entries. Adding a bit of logic to differentiate relay from bridge descriptors so we can parse them as their appropriate type. The cues the difference based on the 'Unnamed' nickname and '10.x.x.x' address scrubbing. commit 19ba0506cec868e118ddfcf89cd30d44d3b43692 Author: Damian Johnson Date: Wed Apr 11 20:59:56 2012 -0700 Handler for bridge descriptor 'or-address' entries Parsing 'or-address' entries in bridge descriptors. This parsing logic should be moved to the ServerDescriptorV3 class if/when ipv6 support is available to normal relays. commit dbd6a80ffc65f8d77d6d798b49f8c899e5ef09e9 Author: Damian Johnson Date: Wed Apr 11 20:50:08 2012 -0700 Utility for validating IPv6 addresses Function and testing to check a string is a valid ipv6 address. commit 57ed21a26d440d32e01924cbc027d97665fe8dbc Author: Damian Johnson Date: Tue Apr 10 09:03:39 2012 -0700 Server descriptor reformatting Cleaning up the recent changes to split apart relay and bridge descriptors, dropping an unnecessary parser class and rearranging things to be more readable. commit b2d1d632696e5d47789ea9d1b4f817df8634e681 Author: Damian Johnson Date: Tue Apr 10 08:25:13 2012 -0700 Dropping duplicate constructors Bridge and relay server descriptor constructors are identical, moving to parent. commit 0f0eac50b59e33d5a25b428ddb9b0d16be023a28 Author: Damian Johnson Date: Mon Apr 9 09:11:13 2012 -0700 Splitting relay and bridge server descritors Making separate classes for relay and bridge v3 server descriptors. This is the more proper object model and allows us to easily handle their differences in terms of parsing and validation. This includes the implementation and an integ test, but the implementation isn't yet complete (missing or-address parsing) and we still need unit tests. The current unit tests pass but the integ test fails due to or-address. commit 57398f137882e22d5ca278a7944212ec62e4d5a1 Author: Damian Johnson Date: Thu Mar 29 09:07:44 2012 -0700 Stop tracking last-modified timestamps if file disappears Improvement suggested by Karsten. For long running processes with a rotating pool of descriptors we could accumulate a lot of 'path => last modified' entries, so only including the entries for files that still exist. commit dd6b71c2b4fe27c33f82ba45e405be9c3fabd409 Author: Damian Johnson Date: Thu Mar 29 08:54:22 2012 -0700 Test for descriptor with non-ascii content Test suggestion by Karsten. This reads a descriptor with non-ascii content (hiragana and something else), checking that the results match expected values. commit 809b34159067c25730d71b35460ba2e0fbf9a815 Author: Damian Johnson Date: Sun Apr 8 14:34:59 2012 -0700 Removing stem.util.system.is_relative_path() Why did I make this function again? Python already has os.path.isabs()... commit d9dc5babf533460975895cca1d6cde14dcea8e41 Author: Damian Johnson Date: Sun Apr 8 14:28:00 2012 -0700 Making system unit tests more platform independent On Windows the alternate path separator caused the unit tests for expand_path() to fail. Switching the expand_path() and is_relative_path() to not make any effort to work on Windows and fixing the test to be platform independent. In the future we should make these funtions work on Windows but this isn't vital at the moment and a pita due to its '[A-Z]:\\' prefixes (I really don't wanna do a regex just to check if a path is absolute...). commit 0354e9e656655f33c04fd75ebcb45ff0f70959c6 Author: Damian Johnson Date: Sun Apr 8 14:03:55 2012 -0700 Replacing os.uname() calls with platform.system() I was doing os detection by calling 'os.uname()' which has the unfortunate disadvantage of not functioning in Windows. The 'platform.system()' apparently does the same for our purposes but... well, works. Caught thanks to reganeet on... https://trac.torproject.org/projects/tor/ticket/5493 commit 35f31d8c1527e106013ceb293302901dce8f9cf0 Author: Damian Johnson Date: Fri Apr 6 09:26:50 2012 -0700 Checking for windows style execs in is_available Stem tries to check that a command is available prior to running it, which choked on Windows due to the '.exe' extension. Issue caught by reganeet and fix suggested by him. commit e427e6d7aff32d15ecf3d5ef3e0c78d2758deb24 Author: Damian Johnson Date: Fri Mar 30 08:14:33 2012 -0700 Fixes for prior version parsing fixes I never cease to be amazed at how error prone something that sounds as simple as 'parse the tor version' can be. Guess that's why we're writing a library... This has a couple fixes for the prior set of fixes. a. The runner was misparsing 'GETINFO version' output for verisons without a space. b. Sinister bug (took me almost an hour to track down) where the results from our mocked "tor --version" call were cached and broke later tests. This only manifested if your tor version wasn't 0.2.2.35 and you ran both the unit and integ tests together. commit c63124069422cbba45f4b08b223f2bcb6eb99ef6 Merge: 222d58a d5cd861 Author: Damian Johnson Date: Thu Mar 29 18:54:23 2012 -0700 Several fixes to tor version parsing Fixes by neena for the tor version parsing, discussed in... https://trac.torproject.org/projects/tor/ticket/5472 commit d5cd861c377d336a5f5f9edb4b85e2c2cccaabf3 Author: Damian Johnson Date: Thu Mar 29 18:52:11 2012 -0700 Trivial tweaks for version fixes Very minor changes to formatting and dropping a no-op line. commit 9408f4a16a45d608a7d7711215496734902f5f12 Author: Ravi Chandra Padmala Date: Thu Mar 29 23:04:17 2012 +0530 Update code to reflect new version parser, all tests pass commit 50a240d7ff83a435a75d54c90d52088cd723c450 Author: Ravi Chandra Padmala Date: Sun Mar 25 23:04:13 2012 +0530 Treat versions with whitespace in them as invalid commit d2f7e1925e61bf73c8d7a78ed48a01badb5399a0 Author: Ravi Chandra Padmala Date: Sun Mar 25 20:58:54 2012 +0530 Neater string finding code commit 27e3806d4f37f47965c3237083d885db147faee7 Author: Ravi Chandra Padmala Date: Sun Mar 25 22:50:55 2012 +0530 Fix Version to match . strictly commit ec9101e44c74ea6e0144c970b0e675dddc3a9aee Author: Ravi Chandra Padmala Date: Fri Mar 23 04:55:48 2012 +0530 Fix version parser to ignore git hashes get_system_tor_version strips the git hash add test for get_system_tor_version remove testcase which doesn't comply with the new version parsing code commit 222d58a9bad7e1675f70002fb00b2a9b962cfe63 Author: Damian Johnson Date: Wed Mar 28 20:19:03 2012 -0700 Typo corrections for descriptor changes Patch by karsten to fix numerous misspellings. commit df2084b0e746e6c5036528b01884a168b3b513ce Author: Damian Johnson Date: Wed Mar 28 20:09:08 2012 -0700 Minor corrections for os.sysconf check The patch from beckbaibai fixed his issue but left CLOCK_TICKS undefined on Windows which would result in a NameError when using proc.get_stats (the function *should* cause an error, but not that one). commit f5950d01e1578a532c03da55165030f8c3ba3919 Author: Beck Date: Tue Mar 27 01:11:30 2012 -0500 fixed system module undefined in windows commit bc0e578615d9a856ab9052acc5a116774c49d133 Merge: 989fd95 975d98d Author: Damian Johnson Date: Sun Mar 25 17:09:21 2012 -0700 Merge branch 'metrics-lib' commit 975d98d7337d4dd566aa5e9967e62afe2fdb0bc0 Author: Damian Johnson Date: Sun Mar 25 16:12:32 2012 -0700 Integer truncation could break reader's last modified check When saving the last modified timestamps they're saved as ints. However, the timestamp itself is a float, so the following check for an unchanging timestamp would fail. commit cf0e790c809d4057eceb37550b33f6f5bf67b065 Author: Damian Johnson Date: Sun Mar 25 16:02:23 2012 -0700 Fixing reader's pydoc example commit 4502701f0ec8462c0848128f000ec18c14a94167 Author: Damian Johnson Date: Sun Mar 25 14:52:43 2012 -0700 Unit tests for server descriptor parsing Borrowing heavily from Karsten's metrics-lib unit tests [1], plus a few of our own. This doesn't include everything, some are covered by util tests and others are effectively covered by parsing metrics and cached-descriptor files. That said, I wouldn't be against including more - my fingers are just getting numb... While looking through those tests I discovered these deltas... * When parsing the descriptor he validated that we had the correct first and last entry but we didn't. This was a bug, now fixed. * He checks that the 'protocols' line only contains numberic protocols but we accept any space separated content. The spec doesn't specifify what the protocols can be so I'm leaving this alone. * He had checks that a 'published' line with a year of '3012' or '1912' would fail. I can understand having a sanity check but the spec does not say that those dates are invalid so again leaving it as-is. * Metrics lib has validation for the exit policy but we don't. This is because we haven't yet implemented an ExitPolicy class - gsathya is currently working on that in... https://trac.torproject.org/projects/tor/ticket/5454 * Karsten is parsing and validating read/write-history lines. We, on the other hand, log an INFO level warning when these appear and assert in the integ tests that they (along with 'eventdns') do not exist in our cached-descriptor file. My understanding is that these are deprecated entries and should not appear outside of the extra-info descriptors. Maybe I'm misunderstanding something here... On everything else we either match or are maybe slightly more strict about only allowing content conforming to the spec. [1] https://gitweb.torproject.org/metrics-lib.git/blob/HEAD:/test/org/torproject/descriptor/impl/ServerDescriptorImplTest.java commit 896bb442fe9cc3039a98188fd68ddfcba4f0c1e5 Author: Damian Johnson Date: Sun Mar 25 12:39:35 2012 -0700 Skipping auth message check when cookie missing When we first run our tests we don't have an authentication cookie unless we've run with the RUN_COOKIE target. We have special handling for this, but it did not account for running with an open connection (the default) in which case we'd encounter a second exception while trying to check the failure message. commit 1b507eae6bd104e9eba68ad94ff96baa7044b267 Author: Damian Johnson Date: Sun Mar 25 12:31:40 2012 -0700 Miswired 'ONLINE' and 'RELATIVE' targets The test runner still used the 'test.' prefixed configuration keys rather than 'integ.'. This caused the 'ONLINE' and 'RELATIVE' testing targets to be ignored. commit e1f7537e939a60e7c902e853b03daab9c28b3d33 Author: Damian Johnson Date: Sun Mar 25 12:20:43 2012 -0700 Skipping integ test on cached descriptors if not available The first time we run our integration tests our data directory is not yet fully populated. This causes our test that runs over the cached descriptor file to fail. Adding a check that the cached descriptors exist before running it. commit b3d357b9f12d9bce72ec411f2681fdf84dcb4139 Author: Damian Johnson Date: Sun Mar 25 12:15:15 2012 -0700 Including deprecated server descriptor attributes I have mixed feelings about cluttering the server descriptor with dead fields, but since we're already this close to being perfectly spec conformant we might as well include this last bit. Adding the fields but logging an info level warning if we see it, and checking in our cached descriptor that they do not exist. commit 092d36a17a9318c42f09199e66458e399af68fda Author: Damian Johnson Date: Sat Mar 24 19:58:48 2012 -0700 Integ test to validate our cached descriptor file Parsing all of the descriptors in our cached descriptors file to see if it has anything that'll make us choke (and by extension indicate either a tor or stem bug). This is an important yet lengthy test, taking around four second. Targets do not effect this test so only running it once for the test run. commit d4fa1279f43c585d9551c6e1f94d3917e14eec99 Author: Damian Johnson Date: Sat Mar 24 19:45:06 2012 -0700 Dropping attribute for block types We already have enough attributes, and parsing out the armor type is redundant since it's within the armor itself. Besides, it's not likely to change and I doubt callers will ever care about it. commit 7dc7d82bd7b7f70f3dfae21dd607502b4eff6477 Author: Damian Johnson Date: Sat Mar 24 19:38:27 2012 -0700 Missing version 3 attributes Adding the server descriptor attributes that are new as of version 3. The only attributes that I'm still excluding are eventdns (because it's dead) and read/write-history (since they're extra-info now). commit e0f8bfcacca963e1cbc7811dd1f9e6e7e104c7f8 Author: Damian Johnson Date: Sat Mar 24 18:50:20 2012 -0700 Allowing for extra args on lines with multiple Well, crap. I've been developing against the version 2 spec on the assumption that was both the majority use case and what my test instance was running. However, that's evidently pretty ancient and I've been running against version 3 all along. First change I'm spotting in the spec is that lines with multiple arguments should ignore extras so doing that. commit d72c21de710ba78e16b6d0fa29949fb43a2e5eb6 Author: Damian Johnson Date: Sat Mar 24 18:15:27 2012 -0700 Server descriptor test for metrics content Test for parsing a single server descriptor and checking its content. This includes a few fixes and improvements for the ServerDescriptorV2 class, the most intersting of which is that declaring instance variables prior to init makes a single reference. Hence all server descriptors were using the same exit policy. Oops. :) commit 9faad3255491642f87801926512c9cb6d334f6d8 Author: Damian Johnson Date: Sat Mar 24 16:50:03 2012 -0700 Integration test for reading non-descriptor files Integ test for reading non-descriptor plaintext and binary data. commit 7e9d454ed4eafe6b5461c5ff5170ca5634899edf Author: Damian Johnson Date: Sat Mar 24 13:41:11 2012 -0700 Cleaning up prior descriptor implementations Adding header pydocs and made a variety of small fixes and naming improvements. commit 8dc2d66b7dde16e0709dc5c5ff1ac0c320397f22 Author: Damian Johnson Date: Fri Mar 23 22:05:29 2012 -0700 Server descriptor parsing fixes Resolving the variety of issues that were breaking the integ tests. We still don't have any tests to actually verify the server descriptor parsing, but at least we're now exercising it on some real data... and not dieing in a fire! Progress! commit c91532362bd04d90c35adab22d497ba09ee7cc64 Author: Damian Johnson Date: Fri Mar 23 20:23:57 2012 -0700 Resolving circular dependency with server_descriptor ... damn this was a pita to debug. A long explanation of the issue is: http://stackoverflow.com/questions/8991520/python-conditional-module-object-has-no-attribute-error- with-personal-package Short story is that the stem.descriptor module imported server_descriptor prior to being fully initialized which caused a server_descriptor use of stem.descriptor.Descriptor to fail. The confusing was that it was actually reporting failure in using "stem.descriptor" (the module itself) which made no sense because it was listed within sys.modules. It turns out that the python importer uses module attributes rather than sys.modules mappings to resolve statements (ie, it was doing getattr(stem, "descriptor")). However, a module's added as an attribute of its parent when its *finished* importing. This fixes the unit tests, but integ tests are still getting stuck when they use the new server_descriptor parsing. commit ab8fe646817b585ceda7c6062c3294556f8f71cc Author: Damian Johnson Date: Fri Mar 23 20:23:17 2012 -0700 Cleaning up minor import issues commit 55db7157d30e62af2fa610f1bb962e8fc8a29817 Author: Damian Johnson Date: Fri Mar 23 09:39:50 2012 -0700 Merging descriptor.py with module init The syntax for using the descriptor module was stupid ("import stem.descriptor.descriptor"). Using the json module as an example for using __init__.py for base functions. Common ones will live in 'stem.descritor' but the contents, like 'stem.descriptor.reader', are also accessable. This is similar to how the os module works, though in that case they're using a lot more python magic to also support multiple operating systems. commit 3b657b57b4cb4ce5304145a7a55051e758cf9cbd Author: Damian Johnson Date: Fri Mar 23 09:28:55 2012 -0700 Using server descriptor when parsing Parsing server descriptors via the new server descriptor class. This breaks the tests and will need quite a bit of work. :) commit 245c85ecf35e7324ce008c351e1707b5ef96567c Author: Damian Johnson Date: Fri Mar 23 08:41:43 2012 -0700 Pylint corrections for server descriptor Corrections from running pylint over the untested server descriptor class (all basic problems that would keep it from running). commit e20fd3efae6a0e3a7f647017da9f62ccc83aebcc Author: Damian Johnson Date: Fri Mar 23 07:31:41 2012 -0700 Parsing descriptor files and annotations Completely untested change to finish the implementation of server descriptors. This parses a cached descriptor file into individual descriptors and annotations. Next comes unit and integ tests to start exercising it in an automated way. commit 989fd953c6cf59deeaeb65a2a6fbde88d3410136 Author: Damian Johnson Date: Thu Mar 22 10:20:50 2012 -0700 Fixing incomplete mocking for protocolinfo unit test The protocolinfo unit tests mock out system calls which should allow relative cookie paths to be expanded. However, this didn't account for the is_available checks which prevented those calls if the command was unavailable on the users system (for instance, OSX). Caught by gsathya. commit 063ce0bd893bca006b13b8781740924b560cb79e Author: Damian Johnson Date: Wed Mar 21 09:24:17 2012 -0700 Removing bold formatting from test results with --no-color The --no-color argument should remove all ANSI escape formatting but it just removed color. As a result the test result text (such as "[SUCCESS]") was still bold. Caught and fixed by gsathya. commit 838a3f37037b7533e010dbb0563191d228fb4947 Author: Damian Johnson Date: Tue Mar 20 09:00:58 2012 -0700 Including exit policy among the attributes We'll later have a proper ExitPolicy class but that's a small project of its own. Until then this will just provide the list of strings that make up the exit policy. commit 7e64119a665ecbfb3ee06d3ddcfe066f212f63f6 Author: Damian Johnson Date: Tue Mar 20 08:53:08 2012 -0700 Making server descriptor validation optional Adding constructor pydocs and an argument to skip validation. This should have performance advantages and allow for best-effort parsing of malformed descriptor content. commit 29fad9389f92557e349ffebb32d9cef58b2796c0 Author: Damian Johnson Date: Mon Mar 19 09:14:11 2012 -0700 Handling the server descriptor's contact and family lines commit 043f739405ec6012725ce4d86361fc01cb1a0c17 Author: Damian Johnson Date: Mon Mar 19 08:55:47 2012 -0700 Handling the server descriptor's router-signature line commit f6221839e8b012fa5848feb5b68eb3f390889ba2 Author: Damian Johnson Date: Mon Mar 19 08:38:40 2012 -0700 Karsten already had a ticket for opt removal commit 54c3e6b7c6ea41ff29230c4e6a476f6f7fbac19a Author: Damian Johnson Date: Mon Mar 19 08:35:12 2012 -0700 Generalizing PGP style block parsing Descriptors include blocks for both "RSA PUBLIC KEY" and "SIGNATURE" so generalizing that parser (the spec around how to handle these blocks is wrong and I'll fix that later). I'm also including the block type since callers will want that to be prepared for encryption migrations. commit 5e7d6fadb520817e21436ecfea6437fa7ae26ab1 Author: Damian Johnson Date: Sun Mar 18 15:56:26 2012 -0700 Handling the server descriptor's onion-key and signing-key lines commit 68bb0d6621ddb4806285ccaab298d2fa166050a6 Author: Damian Johnson Date: Sun Mar 18 15:49:33 2012 -0700 Parsing descriptor signature blocks commit be8e245400161fe05a2899da9a003eefa8110989 Author: Damian Johnson Date: Sun Mar 18 15:21:22 2012 -0700 Handling the server descriptor's uptime line commit da5fcc6ba9b170fce548f08848ae2b0b268bf547 Author: Damian Johnson Date: Sun Mar 18 15:18:56 2012 -0700 Handling the server descriptor's hibernating line commit 4ae18ed246b8f9f762df154ea493adb5cfbf1e80 Author: Damian Johnson Date: Sun Mar 18 15:08:57 2012 -0700 Handling the server descriptor's fingerprint line commit 47f21c97da50392d4d0be38d7292bca920d716df Author: Damian Johnson Date: Sun Mar 18 14:39:13 2012 -0700 Stripping off the 'opt ' keyline prefix commit 0ae8839cd91e658a36e7a4c9e5eeec7a75146651 Author: Damian Johnson Date: Sat Mar 17 19:24:45 2012 -0700 Handling the server descriptor's published line Ugggg, I'm always filled with hate when I need to touch the datetime module. I'm not entirely sure if I'm supposed to be associating timezone information with this datetime object or not - and if I am how to get the f*ing thing to realize that it's GMT. commit ac81552c0b966b1a9bed97c812e5a574ffa6f3db Author: Damian Johnson Date: Sat Mar 17 18:53:16 2012 -0700 Handling the server descriptor's platform line commit 9342515713edbf3c8a8a2db9ac89132a98eb859c Author: Damian Johnson Date: Sat Mar 17 18:44:28 2012 -0700 Handling the server descriptor's bandwidth line commit a2aa3c21ea66a38bedb58b561d081e5367ad3982 Author: Damian Johnson Date: Sat Mar 17 18:25:44 2012 -0700 Validation for server descriptor router line Checking that there aren't any surprises from a server descriptor's router line. commit 8fe000d4de00a073fba060b39f42a8fa4a423077 Author: Damian Johnson Date: Sat Mar 17 18:20:30 2012 -0700 Function to check validity of a port commit 28fb4b1d55952ec905762d4cb400d997e117a0ea Author: Damian Johnson Date: Sat Mar 17 17:59:33 2012 -0700 Funcions to check validity of nicknames and fingerprints Couple functions to check if a string is a valid nickname or fingerprint, plus tests. I needed to guess at the definition of a HEXDIG but otherwise nothing very interesting. commit 31cd9b9aa5ea0b5139d3d32ee3a424d9af1e26ca Author: Damian Johnson Date: Sat Mar 17 17:07:52 2012 -0700 Implementing and testing is_valid_ip_address() Moving over an arm function for testing if a string is a valid IPv4 address. Also throwing in some unit tests and a fix for entries with leading zeros (for instance "1.2.3.01"). commit cb819ce76b011c4ee84fa15014c92166bca49577 Author: Damian Johnson Date: Sat Mar 17 16:19:09 2012 -0700 Completely untested start for descriptor parsing Starting on server descriptor parsing. This is gonna need some other classes and helper functions (such as an ExitPolicy and is_valid_ip_address()) but otherwise should be pretty straight forward. commit af1cdfcedf961b506b63c923fdd9b5a6a3edd1e5 Author: Damian Johnson Date: Sat Mar 17 15:54:15 2012 -0700 Stub for identifying descriptor types The various tor descriptor types were not made to be readily identifyable, so we'll probably need to guess their type based on their first line. Ideally that first line would explicitly say the type and version, but oh well... commit da4ccb0bedc122c4776048751c4de0b6cc0e010e Author: Damian Johnson Date: Wed Mar 14 09:55:38 2012 -0700 Descriptor module with common functionality Adding the descriptor module which has the Descriptor class (the common parent for all descriptor types) and the function for parsing descriptors from a file. This parser will later do type recognition to figure out how descriptors should be parsed. This is identical to how metrics-lib does this. Nice advantage is that we can now move all of the remaining 'TODO' notes out of the reader. It's done. :) commit 5b7eacfce6ac4f370dfb4fb1baa293c4b934689f Author: Damian Johnson Date: Tue Mar 13 09:26:01 2012 -0700 Making _get_raw_tar_descriptors() thread safe At present integ tests are single threaded, but there's a project proposal to change that and spawn threads for each target. Fixing this helper function so it won't be sad when that day comes. commit 149d1bd2932a093ffe7fb75298da9b442b3b64c7 Author: Damian Johnson Date: Tue Mar 13 09:19:48 2012 -0700 Condensing basic example a tiny bit Combining the constructor and 'with' lines since that's probably the common case. The second example already shows a standalone constructor anyway. commit 7faf52e5d122c9266baefec4a31e4d3a3b40ce9f Author: Damian Johnson Date: Tue Mar 13 09:18:24 2012 -0700 Inverting save_processed_files arg order Little nicer when the usually-static argument is first imho. commit 985d1c473526748013bc7f09658c5d24df76ef5c Author: Damian Johnson Date: Tue Mar 13 09:13:35 2012 -0700 Adding test for the DescriptorReader's buffer_size Simple test that we don't read ahead more than the DescriptorReader's buffer size. commit 828d5dac1c0eda9db996438af45515c0dec0cef0 Author: Damian Johnson Date: Mon Mar 12 20:28:16 2012 -0700 Adding tarball support to the DescriptorReader Adding support for reading directly from tarballs (which is how metrics are commonly fetched). This supports all forms of compression that the tarfile module does (gzip and bz2 among other). Including some tests and archives to read against. commit 924e1bd3d3a9b7b0578ad9b9bb86a86d0278ff3b Author: Damian Johnson Date: Mon Mar 12 19:25:06 2012 -0700 Minor pylint corrections commit adbc1991fdae353825b893f5e90aee6ee882e0a4 Author: Damian Johnson Date: Mon Mar 12 19:17:33 2012 -0700 Limiting the number of buffered descriptors Preventing the DescriptorReader from having unbounded memory usage by limiting the number of descriptors that we'll store before we wait for our caller to request some. This doesn't technically make our memory usage bounded since a single descriptor doesn't have a limited size, but if one descripter can trigger the OOM killer then we have a problem. :) This isn't yet tested because we only have a single descriptor in our test data (we need at least two before we can test this). Adding a todo note for now. commit 63461450b7d7651c11d1f911f3443bf8a202f3f6 Author: Damian Johnson Date: Mon Mar 12 09:34:15 2012 -0700 Minor tidying up of integ tests Reread the DescriptorReader integ tests and tidied up a few minor stylistic issues. commit 34d14afca254eac8f0493f9c9b8aabc7c5400899 Author: Damian Johnson Date: Mon Mar 12 09:27:31 2012 -0700 Notifying skip listeners of files that were already read Skip listeners are supposed to be notified of all files that we skip, but wasn't accounting for files that we skipped because their 'last modified' timestamp indicated that they'd already been read. Modifying the integ test for set_processed_files() to test this listener condition (more graceful that the prior test that it had been doing). commit 5d61c479081b73eb241bf4a6c541e15ddb7ea713 Author: Damian Johnson Date: Mon Mar 12 08:55:46 2012 -0700 Condensing no-op iterations over DescriptorReaders At multiple points in the tests we were iterating over all of a DescriptorReaders' items without doing anything. There's no need to do the iteration via a loop - coverting it into a list does the same thing. commit ca837ddcb6f7cc6d118d4e1a774d2c4b22dbda79 Author: Damian Johnson Date: Mon Mar 12 08:52:17 2012 -0700 Allowing the DescriptorReader to be run multiple times Improving the usability of the DescriptorReader class by making it so callers can reuse instance multiple times to get descriptor changes since the last run. commit 16feb65cb0b8141137a14ab0cdde79ed79ed6f5a Author: Damian Johnson Date: Mon Mar 12 08:18:57 2012 -0700 Being more careful to remove temporary test files When tests failed they could leave behind temporary files which, in the case of the 'secret_file' (which lacked read permissions) this would trip up future test runs. commit 77ac27cbe057e9411c28d0a8be814046465d335b Author: Damian Johnson Date: Sun Mar 11 15:20:50 2012 -0700 Integ test for files skipped due to permissions Adding an integration test that listens for a file which is skipped because the user lacks read permissions. commit 827d3495084f762d4e2e0df3253fea00fc532707 Author: Damian Johnson Date: Sun Mar 11 15:13:57 2012 -0700 Integ test for skipping files that don't exist Adding an integration test that listens for a file which is skipped because it doesn't exist. commit 9f8683f182b1199ba1eb6955130e46db0bae5372 Author: Damian Johnson Date: Sun Mar 11 15:05:54 2012 -0700 Integ test for skipping a file due to its mime type Adding an integration test that listens for a file which is skipped due to a type indicating that it doesn't contain descriptor data. commit 3bf5098545db24eb178e6b6670463616f284ec96 Author: Damian Johnson Date: Sun Mar 11 14:39:01 2012 -0700 Implementing skip listeners and exception hierarchy Notifying the skip listeners of files that we skip, and adding an exception hierchy for the reasons why. Also a little minor refactoring to move the file handlers out of the run() method. commit ee5fa60dfb96b4ae0bc8f5a18aa244b4b5f296c8 Author: Damian Johnson Date: Sun Mar 11 13:58:00 2012 -0700 Integ test for the set_processed_files() method Testing that the DescriptorReader's set_processed_files() method properly causes us to skip priorly read files. commit 664552e72719776f17cd654928dbf5d0a12e48d9 Author: Damian Johnson Date: Sun Mar 11 13:41:48 2012 -0700 Fixing sinister concurrency issue Replacing the _is_reading event flag with a 'FINISHED' entry in the _unreturned_descriptors queue. This is because python's queues stupidly have no notion of flushing, so there's no method for me to make a reliable check of 'if the reading thread is finished AND the queue is empty'. I may have called 'put'. I may have a proveably not-empty queue. But can I make that check work? Nooooo. That is... frustrating. >:( commit 2f8a3c838fd63a83a6087313b5a3f52160d2e4fb Author: Damian Johnson Date: Sun Mar 11 13:17:09 2012 -0700 Running test_basic_example() multiple times Rerunning the test_basic_example() test fifteen times to make it more sensetive to concurrency issues. It now fails consistently! (... yay?) commit 1bfadbca0ae379b62c140193883a833519c4a887 Author: Damian Johnson Date: Sun Mar 11 13:07:12 2012 -0700 Integ test for get_processed_files() Integration test for the get_processed_files() method of the DescriptorReader class. This and the test_basic_example tests are intermittendly failing right now which indicates a concurrency issue - though this test itself has passed and should be fine. commit 0c021dc30ac9b4066e7c647f4a60bfda4511c9c3 Author: Damian Johnson Date: Sun Mar 11 12:54:50 2012 -0700 Only provide absolute paths with get_processed_files() The DescriptorReader's targets can be relative paths, and this is fine. However, the load and save functions for processed_files expect absolute paths so making get_processed_files() expand relative paths. I'm on the fencepost about if we should do this or make the load/save functions accept relative paths. Going with this until someone produces a use case where this makes them sad. commit e5e0c3895c0cc9ce540e37da6eec9d88e14bdeda Author: Damian Johnson Date: Sun Mar 11 12:50:56 2012 -0700 Tidying up DescriptorReader attributes Bit of refactoring to limit the scope or improve readability of attributes. commit c16df469c9d49f50ed07bdb8b6c823e8aa31f6ce Author: Damian Johnson Date: Sun Mar 11 12:39:47 2012 -0700 Inverting the 'finished_reading' flag Code is more readable when flags say what we're doing rather than what we're *not* doing. commit 932ded67f961c67c515750bd6e1c9ddb735f8559 Author: Damian Johnson Date: Sat Mar 10 23:30:28 2012 -0800 Fix and test for DescriptorReader stop() method Adding an integraion test and some fixes for the stop() method of the DescriptorReader class. commit 0aba5ba4704acb620915ba5f2115b7d036c9477a Author: Damian Johnson Date: Sat Mar 10 22:55:50 2012 -0800 Finishing basic DescriptorReader implementation Fixing the minor issue that was preventing the DescriptorReader from working and making its first integration test do automated verification, rather than needing a manual check. commit 2d620558fa0885c532e8759c497881146a89ba0a Author: Damian Johnson Date: Sat Mar 10 18:09:52 2012 -0800 Implementation of DescriptorReader concurrency After much hair pulling figured out a relatively producer/consumer simple model for this class. It should be trivial to add stop() later, but making this re-runable would greatly complicate the class and probably isn't worth it. This isn't yet working, but this is a decent breaking point. commit 50ac2381a069a9e72094ff210954f9db1bdad0cb Author: Damian Johnson Date: Fri Mar 9 20:12:42 2012 -0800 Integ test for basic usage example The header documentation contains two examples - the equivilant to 'hello world' with this class and an example of making use of the high water mark the DescriptorReader tracks. This adds an integ test for the first. At present this integ test dies in a fiery... well, fire. This is because the DescriptorReader isn't yet implemented. That's next... commit 02d51e37e54aeefc77d906792d12e1bb711b89c5 Author: Damian Johnson Date: Fri Mar 9 19:51:38 2012 -0800 Rewriting reader documentation Moving all of the examples and substantial documentation up into the header, and minorly revising how I intend for the module to be used. Not all of the methods in the header have been added yet. commit 86b90e84e26ac91e1f876ec96df4d983c5ab384f Author: Damian Johnson Date: Thu Mar 8 09:04:47 2012 -0800 Integration tests for save_processed_files() Basic integ tests for the save function, checking that we can load persisted files and that it has some basic input validation. commit c547ed5688da5015dbddbb091118e351fef31bdd Author: Damian Johnson Date: Thu Mar 8 08:27:51 2012 -0800 Integration tests for load_processed_files() Basic set of integ tests exercising file reading when successful, non-existant, or lacking permission. commit 5e6623ce92e0bee08df9af18aacdd30548d47ad7 Author: Damian Johnson Date: Wed Mar 7 20:28:44 2012 -0800 Unit tests for the load_processed_files() function Unit tests for stem.descriptor.reader's load_processed_files() function. This mocks the open call to simulate file reads and exercise a variety of good and malformed contents. This does not yet test for use cases where the file doesn't exists or fails to be read due to permissions, but those will be integ tests. commit 5f8064b3ba8f47162dc781a5804839c029599d14 Author: Damian Johnson Date: Wed Mar 7 20:20:09 2012 -0800 Typo in import statement Basic syntax issue (import typo) commit cbc0cb1a509911f89e8b14428efa65f0aa5b22b6 Author: Damian Johnson Date: Wed Mar 7 20:17:29 2012 -0800 Adding mocking support for builtin functions Builtin functions, like open, lack the __dict__ attribute among other things which prevents the current mocking scheme from working. Adding workarounds so we can accomidate them. This also adds a function for supporting the 'with' keyword on mock objects. commit d93d894a3339216b7dc1791ea8c009cd280db7ff Author: Damian Johnson Date: Sun Mar 4 11:24:24 2012 -0800 Revising implementation details for processed files Like metrics-lib, the DescriptorReader will track the last modified timestamp for the descriptor files we have processed. Revising the pydocs for it and adding untested functions to save and load. Next step is to add integ tests. commit 5a7bb2584ec976b64a91986e5aa8ca60d8ab60a0 Author: Damian Johnson Date: Sun Mar 4 01:22:36 2012 -0800 Skeletion for descriptor reader Initial skeleton for a class to read descriptor data, not including most of the functional bits. This will be similar to the metrics-lib DescriptorReader class, but with a slightly different API and the ability to handle tarball archives. commit e447d3e24cc852795dce9b77da703c2ec77cd2d0 Author: Damian Johnson Date: Tue Feb 21 15:36:14 2012 -0800 Fixing an unexpected issue during test interrupts When we get a KeyboardInterrupt (ie, the user presses ctrl+c) we sometimes don't have a tor process to kill which results in an unexpected stacktrace. I'm not quite sure why, but if a process doesn't exist then... well... guess we don't need to kill it. commit 9a7b383c18e42cbce093803e8cdd04d98aa900c5 Author: Ravi Chandra Padmala Date: Wed Feb 22 10:13:52 2012 +0530 Stop running (all) integ tests on receiving a SIGINT. Fix #5199 commit be32e6a5017b220643ee4df4aace3d81a6acdd73 Author: Damian Johnson Date: Sun Feb 19 16:33:42 2012 -0800 Fixing another close() deadlock issue The previous fix narrowed the window where close() / recv() calls could trigger deadlock, but it didn't eliminate it. I'm adding another test that reliably triggered deadlock in that case and narrowing the window even more (which fixed the issue). I'm a little worried that this doesn't completely eliminate the issue since there is a theoretical race if recv() calls close after someone else calls close() but before they set the boolean flag. That said, I'm not sure if this is really an issue in practice. commit d5162f4e369f5d2226f0e5262f2b63025f9a68ec Author: Damian Johnson Date: Sat Feb 18 18:15:07 2012 -0800 Fixing deadlock in BaseController Found two concurrency bugs which were causing deadlock issues, and adding a test that's more likely to trigger connect() and close() concurrency issues. The issues were... * The recv() method calls close if the socket is still flagged as being alive. Unfortunately this can cause deadlock if the closing thread joins on the recv thread. * For some reason using a Condition rather than an Event caused the event loop to sometimes miss the notice that caused the event thread to close, causing its join() call to get stuck. commit fa2aeb917ed84433abc76ea77f5794ad695621d7 Author: Damian Johnson Date: Sat Feb 18 16:47:48 2012 -0800 Async event handling for BaseController Implementation for the BaseController. This continually pulls from the control socket, providing three things... - asynchronous events are sent to a callback - msg() function which sends a message and provides back the response - functional is_alive() check (we need a continuous puller to know when the socket is closed) These are done in a similar fashion as the TorCtl class except that I'm aiming to provide thread safety. That said, I haven't got it right yet. Controller tests inconsistently fail with... - deadlock - seg faults (... not good, indicates a python bug) - occasional nonsensical stack trace on shutdown: Exception in thread Tor Listener (most likely raised during interpreter shutdown): Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner File "/usr/lib/python2.7/threading.py", line 505, in run File "/home/atagar/Desktop/stem/stem/control.py", line 389, in _reader_loop : 'NoneType' object has no attribute 'socket' Needless to say, all of this needs to be fixed. However, I've been banging my head against these concurrency issues for days so I should finally commit what I have. commit 4ff7efe297f2a076e1b4dc06a3686fdcbf260f8a Author: Damian Johnson Date: Mon Feb 13 08:59:11 2012 -0800 Separating BaseController into standalone class The BaseController was previously a ControlSocket subclass because it was easier to use when several of its methods were accessable. However, from an implementation perspective the BaseController was a wrapper class, not a proper subclass. After experimenting a bit more I realized that I don't want the BaseController to provide *all* of the ControlSocket methods. In particular, it doesn't make sense for callers to use the send() and recv() when there will be a msg() method similar to TorCtl's sendAndRecv(). The wrapper/subclass mix was also just plain old confusing as an object-oriented design. I'm moving the notifications up to the controller so the ControlSocket is very similar to how it was a couple weeks ago. Also dropping the passthrough integ tests since most of them will break without send/recv - I'll need to add some more targeted tests for passthrough methods later. commit 1b1d48c37ae9092fdb93a76a3bc6201dad3113e4 Author: Damian Johnson Date: Thu Feb 9 10:32:43 2012 -0800 ControlSocket listeners for state changes Adding functions to let users subscribe for state change notifications. This is something that I've found highly useful in the past, and will also be needed for the BaseController implementation. commit 8f92a27705e78ec6751b6974f6f31d772137b026 Author: Damian Johnson Date: Wed Feb 8 20:24:10 2012 -0800 Putting all locks under a 'with' clause At first I was dubious of the usefulness of 'with' keyword gsathya showed me. However, now that I've discovered that locking can be done under it I take that all back - it's a wonderful, wonderful thing and I don't know how I got by with manual locking/releasing before. ... and then they ate Sir Robin's minstrels and there was much rejoicing. commit db0e9ce6d969674a6a46d24d250fe8acd6245af3 Author: Damian Johnson Date: Wed Feb 8 19:35:36 2012 -0800 Raising NotImplementedError for _make_socket() ControlSocket is an abstract class without an implementation for _make_socket(). In python the proper type of exception to raise for this is a NotImplementedError. commit 81fc285ff9c6e257313115d9b1e489966e172fc8 Merge: a3cddb0 e3b78e8 Author: Damian Johnson Date: Wed Feb 8 19:17:32 2012 -0800 Merging gsathya's stem.util.conf save() improvements Work is being tracked in... https://trac.torproject.org/projects/tor/ticket/4913 commit e3b78e898dcfd4d33356c7e791fb92e6c7128484 Author: Damian Johnson Date: Wed Feb 8 19:15:05 2012 -0800 Cleanup for stem.util.conf's save() method Minor changes including: - fixing a 'self.path' assignment to be 'self._path' - making the path optional for load() as well - raising a ValueError for both load() and save() if we use an undefined path - whitespace fixes - added argument to save() pydoc - replacing integ test with one that explicitely tests the three types of values it might handle commit dd29415f27eb509ca496bc2159258a3a6b5d7774 Author: Sathyanarayanan Gunasekaran Date: Thu Feb 9 01:16:51 2012 +0530 Preserve structure of config file while saving Previously the single line config entries were converted to a multi line structure and then saved. Now, the structure of the config file is preserved. Fixed by atagar commit 3f51e6920c1dc6dc28d7f70115cd30ed8c020b29 Author: Sathyanarayanan Gunasekaran Date: Thu Feb 9 00:20:58 2012 +0530 Added integ tests for config.save() Tests check save() with single and multi line configuration files. commit d48aadd31678e6a4dd670a4e23832a3d563c4072 Author: Sathyanarayanan Gunasekaran Date: Thu Feb 9 00:05:19 2012 +0530 Updated config.save() - Added Pydocs - Added handling of multi-line entries - Added path argument - Changed single letter variable 'f' to 'output_file' - Instead of calling sort() on config.keys() and iterate over it, we pass config.keys() to sorted(). commit a3cddb03bd4ccd1dc3824bf2ec76695bde3b9322 Author: Damian Johnson Date: Tue Feb 7 09:47:10 2012 -0800 Replacing send/recv conditionals with RLock I'm not sure if threading.Conditional is re-entreant, but RLock definitely is (that's what it's there for). Quite possable we had a concurrency issue due to this, though if so I've never ran into it. commit 398cbfb93a4982961e1ce27fc05d82265408e78a Author: Damian Johnson Date: Mon Feb 6 08:16:59 2012 -0800 Helper constructors for BaseController instances A controller is a wrapper around a ControlSocket instance so adding convenience methods to construct both the socket and controller at the same time. These will belong to the Controller class later. commit a500dbc23d7b3f012a2040d104c3d009b36b3694 Author: Damian Johnson Date: Sat Feb 4 20:23:02 2012 -0800 Pass-through BaseController class Controller will be a ControlSocket subclass so making it a pass-through to the socket they're constructed from to start with. Testing this turned out to be a pita for a couple reasons... - The mock() function can't handle class methods, nor is it possable to add method handling to it. - Ideally the controller would dynamically pick its parent class based on the socket it's constructed from. However, this is impossable (actually, I don't know of any language where instances can dynamically define their class hierarchy). This mostly just becomes an issue for isinstance checks. Dropping the old, scrap controller implementation for now. Some of its useful bits will come back later. commit 5c01e29839a084018185ce005c9ec90788e13fce Author: Damian Johnson Date: Sat Feb 4 18:05:31 2012 -0800 Stop tracking reverted mocks When we reverted function mocking we didn't clear our registration of it. This caused the revert_mocking() function to attempt to revert things that weren't still mocked. This didn't cause any problems since doing this was a no-op, but still good to fix. commit 2076447cb6b86e02bf9bb0cfc666f7568a8f4d00 Author: Damian Johnson Date: Sat Feb 4 18:03:27 2012 -0800 Integration tests for the ControlSocket class The ControlSocket subclasses were indirectly tested via the ControlMessage tests, but we missed several interesting use cases. Adding more testing for the ControlSocket instances and correcting some of its behavior with respect to reporting socket closer while sending messages. commit 2ae0298e816ccb70bf9ab52996a4c78e26f897f0 Author: Damian Johnson Date: Tue Jan 31 09:23:31 2012 -0800 Exercising is_alive in ControlMessage integ tests Adding a basic check for the is_alive method. The method is a little troubesome because if you aren't continually pulling from the socket then you won't detect a disconnect until the following usage. This was already documented and iirc it won't be an issue in practice for controllers since they continually pull from the socket. But still, for low level controller interaction it's kinda confusing. commit 06498d8b61ad5f01c04c7011c70892a9846d4c27 Author: Damian Johnson Date: Tue Jan 31 07:46:53 2012 -0800 Adding ControlLine peek_key method Method to allow us to switch on the key/value mapping while parsing a line. This isn't needed yet (the only parsing done at this point is PROTOCOLINFO responses and that only uses positional entries). However, it's crossed my mind a few times that this will be needed later. commit c58177c50fcdba85e5eac7a929dbf6d87c666852 Author: Damian Johnson Date: Mon Jan 30 00:34:38 2012 -0800 Previous socket use could break get_protocolinfo If prior use of the control socket had caused a disconnect and already consumed the 'Authentication required.' response then future calls of get_protocolinfo would break. commit a31aa4ad506ccf4e529182e31d0a84966e38d6e1 Author: Damian Johnson Date: Mon Jan 30 00:02:55 2012 -0800 Accounting for big-endian architectures in proc The proc utils were assuming that encoded ip addresses were little-endian. This was fixed in the project these utils are based on (psutil) and I'm adopting the fix... https://code.google.com/p/psutil/issues/detail?id=201 https://trac.torproject.org/projects/tor/ticket/4777 This evidently occures on OpenWRT (ar71xx), thanks to swalker for the catch! This is the same as arm commit 8ec7095d79ecad9d3432193a2b1f9fdefab8d7f3 commit f7d18618e20b440b0efaccb6f4a5786842700f9c Author: Damian Johnson Date: Sun Jan 29 17:35:28 2012 -0800 Removing CALL_MOCKING from the system module While writing the unit and integ tests I realized that the tests could be greatly simplified by adding an override function called 'CALL_MOCKING' that would let the tests modify the call function's behavior. This was a mistake. Production code should be designed to make testing easy because this leads to better production code, but adding hacks that are only useful for testing is bad. Now that we have a nice mocking module the CALL_MOCKING hack is no longer necessary so removing it. commit 5e81a0b546413f6ef21266cb74b81e9be4c6a27c Author: Damian Johnson Date: Sun Jan 29 17:31:53 2012 -0800 Refactoring system integ tests This is the last of the integration tests to be cleaned up. It's also the last user of the system module's CALL_MOCKING so now we can get rid of it - hazaa! commit 0f913df18423f84650b27144f2a89fb9997349ab Author: Damian Johnson Date: Sun Jan 29 15:05:05 2012 -0800 Refactoring conf integ tests commit f38ad0811290b51b99475de28628895a663e7dec Author: Damian Johnson Date: Sun Jan 29 14:40:27 2012 -0800 Refactoring ControlMessage integ tests Besides the normal cleanup I'm pushing most of the skipTest checks into the runner module. The advantage of this is that it simplifies tests a little more and allows a standardized skip message rather than having each module provide their own for the same thing. commit 39672342bf81a5de15f40fe78a3170673f4035de Author: Damian Johnson Date: Sun Jan 29 13:36:27 2012 -0800 Renaming version requirements to be hierarchical Including a prefix with all version requirements for the option's catagory. Currently this includes GETINFO_* and TORRC_*. commit 51e21cf88643e63953e897d0cd05875d00c26991 Author: Damian Johnson Date: Sun Jan 29 00:49:29 2012 -0800 Refactoring protocolinfo integ tests Cleaning up the protocolinfo integ tests, joining a couple that had a similar prupose and other miscellaneous refactoring. commit 39d93f75533243948d3222f800707dcb320b1a76 Author: Damian Johnson Date: Sat Jan 28 23:42:14 2012 -0800 Refactoring connection integ tests More tidying up of the integ tests. The connection integ used a common port/socket tester function but the common code was tiny enough that it's clearer without it. commit a73be509009942b4b2711b49a3b30e3c3f6e609b Author: Damian Johnson Date: Sat Jan 28 17:06:36 2012 -0800 Rewriting authentication integ tests Excessive testing helper functions are bad. They hurt test readability and makes the code a pita to trace through. This cleans up the authentication integration tests to collapse the helper functions into just the few that make life better. Much more maintainable now. \o/ commit dbf8663cf9b5ca1e31f516109b019045ec63a04c Author: Damian Johnson Date: Sat Jan 28 13:14:28 2012 -0800 Integ check for value of get_system_tor_version() We had basic verification tests for get_system_tor_version() but it didn't check the value. This was a refactoring mistake, iirc it once checked the value but since we can run with an arbitrary tor binary I removed that, forgetting that get_system_tor_version() could optionally take that binary path. Making this a separate test since it relies on the tor instance being accessible (which isn't the case with all targets). commit 6f70bd329ab215754241ea56479a1c9490ab2dce Author: Damian Johnson Date: Sat Jan 28 12:55:26 2012 -0800 Tidying the stem.util.system unit tests Improving the mock call function, clarifying usage in its pydocs and bundling it with the partial call. I'm kinda impressed how much this simple change helps the readability of these tests... commit 9b5e2295a2217081cf772ce3b0f24f88eb3dac34 Author: Damian Johnson Date: Sat Jan 28 12:06:29 2012 -0800 Providing higher test runtime accuracy if low It looks kinda odd to say that the unit testing runtime is "0 seconds", so raising the accuracy when the runtime is less than a second. commit db9d695f16dc2f2ec409be40aa69d7d13cf24826 Author: Damian Johnson Date: Sat Jan 28 11:22:38 2012 -0800 Enum constructor function for capitalized values On occasion we want enumerations where the values are capitalized (for instance logging runlevels and integ targets). This could be done in a succinct fasion via a bit of python hackery but it wasn't pretty. Providing a function to do this instead. commit 7fb7946939ed4801cfe79adbd6f9a0702f4464b2 Author: Damian Johnson Date: Sat Jan 28 11:04:11 2012 -0800 Renaming assert_message_parses to be private Very minor correction to the ControlMessage unit tests. commit 92cf5eb531ab7bfda4361f15b8c5eba92b2050f8 Author: Damian Johnson Date: Sat Jan 28 10:54:08 2012 -0800 Removing pydocs from test classes The documentation in the headers and classes for unit and integ tests were redundant. Joining the two and removing pydoc blocks from the tet classes. commit 9a2b50138356f317220819b2351f666bd45564e1 Author: Damian Johnson Date: Fri Jan 27 18:53:39 2012 -0800 Using mocking module for protocolinfo unit tests Expanding the mocking module to provide a ControlMessage (very frequently needed in tests) and using the module to simplify the protocolinfo unit tests. commit 39e9f0cfdf8efd6cd8445e33feaeb9dd4a44d92f Author: Damian Johnson Date: Fri Jan 27 09:24:59 2012 -0800 Rewrite of authentication unit tests The unit tests for stem.connection.authenticate* functions were the first to use the decent style of mocking that was later used as the basis of the mocking module. Using the module to clean up the unit test and making a stab at making the tests less confusing. commit 55c3fa6e6bfa90e47f116029a08999fe5cc4ac2a Author: Damian Johnson Date: Wed Jan 25 09:54:07 2012 -0800 Target accidently included with early test output When printing errors an earlier change included the target with the errors listed in the footer. This also changed the errors listed earlier when running the tests which was a mistake. commit 030579292593a9e8262edfc713e1d86b379dabf5 Author: Damian Johnson Date: Wed Jan 25 08:48:48 2012 -0800 Dropping Config.sync() in favor of config_dict() The Config class' sync method was a step in the right direction, but the name was confusing and the usage was suboptimal. In the vast majority of cases the caller simply wants a dictionary that stays in sync with the configuration. The config_dict() function is essentially the same as sync but with more succinct calls. commit 9e3834d80b0ae10f8ab5076e00186ddd346b39f5 Author: Damian Johnson Date: Wed Jan 25 07:36:11 2012 -0800 Renaming the Config update method to synchronize() Calling this method 'update()' has confused a couple people. It changes a dictionary to match the key/values in our current configuration. Calling this synchronize() makes more sense though now I should change the sync() method - that's next. commit 2ae0452fb59ef5781e0803919b02cc0f3fa5212a Author: Damian Johnson Date: Tue Jan 24 09:47:51 2012 -0800 Integ testing for multi-line configs commit 1f5b05766ce0c2533d8bd2861d142cc6523e4410 Author: Damian Johnson Date: Tue Jan 24 09:26:59 2012 -0800 Config support for multi-line entries Adding support for multi-line values in the configuration files. I've wanted this for a while, mostly for user facing messages. This still needs an integ test. commit bda94c4dffa97f9cc915cddb45d90aab51e8e5b5 Author: Damian Johnson Date: Mon Jan 23 19:11:07 2012 -0800 Including the test type or target with errors When we have a failed test run with numerous targets we know at the end that an error occured and what it is, but not what target it occured under. It's a pita to scroll back looking for a stacktrace so including this with the footer's error information. commit 1b71039087a0420e4138e4eb61291f6b6a752dd6 Author: Damian Johnson Date: Mon Jan 23 09:48:05 2012 -0800 Integ RUN_NONE target wasn't runnable The RUN_NONE target wasn't being included among the runnable integration targets because it had an empty torrc which failed a check that should have been for the attribute's existance instead. Also correcting the case of the 'TARGETS' enum. While it is a contant, I've been going with class case contentions for enumerations. commit 59764266b9dbce91accd6a55a371f24516b257a7 Author: Damian Johnson Date: Mon Jan 23 09:25:46 2012 -0800 Unit testing for stem.util.conf I needed unit tests for the new listener functions so wrote long overdue tests for everything within the utilitity too. Caught a couple bugs in the process. commit cfb2c383fc487f4a129345c5724b6a73d96c712d Author: Damian Johnson Date: Sun Jan 22 23:03:41 2012 -0800 Implementing testing --no-color arg An option for suppressing color escape sequences is important for when test output is piped to a file (the escape characters then just look like gibberish). Still... output is sooooo ugly without color. :( Routing most of the testing stdout writes through the output module, and either respecting or discarding formatting there based on if we had a '--no-color' argument. commit 92f97544455aee94a229daee20f099f14f92829e Author: Damian Johnson Date: Sun Jan 22 22:24:53 2012 -0800 Eliminating the 'test.' config prefix It's redundant for testing configurations to have a 'test.' prefix. Configurations are already segregated by their handle so they would be only accessed via the 'test' config anyway. My heart isn't set on these config hierarchies but this will do until someone proposes something better. commit 0a5ff78074309be31ce6f71fde2b0eca86a8c3e6 Author: Damian Johnson Date: Sun Jan 22 22:15:52 2012 -0800 Synchronizing runner config Using the shiny new sync() method for the runner config to make the world a better place. commit e7cb3b59298088b4a8adf35364ff6d775559d27a Author: Damian Johnson Date: Sun Jan 22 19:10:41 2012 -0800 Overhaul of run_tests.py Several substantial changes to the run_tests.py script to improve readability: - splitting arg parsing from the rest of the main function - adding a config sync method to keep config dictinaries in sync with the main configuration (this will be especially important for arm since it allows for proper runtime configuration editing) - moving remaining print functions into test/output.py - lots of general cleanup Remaining todo items from this... - still need to add testing for the config listeners - we should note the module that failed with the failures at the end - we still need multi-line config entries - the --no-color option was added but not yet implemented - the RUN_NONE target looks to be broken commit a0be1932e96e9e52a12a4bbc49e72a059c005f78 Author: Damian Johnson Date: Sun Jan 22 13:16:01 2012 -0800 Expanding Config class' set() method Users of the set method would usually expect it to overwrite our current configuration value rather than append to it (thanks to Sathyanarayanan for the catch). Also expanding it to handle list or tuple values. commit d465a9162c58915cbf41c544844cd3520329e3e3 Author: Damian Johnson Date: Sun Jan 22 11:53:35 2012 -0800 Renaming CONN_* targets to RUN_* The former CONN_* targets each represent a run of the integration tests. These were originally called CONN_* because they defined torrc parameters for connecting to tor, but as time went on they became more general. RUN_* (for instance RUN_ALL) is far more intuitive to users. Also dropping the 'integ' prefix from target configurations. It's accurate, but pointless to have. Target options are used frequently enough that we shouldn't make them more verbose without a reason. commit fa934a0c842b86788203199e7dff57a85d7c5387 Author: Damian Johnson Date: Sat Jan 21 23:07:04 2012 -0800 Minor runner refactoring Messing with the order, dropping the get_torrc function, and using STEM_BASE rather than figuring this out again in 'run_tests.py'. commit 6c275ae49fae5ca45c8e596b6119841458a95fb6 Author: Damian Johnson Date: Sat Jan 21 22:37:24 2012 -0800 Renaming get_connection_options() to get_options() The Runner.get_connection_options() method is no longer really connection torrc options, but rather all custom options for our instance. This is frequently used so having a shorter name is nice to have, too. commit d7fa1891aaac5f5907cbff05a8fd2043155b475f Author: Damian Johnson Date: Sat Jan 21 22:14:25 2012 -0800 Replacing runner OPT_* with enumeration Replacing the test.runner.OPT_* constants with an enumeration. This is a little more elegant and lets us get rid of the cluncky RUNNER_OPT_MAPPING dictionary. commit 6c3fb340a83865f75399c60b4e9201810f60c8ee Author: Damian Johnson Date: Sat Jan 21 15:34:43 2012 -0800 Moving test target attributes to config Moving the attributes for integration testing targets from a big dictionary in the source to a separate configuration file. Progress! commit 34da3669909c5e89957bcf77c92f2a00714f921c Author: Damian Johnson Date: Fri Jan 20 10:20:07 2012 -0800 Overhaul of integ testing targets The integration testing targets had grown organically and were partly defined in mulitple places. Now that I've played with them a bit I've realized that testing targets are purely a mapping between user input and runner attributes. Hence I'm removing all notion of testing targets from runner.py, and instead having the run_tests.py handle all conversion between test targets and runtime attributes. All target attributes are now generalized and stored in a large dictionary. Good news is that this makes code readability and maintainability far easier. Need a new target? Just add an enum and the attributes to the hash. Down side is that this is more static configuration than code, and run_tests.py would be far nicer without it. I'll next look into moving this to a test_settings.cfg to abstract code from testing configuration but references to constants (test.runner.OPT_* and stem.version.Requirements) will take a little thought. commit 6846ce359ede2c697688a2e143a3b3625f8a540b Author: Damian Johnson Date: Mon Jan 16 10:09:46 2012 -0800 Removing test.runner.get_connection_type() Integration tests used the get_connection_type() method to either query or infer attributes about the connection it was running against. This was stupid - they should query the attributes directly and make test assertions based on that. The get_connection_type() value was our testing target which should simply be a user friendly tag for a set of testing attributes. Next I'll be moving testing targets completely from the runner so they only exist in 'run_tests.py'. commit 94479aadebcaeee26d57b91e4900dad56edaeb91 Author: Damian Johnson Date: Sun Jan 15 22:13:32 2012 -0800 Minor fixes for conf.save() Sorting the keys prior to saving, handling multi-value keys, and trivial changes (whitespace, reordering, and putting the strip call in the try block which is the only place it's needed). commit 451e13eb9552ab448072b863b180a338a4baeb24 Author: Sathyanarayanan Gunasekaran Date: Sun Jan 15 23:16:06 2012 +0530 Add stem.util.conf.save() This saves all the contents in the config file. It overwrites the contents of the original config file. The comments of the original file will be lost. commit 6c5f0205682877c76d6633abce487673787f5d9b Author: Sathyanarayanan Gunasekaran Date: Sat Jan 14 21:32:31 2012 +0530 Minor update to stem/util/conf.py - More pythonic way of doing File I/O - Use try and catch the exception while parsing the config file instead of using an if-else logic. - Log messages when the (key, value) are not provided in the proper format in the config file. - Typos commit 19491fb8a411a5737ca438b9dc943a13f8eb44b8 Author: Damian Johnson Date: Fri Jan 13 09:15:49 2012 -0800 Integ target for running with ptrace Integraion target that runs with 'DisableDebuggerAttachment 0'. This allows us to exercise use cases that require ptrace with newer tor versions. If 'DisableDebuggerAttachment' isn't supported then we skip the target. commit 4a0e1a755e43802340020b663f6982536bd57ecf Author: Damian Johnson Date: Fri Jan 13 08:46:19 2012 -0800 Skipping integ that need ptrace when inaccessable When the tor connection is inaccessable I'm skipping the system integ tests that need ptrace support, since we can't check if it's enabled or not. commit 0071fadd8b69eda81cdd9826719037fafdebbb74 Author: Damian Johnson Date: Fri Jan 13 08:28:32 2012 -0800 Making explicit checks for tor accessibility Several 'test/runner.py' functions query against our tor instance. However, in some integ tests we don't have any method for doing that. Rather than returning a default value I'm raising in those cases, and adding an is_accessible method to make our tests more readable. commit b01a790b902bfc8a0bb368d6b5bc72c52ff528af Author: Damian Johnson Date: Fri Jan 13 07:43:15 2012 -0800 Placing integ control socket in tor owned dir Tor's new version (0.2.3.10) checks that the control socket is in a directory that's only accessable by the tor user, and refuses to start up otherwise. Accounting for this in stem's integ test startup. commit 27e84652f1bd97747a1360f7632783923cf762b2 Author: Damian Johnson Date: Thu Jan 12 09:51:40 2012 -0800 Making integ account for DisableDebuggerAttachment Stem's system integration tests failed with tor's new version because DisableDebuggerAttachment prevents cwd and port lookups. Skipping those tests when that option is set. commit 479a620ee8d3351e07d1bc86bec3f1107c04cc69 Author: Damian Johnson Date: Thu Jan 12 09:17:20 2012 -0800 Making lsof cwd checks account for permissions With the anti-debugger tor change the /proc//cwd symlink becomes only readable by root. This causes lsof queries for a process' cwd to return the proc path with a message saying that permission is denied, and the sys tools misparsed that as meaning that the cwd _was_ the proc path. commit 13c3975bc3585847dd204d074a66394786bf2007 Author: Damian Johnson Date: Wed Jan 11 10:20:30 2012 -0800 Testing argument to run with a tor binary Adding a '--tor PATH' argument so the caller can select the tor binary they want to run against. commit 11cbf61c2dca7cde628e5eec4f3dd4da7867860f Author: Damian Johnson Date: Tue Jan 10 09:49:26 2012 -0800 Mocking utility functions As the tests grow I've found that I'm repeating a lot of mocking logic. Making a nice, general purpose utility to make mocking easier and less error prone. A benefit of this (and the original goal) is that we'll be able to remove stem.util.system.CALL_MOCKING. Testing code in the library is icky, and once we swap over the system integ tests we'll be able to remove this override function. commit 608a5371370796a3d6ace5deb54bc019783cf2c9 Author: Damian Johnson Date: Thu Jan 5 22:42:33 2012 -0800 Moving log handlers to log util Adding a couple basic log handlers to the log util... log_to_stdout - sends further events to stdout (mostly intended for debugging at the interpretor) LogBuffer - simple handler used by the testing to buffer logged events so we can run over them later commit 1f13b0b665ca22fe11febeeb3d610f5852a3ba49 Author: Damian Johnson Date: Thu Jan 5 21:23:12 2012 -0800 Replacing split("\n") with splitlines() Oops, didn't spot that there was a str method for splitting on line breaks. This behaves slightly differently with respect to trailing newlines and preserving '\r\n' entries, but is something I should have been using... commit 21e8f65ad3e981f0060eb0fc71c2c432f32194b0 Author: Damian Johnson Date: Thu Jan 5 18:39:41 2012 -0800 Improving system TRACE logging Showing request/reply with a debug level message for system calls with runtime is redundant. In case of a call failure we just need to give a debug, and in success we show the debug followed by an improved trace message. commit ed68e97e2d6a27ee979bf75b1e1b7f3df2d6c8d0 Author: Damian Johnson Date: Thu Jan 5 10:39:09 2012 -0800 Skipping unreliable lsof test on osx On OSX the Quicklook process sometimes claims '/tmp' and possibly other common paths too, making the test unreliable there. Skipping is better than a false positive to avoiding the test when running on OSX. Caught and patch submitted by gsathya commit b7e9f5c9314159679c73c707e3a6a15f5ff03fdc Author: Damian Johnson Date: Thu Jan 5 10:34:09 2012 -0800 Implementing TRACE level logging Adding request/reply logging for tor and system calls. I'm also messing with the runlevels and messages of other logging to make the integ test output more readable. commit ad5483352456ab3556068baa43db4fa97c008760 Author: Damian Johnson Date: Wed Jan 4 17:41:10 2012 -0800 Indenting multi-line logging output When sending logging to stdout indenting the lines in multi-line log entries to make them far more readable. commit 228a431b06b2a89182335216b4d9cfce166c6ce9 Author: Damian Johnson Date: Wed Jan 4 17:34:50 2012 -0800 Registering TRACE and NOTICE level names Registering the custom TRACE and NOTICE runlevels with the logging module so their names will appear when we log a runlevel. commit 4788095929ab2a3bfde04ade0b2cdf52fe82dc82 Author: Damian Johnson Date: Tue Jan 3 18:52:14 2012 -0800 Tidying up lower runlevel logging Cleaning up the log output for DEBUG and INFO runlevels we encounter when running the unit tests to be more helpful and user friendly. commit 4cd8ac450fd0f3c80a74c4f7835bee1fd2644ef5 Author: Damian Johnson Date: Mon Jan 2 20:39:50 2012 -0800 Making integ test ps query just match 'tor' The fix in c421b54 checked for endings of 'tor' rather than ' tor', causing things like 'monitor' to be picked up. commit c421b54e52d181e14a6677752fe2c07d2b56a796 Author: Sathyanarayanan Gunasekaran Date: Tue Dec 20 17:13:53 2011 +0530 Remove awk and grep from ps call Instead of piping the output from the ps call to grep and awk, use python itself to parse the result. commit 8c8ad75bd1e3dc5541273336b03d6300f689b66e Author: Sathyanarayanan Gunasekaran Date: Mon Dec 19 22:23:41 2011 +0530 Replace pgrep with ps in the BSD systems Now, the integ test checks if the os is a BSD system with a call to stem.util.system.is_bsd(). If it is BSD, then it uses ps to check if an extra Tor instance is running, instead of using pgrep which does not exist on BSD. commit 29652d978ce7465f288575061c819e77a7e67013 Author: Damian Johnson Date: Mon Jan 2 16:42:14 2012 -0800 Dropping get_system_tor_version() usage from tests At one point our integ tests was using the get_system_tor_version() function to determine our test instance's version. This is fine for now since we run against whatever tor binary is first in our path, but in the future tor devs will want to be able to select the binary so that won't work. Adding a test runner method for querying our version from tor instead. commit a08c09a21743225181569a9ba08b3b15a7c584e2 Author: Damian Johnson Date: Mon Jan 2 16:27:48 2012 -0800 Integ tests for stem.version I didn't have any integ tests for versions? Seriously? *sigh* Adding some basic verification tests that we can handle the version of our system and test instances. Earlier boerni was running into a bug where another integ tests inadvertantly failed due to an uncaught exception from get_system_tor_version(). It was inappropriate for that test to fail (it wasn't testing versions) but we should have some test for the function so this is it. commit b4f86854437d0e81cc1adb18de66ca46c49dabb4 Author: Damian Johnson Date: Mon Jan 2 14:49:12 2012 -0800 Integ for cookie auth failed due to missing cookie If we're running the integ tests for the first time then it would fail because it attempts cookie auth but the cookie file doesn't exist yet. This went uncaught for quite a while because after the first test run we leave the data directory (to speed up future test runs). Caught thanks to boerni. commit ea511db40259a74d57d87a40339b41e66fe34d9a Author: Damian Johnson Date: Mon Jan 2 14:32:13 2012 -0800 Uncaught IOError when checking version in integ If our attempt to look up 'tor --version' failed then we'd trigger an uncaught IOError. We're only checking the version there to determine if we should skip the test due to tor being out of date, so making that a best-effort lookup instead. Caught by boerni. commit 4a5586afb64d50c627652c1ead9314596cb0b36c Author: Damian Johnson Date: Sun Jan 1 10:24:38 2012 -0800 Deduplicating unknown auth type messages There's quite a few log messages that really don't make sense to repeatedly log. Adding optinal logging deduplication and applying it to the INFO level message about getting an auth method we don't recognize. commit ad77d82535a5d86cb9b059d2365ff70cfa260c53 Author: Damian Johnson Date: Sun Jan 1 10:12:18 2012 -0800 Reducing socket/proctol error logging to INFO The base socket and protocol errors are too low level to be of interest to users. They're interested in the symptoms and what it means for them not that, say, a control message was malformed. Also escaping the logged message content - much easier to make sense of that way. commit 5a988db9a78161240a9f3146936a57dfc0a6734e Author: Damian Johnson Date: Sun Jan 1 09:56:26 2012 -0800 Downgrading protocolinfo version mismatch to INFO The response for PROTOCOLINFO queries do not necessarily need to be what we requested. We make a best effort to parse any response as a v1 response, and warned if the response was for something besides v1. Downgrading the warning from WARN to INFO since this is a valid tor response and, if we're still able to parse it, the user doesn't really need to be alerted. If the version mismatch can't be handled like a v1 response then that most likely *will* cause warnings or errors that we'll alert the user of. commit 94e009c8fcba1569b0e2439c626f0031d62a43b6 Author: Damian Johnson Date: Sat Dec 31 13:51:18 2011 -0800 Using log util runlevels for test runner Replacing hardcoded logging values with util abstraction. commit 249035cf3338a10ba1e7e311cb650f10fcbbcb60 Author: Damian Johnson Date: Sat Dec 31 13:34:46 2011 -0800 Moving null-handler logging hack into log util Without a handler the builtin logging class emits a warning. To get around this I added a no-op handler if none already existed before any logging was done via a module import hack. However, now that we're using a log util we can put the hack in that instead. I tried to repro the warning but didn't have any success... maybe the issue only exists for older python versions? Reguardless, this workaround doesn't do any harm so keeping it around. commit af9cdc40e9aa3c8ae625dd6941fb63815ad361f3 Author: Damian Johnson Date: Sat Dec 31 13:23:01 2011 -0800 Clarifying that jail ids only exist on FreeBSD Sebastian and rransom confirmed that jail ids do not exist on OSX and OpenBSD so clarifying that the function is only known to apply for FreeBSD. commit a3f9647832dab69c5eda8047748d0de534fac073 Author: Damian Johnson Date: Sat Dec 31 12:54:49 2011 -0800 Routing all logging through stem.util.log Back and forth I go - first I wrote my own logging then used the builtin python logging module. Now I'm going to a hybrid of the two. The builtin logging module is rich but somewhat hard to customize without a wrapper. Changes we want on the logging module are aliased commands, additional runlevels ('NOTICE' and 'TRACE'), and logging sends complaints to stdout if there isn't a listener (very, very dumb of its author and I've been working around this via a __init__ module hack). Functionally this does not change anything. I'll be changing the modules to take advantage of these new runlevels later. commit d84376a1ff3c459de603878763e4fddff019fa43 Author: Damian Johnson Date: Sat Dec 31 12:13:45 2011 -0800 Making the testing log runlevel customizable I'm gonna be making stem include two runlevels that it doesn't natively support: TRACE and NOTICE. The later is so we have runlevels to match tor, and the former is so we can have request/reply logging without cluttering the DEBUG logs. commit b56b9c627332208913b91357c02267c472644fe9 Author: Damian Johnson Date: Fri Dec 30 10:12:57 2011 -0800 Only issuing WARN for get_bsd_jail_id on FreeBSD The stem.util.system.get_bsd_jail_id() function is only available on FreeBSD (and maybe OSX) so only issuing a warning for failures on that platform. Everywhere else this now gives a DEBUG level message to warn the developer that the function's unsupported. commit 08f3061dc7c03a735a27024ad417fb2e4026a7c5 Author: Damian Johnson Date: Fri Dec 30 09:49:28 2011 -0800 Testing argument to include logging output Adding a '-l' or '--log' argument that includes DEBUG level logging output with the test results. Hopefully by making the log output more visible it'll become more helpful. commit a5023f160e184bb0b800d4789a96c47d411fef2c Author: Damian Johnson Date: Wed Dec 28 17:51:50 2011 -0800 Expanding success/failure label and other fixes Multiple minor fixes and display improvements around how test output is displayed. commit 8795873a50c722419700dec1fff56d289799a1c0 Author: Damian Johnson Date: Wed Dec 28 10:24:58 2011 -0800 Making test results right aligned and bold Instead of showing test results as... test.foo ... ok test.hello_world ... ok I'm now showing them like... test.foo [OK] test.hello_world [OK] with the result text bolded. Thanks to Josh Barr for the idea. commit 285bffdab1c818ee675390701a0539510f41b4f7 Author: Damian Johnson Date: Wed Dec 28 10:01:23 2011 -0800 Moving testing module from results to header Dropping the custom header titles and putting the module name there instead. This means that the module name is redundant with the test results, which was cluttering the output (bad for readability). Thanks to Josh Barr for the suggestion. commit d02aa3e9128e92982e6f2760594523556bd45e85 Author: Damian Johnson Date: Wed Dec 28 09:24:35 2011 -0800 Moving python testing output filters to module Rewriting the hacks I was applying to test output for better readability as a proper module. Nicer code and now much easier to add more filters. commit 839df4492c61777eaf660665a2aeb0d4267251bb Author: Damian Johnson Date: Tue Dec 27 10:24:40 2011 -0800 Connection pydoc spelling corrections Just some quick corrections now that this module has reached maturity. I'll check the other modules later. commit ad7d822afb59692686b9fe3c97b39f5a7b5a5d21 Author: Damian Johnson Date: Tue Dec 27 10:18:02 2011 -0800 Expanding connection header pydocs Explaining what the module is most commonly used for and moving the example to the top. commit a9a240c553cc154f8019c6332ece686202398c34 Author: Damian Johnson Date: Tue Dec 27 09:58:00 2011 -0800 Implementing and testing connect_* functions Similiar to TorCtl.connect(), stem's connect_port and connect_socket_file are convenience functions for trivially getting an authenticated connection. This isn't ideal for applications since it hijacks stdin/stdout and lacks exceptions, however for CLI apps and the interactive interpretor it's very nice to have. commit b7b9ff1f214b87fc84425cb26882ec4f96362736 Author: Damian Johnson Date: Tue Dec 27 05:20:10 2011 -0800 Reordering header pydocs so exceptions are last The ordering of the header documentation was based on the order of things in the file. This placed exceptions at the start which isn't what readers usually want. commit a7772f8c62957ea46db7c3aa012c0ba6b018a3d0 Author: Damian Johnson Date: Mon Dec 26 16:53:20 2011 -0800 Defining when auth success should occur Specifying in the general auth unit tests when authentication should succeed or fail. I'd wanted to also check for the highest priority exception but after messing with this quite a bit I'm waiving the white flag. This is decent enough... commit e642cb669ebc01055ac1873c709ef57585efe99f Author: Damian Johnson Date: Mon Dec 26 13:52:17 2011 -0800 Unit test to exercise all general auth use cases I'm aiming for both expansive and deep unit testing for the general authenticate method and this does the former. It runs the authenticate() function under all combinations of valid input and having authenticate_* functions either succeed or raise one of its exceptions. At present this just checks that the authenticate() function always either succeeds or raises a AuthenticationFailure - an important test, but I'll next try to make it a deeper validity check. commit 39528a51489598b970d1f64cd37a3a5de97fc4cb Author: Damian Johnson Date: Sat Dec 24 17:37:39 2011 -0800 General auth unit test for protocolinfo fetching Unit tests for the general authentication function will be exercising it in various error conditions by mocking out the functions it uses. This first test exercises all of the use cases when the function needs to make a PROTOCOLINFO query and, more importantly, lays the groundwork the rest of these unit tests will use. commit 22e64d083ee9a7f15e3f7333477c9c9dde7609cc Author: Damian Johnson Date: Wed Dec 21 09:37:52 2011 -0800 Integ test for general auth's password arg Testing the various use cases for a password argument (missing argument, incorrect password, and correct password). When we're running connection tests with something other than password auth this is a very dull test (since we succeed without raising a password exception), but when we just have a HashedControlPassword this is a decent test. commit 22481b03901f84e9bc9d1951cc0f7343fbc6eb84 Author: Damian Johnson Date: Wed Dec 21 09:00:58 2011 -0800 Replacing runner authenticate with shiny new auth The test runner authenticated tor sockets manually for our integ tests. Replacing that with our nice, new general auth function. commit 6d30fe9c9ca67a167b0085306ee5590a26f61d4b Author: Damian Johnson Date: Tue Dec 20 10:01:02 2011 -0800 Basic verification integ test for general auth Simple test that checks that the general authentication function can authenticate to all of our tor instances. commit b4ca09871c7650ad6469c3b998ebc174605f4a7d Author: Damian Johnson Date: Tue Dec 20 09:41:29 2011 -0800 Transparently allowing multiple PROTOCOLINFO calls Tor hangs up on the socket for multiple reasons prior to authentication. One of them is if multiple PROTOCOLINFO queries are made. This is annoying and does not make sense to controller callers, so transparently re-establishing those socket connections. commit 1483606df372bebbd5c2b6707f777bfb898f21b8 Author: Damian Johnson Date: Tue Dec 20 09:18:16 2011 -0800 Integ test for general auth example Testing to make sure that the example in the authenticate() function's pydoc works. commit f1a41ecc8534327772f9eca3430e373a2b8a54b8 Author: Damian Johnson Date: Mon Dec 19 09:52:03 2011 -0800 Strip whitespaces from is_running ps results Fix and unit testing for whitespace in the ps results. Also fixing the unit test so we make sure that we get False rather than None when we confirmed that the command wasn't running. commit 292a90b6aac0574453f0187f3705e84aee59f808 Author: Sathyanarayanan Gunasekaran Date: Mon Dec 19 22:20:01 2011 +0530 Fixing the reg exp to accept git SHA1 in Version Originally the reg exp checked didn't take into account the git commit [0.2.2.23-alpha (git-7dcd105be34a4f44)], which produced an Error. Now, the status tag consumes the git SHA1 as well. Appropriate test cases have been added for integ testing. commit def66d45d3cf4c82786fa9a8526a18839fcbf816 Author: Damian Johnson Date: Sun Dec 18 17:25:46 2011 -0800 Implementing general authentication Finally finished the fabled general authentication function, which uses PROTOCOLINFO information for authentication. The vast majority of the effort behind this function was getting the exception hierarchy right. This is totally untested. Unit and integ testing comes next. commit 297a41b085a92eb1c5b5c92464d3bc9cd27cef94 Author: Damian Johnson Date: Tue Dec 13 10:06:53 2011 -0800 Reconnecting socket after auth failures Tor disconnects the control socket after Failed AUTHENTICATE calls in an effort to mitigate the issue discussed in... http://archives.seul.org/or/announce/Sep-2007/msg00000.html This is unintuitive to stem users so I'm making a best effort attempt to reconnect the socket in those cases. This isn't guaranteed to succeed, but checking is_alive() is far nicer for callers than a try/connect/except block. commit 4d0d5598eec4936f103293802c217377ada8751f Author: Damian Johnson Date: Mon Dec 12 18:51:03 2011 -0800 Joining get_protocolinfo_* functions Having a joined socket constructor and PROTOCOLINFO query doesn't make sense. It allowed the user to make a PROTOCOLINFO query in a single line without a leftover socket but that use case is so rare that it's fine for it to take a couple more lines. This api is simpler and better for the common case. commit 8164af5ba0e571dfcd735274e1c4079c79a6fd8d Author: Damian Johnson Date: Mon Dec 12 18:24:44 2011 -0800 Using input version string for __str__ Originally we reconstructed version strings from its components since that's all the Version class had (a separate factory translated strings into Version objects). However, that's no longer the case and much simpler if we just have the __str__ method provide the constructing string rather than try to recreate it. patch by gsathya commit a584686dd01f12c8209ea7bc55019fec752eeaa2 Author: Damian Johnson Date: Mon Dec 12 18:05:23 2011 -0800 Defaulting ControlSocket constructors to connect ControlSocket users almost always want the socket to be connected initially to defaulting it that way. commit 98ad681bc05d91b596ceebb8f46e04f1f8598b84 Author: Damian Johnson Date: Fri Dec 9 10:01:27 2011 -0800 Removing tag from version comparisons As Sebastian pointed out, the version-spec says "The status tag is purely informational". Removing it from comparison operations so versions like "1.2.3.4-foo" and "1.2.3.4-bar" will evaluate to being equal. commit bd51ff4d69decfc866366a5381bd1ba0dc7ee866 Author: Damian Johnson Date: Fri Dec 9 09:47:17 2011 -0800 Testing and fix for invalid auth cookies Adding a test and fix for when the authentication cookie is the right size but has the wrong contents. commit 76be3997e31d5a56e335a111eb1f3c823dbc9787 Author: Damian Johnson Date: Fri Dec 9 09:06:26 2011 -0800 Adding an authentication exception hierarchy For arm a substantial torctl pain point has been poor exception raising from authentication functions. If everything raises a ValueError then the caller needs to parse message responses or re-implement the code to get finer error granularity. Providing a hierarchy of authentication exceptions that can both make life simple for callers, letting them catch the exceptions they want then fall back to a catch-all for the AuthenticationFailure base. One mildly icky point is that this relies on the error message provided from tor to differentiate between when an authentication value is rejected verses the type. That said... - this issue is noted in the pydocs - I have integ testing to check for error message changes - callers are encouraged to use a general authenticate method which doesn't have this flaw since it uses the PROTOCOLINFO response (barring edge cases like a buggy PROTOCOLINFO response or the auth type changing in the split second between PROTOCOLINFO/AUTHENTICATE) - even if the message changes direct callers will still be getting the right category of exception Pydocs reference a general authenticate method but it's not included yet (first draft is written but needs some work and testing). commit 3b89feb04f2d9389e387f7d2e1af5f69158d52e9 Author: Damian Johnson Date: Fri Dec 2 07:47:49 2011 -0800 Issuing IOError for cookie read failures Replacing OSError for read failures with IOError (more appropriate for this). commit 8fd556572a170d06359458a37848d947313984d2 Author: Damian Johnson Date: Thu Dec 1 10:10:43 2011 -0800 Function and testing for cookie authentication Adding a function for password authentication. This includes checks for the file's existance and that the size is valid (for 4303). commit 2c91e342a58dd389d5cb777cb1f4f59bfc198f71 Author: Damian Johnson Date: Wed Nov 30 19:02:32 2011 -0800 Skipping get_pid_by_name test when doomed The get_pid_by_name function fails when there's multiple instances of a process with that name. As a result the integ test for the function fails if there's extra tor instances running on the system. Using pgrep to check for other instances and skip those tests if they'd be doomed to failure. commit 7f760f86414ee0bfbd050480e1753555c66e9a5b Author: Damian Johnson Date: Tue Nov 29 10:02:24 2011 -0800 Function and testing for password authentication Adding a function for password authentication. This included escaping quotes but otherwise is trivial - most of the effort was refactoring the authentication integ tests. commit 312423c9d0b68cea486c3269ddcaf95fcb3314cc Author: Damian Johnson Date: Mon Nov 28 10:08:20 2011 -0800 Function and testing for null authentication Function for authenticating to open connections and integration testing for it. The tests both check the happy case and responses we get in a variety of 'authentication needed' scenarios. commit c033f19b081149f49a65cb6ab79e836af5352a2e Author: Damian Johnson Date: Mon Nov 28 09:39:14 2011 -0800 Using ControlSocket for integ tests Replacing the send_message and recv_message calls via raw sockets with the ControlSocket class. Neither of these integ tests are for testing those methods and the higher level objects make the tests much more readable. commit 206e4de3ffab8764aefbc02d969092cc47dfc37a Author: Damian Johnson Date: Mon Nov 28 07:00:45 2011 -0800 Exercising the get_socket arg with integ testing The get_socket (previously keep_alive) argument wasn't being exercised so adding that to the test for fetching a protocolinfo response via the control socket. commit 04975ff9d149d8a881edd8ce26925d785150eda1 Author: Damian Johnson Date: Mon Nov 28 06:56:38 2011 -0800 Splitting socket attribute from protocolinfo Bundling the requesting socket with the protocolinfo response was kinda clunky. I thought that it owuld make the api a little nicer, but in retrospect it's just weird so going to a more conventional tuple response instead. commit 7bae33db31f26440810596b0b702d7f85bbfb1cd Author: Damian Johnson Date: Mon Nov 28 06:18:03 2011 -0800 Specialized subclasses for ControlSocket Adding a ControlSocket subclass for control ports and control sockets. This allows for a connect() method which we'll need when trying multiple connection types since the socket becomes detached after a failed authentication attempt. This is also gonna be a bit nicer for callers since it bundles the connection information (the port/path we're using) with the socket. commit ec5c082e570123b011e0d9cb416d5ede1eba5c95 Author: Damian Johnson Date: Sat Nov 26 10:23:48 2011 -0800 Moving scratch control connection into a module Later there will be a stem.control for the general controller code (ie, most of what TorCtl encompasses on its surface). Moving the first draft at that out of stem.connection, which it didn't really belong in anyway. Now none of the modules except control contain untested, scrap code. commit 5b505ec579ab7edcaf55d1e35a2ccd3b9aee6672 Author: Damian Johnson Date: Sat Nov 26 10:10:21 2011 -0800 Fixing relative cookie expansion test A couple protocolinfo tests filtered system calls so that pid lookups by process name would fall and we'd fall back on looking it up by the control port or socket file (to exercise alternative code paths). However, I'd forgotten that this would also filter out the get_cwd lookup calls, causing those tests to fail. The relative cookie expansion by socket file wasn't being exercised at all because I didn't have a integ test configuration where we had both a control socket and authentication cookie. I've added this test now and fixed this issue with the socket test too. commit fd90d6c50011b8764b1c82c448fe39eab76c9c20 Author: Damian Johnson Date: Fri Nov 25 22:55:57 2011 -0800 Using space or newline div when logging When logging a multi-line message using a newline divider with the "Sending:" or "Receiving:" prefix, otherwise using a space (minor bug had the space always included previously). commit 8f8257d4ddcf6830b2413f53890a1fab4414ab45 Author: Damian Johnson Date: Fri Nov 25 22:42:35 2011 -0800 Only respecting open default when undefined When no target is defined we should have a test.runner.TorConnection.OPEN default for integraiton tests. However, if we have an alternative connection target then this should be overwritten. commit e313bb35c827fdb75a382d7c5e837692baeead73 Author: Damian Johnson Date: Fri Nov 25 22:36:57 2011 -0800 Replacing protocolinfo lookups with ControlSocket Replacing raw socket use in the protocolinfo lookup functions with the ControlSocket class, and attaching it to the responses instead. commit 62ecde1d4432eb96a3346289c8257561baa4faff Author: Damian Johnson Date: Fri Nov 25 22:13:56 2011 -0800 Refactoring version functions into module Moving the last of the types.py contents and a related function from process.py into a module specifically for handling tor versions and requirements (the later part will grow as the library matures). commit 9a06ff17de582e834e2791c096bd9a51834a6d66 Author: Damian Johnson Date: Fri Nov 25 17:54:59 2011 -0800 Moving control message handling into stem.socket Making a module for all low-level message handling with control sockets (ie, pretty much all of the library work done so far). This includes most of the code from the grab bag 'stem.types' module and the addition of a ControlSocket class. The socket wrapper should greatly simplify upcoming parts of the library. commit e3d4311962b7e95b882a9a50b2402b17f2f2d3d9 Author: Damian Johnson Date: Wed Nov 23 10:05:15 2011 -0800 Showing shorter single-line controller messages When controller messages are on a single line logging them that way too, making the output a little more readable. I should probably file send/recv at a trace runlevel or with a separate logger... commit 8dc796d142bebb6d370188f7e5be6bf65e402743 Author: Damian Johnson Date: Wed Nov 23 09:55:16 2011 -0800 Function for writting to control sockets Writing directly to the socket file isn't hard (it's just a write and flush). However, this is nicer since it wrap the control formatting, logging, and exception quirks. Functions still need unit tests and I might just wrap the socket object completely... commit f98822f01e00197140ccea9dc88690d6ece004df Author: Damian Johnson Date: Wed Nov 23 09:23:42 2011 -0800 Defaulting open connection integ target All connection targets were being defaulted to false, causing plain "run_tests --integ" runs to be no-ops. Hacking in the default values. I should probably use the more conventional dict/update pattern later. commit 1c2337a71f9f19e262cd222679bf66eb046a9e73 Author: Damian Johnson Date: Wed Nov 23 06:47:11 2011 -0800 Nicer logging when receiving controller data Changing the logged controller messages a bit. This is a bit less faithful to the raw controller response but it's more readable. commit b3afd554bad16bcde8b5ed5a70ed2a34c7e84762 Author: Damian Johnson Date: Tue Nov 22 10:13:39 2011 -0800 Better exercising cookie expansion in integ tests The get_protocolinfo_by_* functions weren't exercising cookie path expansion by port or socket file because lookups by process name would succeed and bypass this logic. Added a filter to the integ tests so we exercise both. When running with both the 'RELATIVE' and 'CONN_COOKIE' targets this reveals a bug with the stem.util.system.get_pid_by_port function that I'll address next. The test using the socket file passes. commit 03d2e84894822825fc59507cdc93599f81cd8e36 Author: Damian Johnson Date: Tue Nov 22 10:06:48 2011 -0800 Noting at the end of testing if it passed/failed As testing output has gotten longer its become less clear at the end if all tests passed or not. Adding a note at the end saying if they all passed and, if there were failures, what they were. commit 9fb52210494558a63d494d5ceab7a01282e8f88b Author: Damian Johnson Date: Tue Nov 22 09:53:30 2011 -0800 More selective options for running integ tests Providing targets for all of the tor connection configurations so the user can opt for any combination of targets. Previously you needed to run the 'CONNECTION' target which exercised them all and took around forty seconds to run (kinda a pita if you just want to test cookie auth). commit d3b8f0aed7d715721724a7a94e96a188f1ee4e79 Author: Damian Johnson Date: Mon Nov 21 10:13:10 2011 -0800 Integ target for running with a relative data dir Having a relative path for our data directory can cause headaches since tor then provides relative paths for the data it gives (for instance, for the authentication cookie location). Adding an integration testing target to have a relative data directory, to better exercise the path expansion code. commit 56c2ea8c33e63456dc1316f3936439478e12cc03 Author: Damian Johnson Date: Mon Nov 21 09:21:21 2011 -0800 Function to query PROTOCOLINFO via control socket Same as before, implementation and integ sanity check for making a PROTOCOLINFO query via a control socket. This has the common bits between it, the control port function, and a bit of the PROTOCOLINFO response parsing delegated to helper functions. commit 7f89ff58d2ad6ecbde7f8225e7b2c02ec400be1b Author: Damian Johnson Date: Mon Nov 21 08:36:20 2011 -0800 Function to query PROTOCOLINFO via a control port Implementation and integ testing to query a PROTOCOLINFO response via a control port. Next is to do the same for control sockets. commit e17bfdd0c44b004468bd3121fd5408e2a29074fc Author: Damian Johnson Date: Mon Nov 21 07:51:00 2011 -0800 Resetting system call mocking after test The protocolinfo test mocked system calls but didn't reset the mock when it was done. This didn't cause any errors but that was only luck (the system unit tests probably ran afterward and cleared the mock when it was done). Oops, this is gonna be an easy testing bug to introduce... :/ commit 708e082fb7bdddf2261a44fb290d20d768bc180c Author: Damian Johnson Date: Mon Nov 21 07:42:14 2011 -0800 Moving PROTOCOLINFO tests to be grouped by subject All the PROTOCOLINFO related tests might as well be together. Shuffling them around so all the tests can reside in a test/*/protocolinfo.py rather than have separate protocolinfo_response.py, protocolinfo_query.py, etc. commit 07fd7f7d23f5312d4b4ee4ef64d63fbe97a911ba Author: Damian Johnson Date: Mon Nov 21 04:29:10 2011 -0800 Expanding coverage of PROTOCOLINFO integ test Providing the assertions for all connection configurations in the integ test. commit 22667eaa6a02e6585ad502ea164fce15cd96cc1f Author: Damian Johnson Date: Mon Nov 21 03:59:27 2011 -0800 Unit test for comparing Version class with others Had a pesky bug where comparison of Version instances with other classes would raise an exception and had forgotten to add a unit test - fixing that. commit 3d14e1bd1076bdfd6fd109be0b76f71b12fbebaa Author: Damian Johnson Date: Sun Nov 20 15:51:29 2011 -0800 Running integ tests multiple connections methods Adding a 'CONNECTION' target that, if set, will run integration tests with multiple connection and authentication methods... - no control connection - control port with no auth - control port with an authentication cookie - control port with a password - control port with both an authtication cookie and password - control socket This means running through the integ tests six times which currently results in a runtime of arond fourty sectons, so this isn't the default. The primary purpose for doing this is to exercise the PROTOCOLINFO parsing and upcoming connecion methods with all of these tor configurations. The ProtocolInfoResponse integ test doesn't yet actually test all of these - fixing that is next. commit e62502661059e8a3c93babf3e15df8a0a07d529d Author: Damian Johnson Date: Sun Nov 20 14:22:07 2011 -0800 Adding a set method for config instances commit da937730bb3aaa303e584eb3e23494703923c46b Author: Damian Johnson Date: Sun Nov 20 14:20:13 2011 -0800 Using enum.__iter__ for values rather than keys Enumeration keys are of very limited use. Iteration over an enumeration should give the values instead so swapping values() and __iter__() to be keys() and __iter__(). commit a34316c59efbce56b82214bc257d8ddfa11e489c Author: Damian Johnson Date: Sun Nov 20 00:55:15 2011 -0800 Runner startup argument for tor connection type Integration tests are about to get an option for exercising multiple connection methods, so adding a runner initialization argument for starting with a torrc for all of the connection methods that we care about. This also includes a minor fix where we'd get a stacktrace when the torrc had an empty line. commit 7b302e2eac72d9c3b4f09b705fd6c3adcb755f84 Author: Damian Johnson Date: Sat Nov 19 23:58:26 2011 -0800 Integration test for general PROTOCOLINFO parsing Integ test for parsing a PROTOCOLINFO reply from our general integraion test instance. We'll need a separate target for testing multiple connection methods (password auth, cookie auth, and control socket). This also includes a fix for the Version class (equality checks with non-Version instaces would raise an exception - didn't expect __cmp__ to be used for that...). commit 2c4686adcecbe73c32250b6201a3970284a1ca8a Author: Damian Johnson Date: Sat Nov 19 23:16:26 2011 -0800 Unit test for relative cookie expansion Now that we have system call mocking we can have a unit test for expanding relative cookie paths. It kinda bugged me that testing wasn't complaining when we had a system api change. :) commit 5847301f6011df37d517041c00ff23419e107316 Author: Damian Johnson Date: Sat Nov 19 22:25:21 2011 -0800 Using new system api for protocolinfo queries The protocolinfo uses system utils for expansion of relative cookie paths. Making it use the new api (the breakup of the get_pid_by_* functions was largely for this class). commit 314804dfce5069a13f07b79e5640e2e1b57e924f Author: Damian Johnson Date: Sat Nov 19 22:05:52 2011 -0800 Dropping log inference from conf util The config's get method inference for logging runlevels no longer makes sense since the log util has been removed. Dropping this inference entirely rather than trying to make it work with logging - those config options have always been unused anyway. commit ff5895f821370fe34609388fa44ebfb6cb1ff923 Author: Damian Johnson Date: Sat Nov 19 15:37:11 2011 -0800 Color for testing output Applying color to the unittest output: green for success, blue for skips, red for failure. Bit easier on the eyes and makes issues easier to spot (at least on my terminal). commit 70ab8ff1c2ccb858abe462dd1f9c3e81c1733659 Author: Damian Johnson Date: Sat Nov 19 14:18:13 2011 -0800 System test bugs caught by pylint Couple copy-paste bugs with the system integ tests. commit f6560b1210a20bdee7ccb68567b44b7f877a3e38 Author: Damian Johnson Date: Fri Nov 18 23:23:28 2011 -0800 Rewrite and testing for system utilities Spent most of this week improving the implementation, api, documentation, and most importantly testing for the system functions. They now have almost complete code coverage by both unit and integ tests. Besides the obvious, this will help cross-platform compatability in the future since I'll have a sampling of input for platforms I don't have. Generated real output for all commands except sockstat (I only have access to linux and mac, not free/openbsd). I'll probably contact Fabian for help with this one. commit dbe041ac1bbb775f71f34f0d705d4dd654d74858 Author: Damian Johnson Date: Sun Nov 13 16:21:09 2011 -0800 Header documentation for the proc and system utils commit 382bc5627b8d4f29645207628ca224f08f7b6883 Author: Damian Johnson Date: Sun Nov 13 16:13:33 2011 -0800 Basic integ test for stem.util.conf Adding an integ test for the example given by the conf utility. There's a whole lot more that could be tested in that class (especially parsing and type inferences) but this doesn't seem too worth while so just adding this basic test for now. I might expand it later. commit b34f8990b6604996d7d60d139267ae43bac1de02 Author: Damian Johnson Date: Sun Nov 13 15:09:51 2011 -0800 Shuffling integ tests to match unit tests Unit tests are nicely categorized under their respective modules, so reordering integration tests to match. commit d30e229c325bc243697d1d015a7bda03082c9756 Author: Damian Johnson Date: Sun Nov 13 15:02:47 2011 -0800 Unit tests for enum commit db7635abc7878d873a989c35e7aea413ecde2f52 Author: Damian Johnson Date: Sun Nov 13 14:38:56 2011 -0800 Revising enum docs and methods Minor changes including... - standard header documentation - replacing the keys() method with making enums iterable (functionally the same, but a little nicer for callers) - dropping the alternative LEnum - I've never used it commit f9aeefb2805e2ef9cd63f236c7a27308718575e9 Author: Damian Johnson Date: Sun Nov 13 14:01:55 2011 -0800 Rearranging conf util to improve usability Adding the standard configuration header (... sooo helpful) and moving the rest of the util's contents around to be more intuitive. commit fc562144fd8a333ad517575976328bc2dc3cec5d Author: Damian Johnson Date: Sun Nov 13 13:43:59 2011 -0800 Including stem prefix for paths in stem comments commit 2b797d0ba9310a015bef5bc2605020ebe8ffafdd Author: Damian Johnson Date: Sun Nov 13 13:15:49 2011 -0800 Unit tests for PROTOCOLINFO responses commit 5756f9940ff0bbd55330a700d648af347dec713a Author: Damian Johnson Date: Sun Nov 13 02:19:26 2011 -0800 Parsing and class for PROTOCOLINFO responses Finally have enough plumbing in place to write the parsing for the PROTOCOLINFO queries. I'm pretty happy with how it turned out - next is testing for the class, then moving on to functions for issuing the PROTOCOLINFO queries. commit 5d200b1cdbebcd60cadf6455a9b1baf1d9431fb3 Author: Damian Johnson Date: Sat Nov 12 16:19:34 2011 -0800 Avoid adding duplicate null logging handlers Checking if logging already has a handler before adding a NullHandler, to avoid having _every_ import trigger an extra no-op handler. commit efe28987ffd27e69a2c6c566bdad2efc5f16e93e Author: Damian Johnson Date: Sat Nov 12 15:55:09 2011 -0800 Expanding contorller exception types Exception types pretty much mirror what TorCtl has (protocol error, socket error, and controller closed) with a base type users can catch instead. One difference though is that if stem functions raise a socket.error (without documenting that they do) then that's a bug - those errors should cuase a stem.types.SocketError instead. commit d1c3c8db724eb17c754c26f66016315d0524a74a Author: Damian Johnson Date: Sat Nov 12 15:46:13 2011 -0800 Standardizing on suppress_exc args for sys util The system functions are mostly best-effort, platform specific attempts to retrieve data and usually don't have meaningful information to provide back to the caller via exceptions. I'm defaulting to have them return None, with an optional arg for having them raise IOError exceptions on failure instead. I've gone back and forth on the fencepost here a few times, and I'm still not positive if this is the right choice. Might change this later if it doesn't work out. commit fab872725d791e8a9ca20f28f6eb5c5358aff87b Author: Damian Johnson Date: Thu Nov 10 09:30:06 2011 -0800 Function for querying a process' pwd Implementing and testing a stem.util.system.get_pwd(pid) function. This will be needed for correcting the relative cookie paths mentioned in... https://trac.torproject.org/projects/tor/ticket/1101 commit 7c7bd6adeca59e593635d6eb21c6a9a0e7cf585e Author: Damian Johnson Date: Wed Nov 9 10:23:07 2011 -0800 Fully qualifying util paths Changing the conventional usage of util imports from "from stem.util import X" to simple imports (not pulling them into our namespace). There's some exceptions to fully qualified util usage where it hurts readability (often the case with stem.util.term), but explite paths will be the more common case. commit 713eec7b70c06ab3361f22fb49f6eb6a8b9e82d3 Author: Damian Johnson Date: Wed Nov 9 10:13:55 2011 -0800 Replacing custom logging util with python builtin Finally abandoning my custom logging implementation for the python builtin logging module. It's far more customizable, standardized, and what users would expect from a library like this. The only disadvantage is that it doesn't buffer past events so we lose anything prior to adding a handler. commit 78de4e4cf1f442b50ec1b30733127f9b83a97fce Author: Damian Johnson Date: Tue Nov 8 09:04:16 2011 -0800 Util for reading proc contents Rewrite of arm's utility for parsing proc contents, changes including better error handling and refacotring for stem's coding conventions and commenting. While writing the PROTOCOLINFO response handler I needed a system util for querying tor's pwd, which works best with proc information, so pulling it all in. For now most of this isn't being exercised, but will be later. commit eb5b522430bc32acb0f8f18c1e92e8ae7782df62 Author: Damian Johnson Date: Sun Nov 6 11:24:21 2011 -0800 Adding optional key arg to is_next_mapping checks Option to simplify testing for the next key being parsed. commit 40b7ab1b36dd4bd0ce455714ea8a9dfd4d3bb76b Author: Damian Johnson Date: Sat Nov 5 17:18:10 2011 -0700 Unit testing for stem.types.ControlLine General unit tests to exercise the ControlLine class with PROTOCOLINFO output. This also has a minor fix so we throw an IndexError rather than ValueError when pop_mapping() is called while empty. commit c0525d58a7e98e2c4b3fa7e2130e69e85aee5247 Author: Damian Johnson Date: Sat Nov 5 15:49:48 2011 -0700 Unit testing stem.types.ControlLine.pop() examples Adding unit tests and correcting the pydocs for the pop method examples. commit 78d44b0995fb607715d189051a6aec13f512ad35 Author: Damian Johnson Date: Sat Nov 5 14:15:33 2011 -0700 Moving testrc.sample into testing module Tired of having 'test' autocompletion trip up on the sample rc file, so moving it into the test directory. It's probably not of much interest to general library users anyway. commit c4c998f6201dae0123c068c3e081a26d6af8a99b Author: Damian Johnson Date: Sat Nov 5 14:10:07 2011 -0700 Better organizing unit tests for stem.types commit aba675c3f565aee7f1855215e90b1d7af26a1ce8 Author: Damian Johnson Date: Sat Nov 5 13:56:13 2011 -0700 Header documentation for ControlLine class commit 12064b7b9fc4799b50ff83cffbab55829c0e1046 Author: Damian Johnson Date: Fri Nov 4 10:16:28 2011 -0700 Replacing get_entry with ControlLine class Making a string subclass to help with the parsing of controller output. Most entries are space separated lists of elements, which this class has functions to easily parse. For controller messages that don't follow this pattern we can still treat it as a normal string. Next is to add header documentation and tests. commit 45d3c819b21dd29aec70178f66777a365200a617 Author: Damian Johnson Date: Thu Nov 3 07:07:11 2011 -0700 Unit tests for types.get_entry examples Unit tests for the examples in the pydocs of the function and fixes for the issues they uncovered. commit a090373e5384026295b31daee8ef622a589a913d Author: Damian Johnson Date: Tue Nov 1 18:49:55 2011 -0700 Utility function for parsing control messages Trying out a utility function to pop the first item off a control response, with the caller providing attributes it should have. commit 263dfd53cec592c10d9dbe40211ea56915ac7552 Author: Damian Johnson Date: Sun Oct 30 00:51:47 2011 -0700 Config argument for the test runner Integration tests can have custom behavior via a testrc, which is simple for now, and will grow as tests become more complex. Previously I was loading a static settings.cfg but that was stupid. The user's config file should neither be hardcoded nor under version control. This change also includes the config loading in the status output to stdout. commit 63a9a5fa524e352c94bab7beabd5052e4972c6c4 Author: Damian Johnson Date: Thu Oct 27 20:31:43 2011 -0700 Merging version constructor and parser The version constructor wasn't really useful so merging it with the get_version function to make its usage more intuitive. commit 52be689448931b1eca7de3add345f9aa2d79319f Author: Damian Johnson Date: Thu Oct 27 19:43:45 2011 -0700 Putting stem under the LGPL v3 commit f07c6cc77f4ad77b2390d03f9aad5f32aecd245e Author: Damian Johnson Date: Thu Oct 27 09:39:41 2011 -0700 Expanding header documentation Adding a summary of a files functions/classes/methods at the top of the file. This makes using a library much nicer. commit 11b2a71d3a6ae24365ebd1e9a30c323a6a3ce275 Author: Damian Johnson Date: Thu Oct 27 09:17:26 2011 -0700 Caching for stem.process.get_version() Caching results of the stem.process.get_version() function to avoid unnecessary lookups. commit 7766cb79fff29020b72b5fe45fc43d184abada23 Author: Damian Johnson Date: Thu Oct 27 07:16:06 2011 -0700 Requirement check for the config-text test An integration test requires 'GETINFO config-text' requires tor version 0.2.2.7 so adding a check for it. commit fbfa73a099d9645f18d9846420cbf0145065b11d Author: Damian Johnson Date: Wed Oct 26 20:13:34 2011 -0700 Rewrite of integration test runner Several imporvements for the integration tests, most notably including... - Test configurability via a 'test/settings.cfg' file - Thread safety for runner usage - Vastly better startup time for how integration tests run by default... - Reusing data directory so we don't need to request as much from authorities when starting (faster startup and less burden on them). Users can opt for a fresh temporary directory instead by setting 'test.integ.test_directory' to a blank value. - Starting tests when bootstraping reaches 5%. This is enough for tests that don't require network activity to run, and we can explicitly run those tests by setting the 'test.integ.run.online' option. This change also means that we can now run integration tests while offline. commit 51d2ded44fe48f54c63cf4e5937a2ff91fa99e95 Author: Damian Johnson Date: Fri Oct 21 08:10:57 2011 -0700 Adding function to query tor's version Command to issue a 'tor --version' query with a stem.types.Version response. commit 52768d36f85083923c4f09a1b0956ee146ae0b16 Author: Damian Johnson Date: Fri Oct 21 07:33:40 2011 -0700 Quietly stopping integ launch on ctrl+c Keyboard interrupts while launching tor would result in a stacktrace. Quietly exiting instead since this isn't really an issue. commit 8293c518d02f879110d45e0cff074e7d31d2fbfb Author: Damian Johnson Date: Fri Oct 21 07:28:27 2011 -0700 Making the tor launching timeout an arg Moving the timeout for launching a tor process from being a constant to being an argument of the function. commit 0a083c8b73a663d38f473e0180b0009d28e1b5d9 Author: Damian Johnson Date: Thu Oct 20 10:09:36 2011 -0700 Accidently suppressing integ asserts A previous commit to suppress errors from the close() method in python 2.7 also suppressed the following asserts. These asserts still work with 2.7 so removing them from the try block. commit 3a913b6b92796f2195c28e55ead532a0fef906e6 Author: Damian Johnson Date: Thu Oct 20 09:58:37 2011 -0700 Moving tor launcher into stem function In writing the stem integ tests I needed a function for starting tor then blocking for its bootstraping to complete. Yesterday Jake mentioned that he could use a launch_tor() function in TorCtl so I'm generalizing this code and moving it into the stem lib. commit 26374bafd5957d3039cf10596de0f7a021f1afb9 Author: Damian Johnson Date: Thu Oct 20 09:49:45 2011 -0700 Message integ tests compatability issues The message integration tests had compatability issues with... - Newer python versions (2.6 -> 2.7) due to an unexpected failure when calling close() on the socket's file object. We're operating with a closed socket at that time so all bets are off about if this should/shouldn't raise so the difference in behavior just warranted a testing workaround. - Older tor versions don't support 'GETINFO config-text'. I've left this as a TODO note for now and will add a version check for that test later. commit 1b44b967e75b08d6f702ffa1507d8ad8c4980bda Author: Damian Johnson Date: Mon Oct 17 09:45:43 2011 -0700 Integration tests / fixes for types.ControlMessage Adding integration tests for basic control port communication, exercising... - connection failure - bad commands - bad getinfo queries - general getinfo queries - setevent/basic event parsing This also includes fixes for a variety of issues found while testing. commit 24d4881025c891c829a63de9b9b1b8849c8c4faf Author: Damian Johnson Date: Sun Oct 16 19:30:57 2011 -0700 Moving integ runner into testing base Moving the integ runner module from 'test/integ/runner.py' to 'test/runner.py'. The 'test/unit/*' and 'test/integ/*' are for test cases, and will later be kinda crowded. The runner is special since it doesn't test, but rather provides the runtime context for the integ tests so probably best to keep this separate from the rest. commit aa01c0798960a520ac1da81e5a1a447c29dbb962 Author: Damian Johnson Date: Sun Oct 16 19:23:57 2011 -0700 Adding get_pid and get_bsd_jail_id util functoins Stealing arm's getTorPid and getBsdJailId from the torTools util, generalizing both functions to be for arbitrary processes rather than just tor. This also adds unit tests for get_pid and a simple exercise of get_bsd_jail_id (I can't really test the later since I'm not on BSD). commit 3731bba464e22dc5501b7b9ae62c29ebc34ad7ba Author: Damian Johnson Date: Sun Oct 16 17:27:57 2011 -0700 Adding BSD compatability to system.is_running Copying over an arm fix I made earlier this week from util.torTools.isTorRunning so that it would work on OSX/FreeBSD/OpenBSD. commit 1a30fbd6526b243468a6ca92f0223eb84ff9b21b Author: Damian Johnson Date: Sun Oct 16 17:19:01 2011 -0700 Integration tests for stem.util.system functions Very, very basic tests that the tor process is running and available. commit 0c03d4eb2a04f71942c53caf6314c25ce66be18e Author: Damian Johnson Date: Sun Oct 16 15:19:03 2011 -0700 Moving integration test init/shutdown into class Making a Runner class which will be available to integration tests. This both better modularizes the code and will allow for us to give runtime context to the tests. commit bdbf352ad7f56aec17ce0156eaf53c9a6ac1fe68 Author: Damian Johnson Date: Sun Oct 16 01:17:38 2011 -0700 Running tor instances for integ tests First draft for the startup/shutdown of integration tests. This... - makes a test directory - generates a torrc - starts a tor instance, waiting until its bootstrap completes (timing out if it gets stuck) - runs tests (not done yet - those are next) - shuts down the tor instance This'll certainly go through some refactoring to better modularize, but it's functional as-is. commit a2fde9374e188725b9976c42f8a679814d87fac1 Author: Damian Johnson Date: Fri Oct 14 20:33:12 2011 -0700 Adding util for handling configuration files Integration tests and other things will be stored via configuration files (hardcoding data in source is a horrible thing and you have no excuse unless you're writing in LISP). This is being stolen from the arm codebase, with the code refactored for this project's conventions, some fixes to better generalize the util, and vastly improved documentation. Tom has been using the config util a bit for his torperf rewrite and having difficulties so hopefully this will make its usage easier. commit c74ac0f7f0c75d429a7d32f7d516b3518fc90ff0 Author: Damian Johnson Date: Wed Oct 12 09:37:49 2011 -0700 Tesing and fix for disconnected socket reads Adding a unit test and fix for when the input file is derived from a socket that's never been connected. I'm suspicious that this won't catch disconnects that accur a little while into the socket's use, but I'll need to implement integration tests for that. Guess that's next... commit 05b106bb575f669487fc376a29b7710d2addb2b0 Author: Damian Johnson Date: Wed Oct 12 09:17:59 2011 -0700 Protocol error tests and fix Expanding the ControlMessage unit tests to cover causes of protocol errors, and fixing an issue that this revealed where it was possable for a malformed line prefix to go undetected (if, say, a dropped character caused a valid line divider to fall into that place). commit 5b1a7e8f2d375db5ad4d3c212fbae93ed4724277 Author: Damian Johnson Date: Mon Oct 10 21:24:58 2011 -0700 Basic unit tests for ControlMessage Just some sanity checks that ControlMessage parses common GETINFO responses. I'll expand on these including some error cases next. commit 92f78d459497dd3c0d688c040e06d6cc6668c43c Author: Damian Johnson Date: Mon Oct 10 19:54:30 2011 -0700 Moving ControlMessage and parsing into types The ControlMessage is a standalone class so moving it into types. This is both appropriate for the types module and will make testing easier. commit 5d7688e7233a5a1f8a4bb4a1c939639031a71aa0 Author: Damian Johnson Date: Mon Oct 10 10:05:14 2011 -0700 First draft controller connection This is a functional rewrite of the sendAndRecv TorCtl functionality. It follows a similar pattern, having a couple threads to continually pull the socket and provide event notifications. This still needs testing, better exception handling, and some more thought about the ControlMessage api. commit 1aff6b875213d70a05d0cb5d86a5ff9914ce6e41 Author: Damian Johnson Date: Sun Oct 9 03:24:44 2011 -0700 General system utilities Adding some simple utility functions that I'll need later. These are rewrites of their arm counterparts, which I wrote before discovering the subprocess module. commit 00739ee4ebf3c74d1451225c026a90ab52eed711 Author: Damian Johnson Date: Sat Oct 8 23:23:08 2011 -0700 Adding logging utility Borrowing the logger from arm, with some refactoring to remove unneeded functions and make it conform with this project's coding conventions. commit 88921da126612d90a2dbf002f37ba236da047cd4 Author: Damian Johnson Date: Sat Oct 8 16:15:48 2011 -0700 Adding gitignore for pyc and vim swap files commit 3dd9ea0d9222ddf22e63c7f5dcf8e7032e6fb494 Author: Damian Johnson Date: Sat Oct 8 16:04:08 2011 -0700 Running tests based on input arguments Making the test runner accept arguments for the type of tests to be ran. The integration tests especially will take a while when they're implemented so letting the user specify the use cases for those. This included copying and refactoring some basic utilities from arm for enumerations and terminal text attributes. commit 39e10eed903c1920a50a175803726035b8448712 Author: Damian Johnson Date: Sat Oct 8 14:03:08 2011 -0700 Using Pascal Case styling (unscores) for variables According to PEP8 [1] both functions and variable names should use the underscore naming convention (ie, 'my_var') rather than camel case ('myVar'). This is a little weird for me and python standard libraries use both, but I see a readability advantage to this for functions and my previous approach of using camel case just for variables is kinda weird. Hence switching to the 'right' convention while the codebase is still tiny. As an added bonus this conforms with torctl... [1] http://www.python.org/dev/peps/pep-0008/ commit d38b8830804e78cd0921333c39c5e9a2ee8ad42f Author: Damian Johnson Date: Sat Oct 8 13:10:38 2011 -0700 Subdirectories for integ and unit tests Unit and integration tests will be separate subdirectories of test. commit e1c31c060c0246a62e911643a383372889214300 Author: Damian Johnson Date: Thu Oct 6 09:57:24 2011 -0700 Removing checked in binaries Forgot to establish a gitignore before the prior commit. An extra copy of run_tests.py also snuck in (haven't a clue how). commit 69083da95c5ab536adcd23f19a424ed9395015bb Author: Damian Johnson Date: Thu Oct 6 09:52:04 2011 -0700 Class and function for Tor Versions Starting with a simple (but non-trivial) class that will be needed for handling PROTOCOLINFO responses. This is partly to establish conventions for documentation and unit tests. commit 211c52e6f24440909bf243e0b8c11e3158c7fcee Author: Damian Johnson Date: Thu Oct 6 09:33:34 2011 -0700 Initial commit Simply filling in an empty file to initialize a root for the repo. debian/gbp.conf0000644000000000000000000000003612231246432010603 0ustar [DEFAULT] pristine-tar = True