foolscap-0.13.1/0000755000076500000240000000000013204747603014031 5ustar warnerstaff00000000000000foolscap-0.13.1/.coveragerc0000644000076500000240000000124012766553111016150 0ustar warnerstaff00000000000000[run] # only record trace data for foolscap.* source = foolscap # and don't trace the test files themselves, or Versioneer's stuff omit = src/foolscap/test/* src/foolscap/_version.py # This allows 'coverage combine' to correlate the tracing data built while # running tests in multiple tox virtualenvs. To take advantage of this # properly, use "coverage erase" before tox, "coverage run --parallel-mode" # inside tox to avoid overwriting the output data (by writing it into # .coverage-XYZ instead of just .coverage), and run "coverage combine" # afterwards. [paths] source = src/ .tox/*/lib/python*/site-packages/ .tox/pypy*/site-packages/ foolscap-0.13.1/ChangeLog.0.6.40000644000076500000240000064202512410141417016245 0ustar warnerstaff000000000000002012-06-18 Brian Warner * foolscap/_version.py: release Foolscap-0.6.4 * NEWS: update for 0.6.4 release 2012-06-18 Brian Warner * NEWS: update with recent changes * foolscap/logging/filter.py: fix a missing mode=b when opening the output of "flogtool filter", which could result in corrupted flogfiles on windows. Should close #179. * setup.py (extras_require): remove "secure_connections" dep on pyOpenSSL, closes #174. Patch by Zooko, thanks! * README.packagers: explain what OS packages and python-package authors need to depend upon to get secure connections * foolscap/logging/log.py (bridgeLogsFromTwisted): ignore multiple calls with identical twisted_logger/foolscap_logger pairs, to avoid recording duplicate twisted log messages in flogfiles. Closes #194. * foolscap/negotiate.py (Negotiation.negotiationFailed): don't log tracebacks for normal negotiation failures, since they aren't helpful. Closes #195. * doc/examples/git-foolscap (create): add the git reponame to the generated FURL (e.g. pb://TUBID@HINT/SWISS/REPONAME) so that 'git clone' will give the checkout a meaningful name. Technically, the 'swissnum' is thus SWISS/REPONAME, but foolscap doesn't care about slashes inside swissnums. Closes #197. * foolscap/appserver/cli.py (Stop): add --quiet to hush the "this does not look like a running node" message (Restart.run): use --quiet * doc/examples/git-foolscap: make it importable, add run(). Fix help text. 2012-06-15 Brian Warner * NEWS: update with all recent changes * foolscap/pb.py (Tub.stopService): make copies of any lists that we might mutate before iterating over them, in particular self.reconnectors. Otherwise we can fail to shut everything down properly. Closes #196. 2012-06-09 Brian Warner * foolscap/logging/gatherer.py (IncidentObserver): keep fetching incidents even when some fail, to skip ones that can't be serialized due to buggy code calling log.msg() with non-data. Patch by David-Sarah Hopwood. Closes #190. * foolscap/logging/web.py: make 'flogtool web-viewer --timestamps' use format_time() too (adds --epoch) * foolscap/test/test_logging.py (Web.test_basic): update to match * foolscap/util.py (format_time): add --epoch, closes #192 * foolscap/util.py (format_time): factor out timestamp formatting from logging/dumper.py * foolscap/logging/dumper.py (LogDumper): same * foolscap/test/test_logging.py (Dumper.test_dump): update 2012-05-31 Zancas * foolscap/logging/dumper.py: improve docs, closes #191 * foolscap/logging/web.py: same 2012-03-31 Zancas * doc/using-foolscap.xhtml: minor edits 2012-03-29 Brian Warner * foolscap/referenceable.py (decode_location_hints): update connection-hint extension rules to require extensions start with "TYPE:" and not be followed by all-digits. Reject "HOST" without a portnum to catch errors like setLocation("HOST") earlier (it should be setLocation("HOST:PORT")). Allow "pb://TUBID@" as legal but unrouteable (still valid for inbound connections). (NONAUTH_STURDYREF_RE): fix no-crypto tests by allowing "pbu:///SWISS" as legal but unrouteable. It's completely useless, though, because unauthenticated FURLs can't accept inbound connections. * foolscap/test/test_sturdyref.py (URL.testBrokenHints): enhance * foolscap/test/test_tub.py (BadLocationFURL): update * foolscap/logging/dumper.py (LogDumper.run): when 'dump' is pointed at a FURL file instead of an event log file, give a helpful error message instead of the confusing pickle error * foolscap/test/test_logging.py (Dumper.test_oops_furl): tests * foolscap/logging/cli.py: let commands control the sys.exit() return code, to prepare for tests of improved error handling * foolscap/test/test_logging.py (run_wrapper): update to match 2012-01-05 Brian Warner * foolscap/_version.py: bump version number while between releases 2012-01-05 Brian Warner * foolscap/_version.py: release Foolscap-0.6.3 * NEWS: update for 0.6.3 release 2012-01-05 Brian Warner * README (DEPENDENCIES): give up on twisted-2.4.0 compatibility, the tests fail now * NEWS: update 2012-01-04 Brian Warner * foolscap/call.py (CallUnslicer.receiveChild): raise a Violation when the CLID is stale, so the caller gets an errback. Previously, the otherwise-unhandled exception got put in the receiver's log, but the caller didn't get a clear notification. Closes #106. * foolscap/broker.py (Broker.remote_decref): tolerate "late" decrefs that happen after their target has gone away, which causes a mostly-harmless error to be logged, typically at the end of unit test cases when Tub.stopService is called on multiple Tubs in the same process that are talking to each other. refs #106. * foolscap/broker.py: oops, un-break no-SSL support * foolscap: Twisted-11.1 has new TLS code which reports connectionLost() significantly after we call loseConnection(), such as when we drop a stale connection in favor of a replacement. This breaks a number of assumptions, generally by allowing Tub.stopService() to finish too early, so tests for users of foolscap (like tahoe) fail with DirtyReactorError, or port-already-in-use errors. These changes fix the problem, and update the Foolscap unit tests to handle the changes. * foolscap/test/test_negotiate.py: tolerate RemoteNegotiationError error too, since now it can appear earlier and beat the local NegotiationError. * foolscap/pb.py (Tub): track connections separately from Brokers. Broker.shutdown() fires brokerDetached (which severs all rrefs and stops using that connection). The subsequent connectionLost() fires connectionDropped(), which allows Tub.stopService()'s Deferred to fire. This should fix the tests-end-too-early problem. * foolscap/negotiate.py (Negotiation.switchToBanana): call tub.connectionAttached() to track connections * foolscap/broker.py (Broker.connectionLost): and tub.connectionDropped() * foolscap/broker.py: the new TLS code sometimes reports abandoned connections with SSL.Error, so when we're ignoring ConnectionLost and ConnectionDone, ignore SSL.Error too. * foolscap/test/test_negotiate.py: wait more turns for each step, now that connection-loss events are not synchronous * foolscap/test/common.py (ShouldFailMixin.shouldFail): improve error message 2011-10-16 Brian Warner * misc/*/debian: remove debian packaging tools. They were pretty stale anyways. * MANIFEST.in: same * Makefile (debian-*): same * MANIFEST.in: remove entries for figleaf code and misc/testutils to hush some build-time warnings. I should have done before releasing 0.6.2, but the warnings are benign. * foolscap/_version.py: bump version number while between releases 2011-10-15 Brian Warner * foolscap/_version.py: release Foolscap-0.6.2 * misc/*/debian/changelog: same * NEWS: update for 0.6.2 release 2011-10-08 Brian Warner * foolscap/test/test_tub.py (TestCertFile.test_generate): tolerate certs generated by more recent versions of OpenSSL (which say "BEGIN PRIVATE KEY" instead of "BEGIN RSA PRIVATE KEY"). Thanks to jtaylor, zooko, and davidsarah for the catch. Closes #184. * Makefile: remove figleaf code. It was out-of-date anyways, coverage.py is a better modern choice. Tahoe has some sample code, including emacs integration. * misc/testutils: same, removed the whole thing 2011-10-07 Zooko O'Whielacronx * foolscap/pb.py (Tub.setLogGathererFURL): accept an iterable of log gatherer furls in addition to a single log gatherer furl. Closes #176 * test/test_logging.py (Gatherer): test for same * doc/logging.xhtml: update docs 2011-10-05 Brian Warner * foolscap/appserver/client.py (UploadFile.run): fix lambda-capturing-loop-variables bug spotted by David-Sarah Hopwood's automated tool. The fault would appear when trying to upload three or more files in the same call: only the first and last would actually happen (the last file would be uploaded multiple times). * foolscap/test/test_appserver.py (Upload.test_run): test it * NEWS: mention the bug 2011-10-04 Brian Warner * NEWS: update for upcoming release 2011-06-09 Brian Warner * foolscap/test/test_appserver.py (RunCommand.test_run): use a helper script, instead of /bin/cat and /bin/dd, so this test can run on windows too. Closes #178. * foolscap/test/apphelper.py: same * foolscap/test/test_reconnector.py: hush pyflakes 2011-06-05 Brian Warner * foolscap/negotiate.py (Negotiation.switchToBanana): rather than replacing self.transport.protocol with the Banana protocol, leave the Negotiation protocol in place, but swap out its .dataReceived and .connectionLost methods with pointers to the Banana's methods. This avoids a bug with current Twisted trunk (>11.0.0) in which the new TLS implementation (enabled if you have pyOpenSSL>0.10) doesn't appreciate having its .protocol changed on it after initialization. Closes #173. * foolscap/banana.py (Banana.handleError): stop passing a 'reason' argument into loseConnection(), not all implementations accept it. * foolscap/test/test_reconnector.py (Reconnector._got_ref): same * foolscap/logging/incident.py (TIME_FORMAT): change to use hyphens on all platforms, despite colons being easier to read and valid on everything but windows. Filenames now look like incident-2011-06-05--14-40-21Z-w2qn32q.flog . Closes #177 again. * doc/logging.xhtml: update to match * foolscap/_version.py: change "in-between releases" version from "0.6.1+" to "0.6.1.post0", to satisfy PEP 386 (and make it possible for Tahoe to use development versions of Foolscap). I am slightly annoyed by the need to use a less-readable versioning scheme, but a number of people (in particular Tarek and the Pycon community) seem to support PEP 386, so I'll play along. Closes #175. * misc/*/debian/changelog: same * .gitignore: ignore generated files 2011-05-09 Brian Warner * foolscap/appserver/client.py (run_flappclient.oops): write error message to stderr, not stdout, else git-remote-pb (which expects protocol messages on stdout) gets confused * foolscap/appserver/client.py (StandardIO): improve TwitchyStandardIO workaround: use f.check(), not f.value.__class__ * foolscap/logging/log.py (FoolscapLogger.msg): return event number even if generation_threshold says to not record event 2011-05-08 Brian Warner * foolscap/logging/filter.py (Filter.run): when doing modify-in-place on windows, don't attempt an atomic rename of the output file. Closes #180. * foolscap/logging/incident.py (TIME_FORMAT): remove colons from incident filenames when running on windows, since it can't handle them. Thanks to David-Sarah Hopwood for the catch. Closes #177. * foolscap/logging/gatherer.py: same (just import TIME_FORMAT from incident.py) 2011-02-07 Brian Warner * foolscap/appserver/client.py: work around an apparent twisted StandardIO bug when git closes our stdout and then writes to our stdin. * doc/examples/git-foolscap, git-remote-pb: new git-over-foolscap tools, uses 'git-remote-helpers' for easy setup. * foolscap/appserver/cli.py: split make_swissnum() out 2011-02-06 Brian Warner * .darcs-boringfile, .hgignore, .hgtags: remove old files now that Foolscap lives in git * foolscap/appserver/cli.py: split list_service/add_service out into separate functions, for use by other code * foolscap/test/test_appserver.py (CLI): test them * foolscap/appserver/cli.py (BaseOptions): fix --help text for subcommands: override getSynopsis() to prevent usage.Options from simply concatenating root synopsis with subcommand synopsis. Add -h as an alias for --help. Also improve some of the docs * foolscap/appserver/services.py (FileUploaderOptions): same * foolscap/_version.py: bump version while between releases * misc/*/debian/changelog: same 2011-01-16 Brian Warner * foolscap/_version.py: release Foolscap-0.6.1 * misc/*/debian/changelog: same * NEWS: update NEWS for next release * foolscap/eventual.py (_SimpleCallQueue._turn): remove unreachable code. Thanks to 'ivank' for the catch. Closes #165. * foolscap/referenceable.py (RemoteReferenceOnly.getDataLastReceivedAt): new method to let application code know whether the connection appears alive: tells you when we last heard anything from the other end on this connection. Only enabled if keepalives are turned on, since otherwise we don't want to spend the overhead of calling time.time() upon every inbound packet. (if keepalives are disabled, which is the default, you get None). Closes #169. * foolscap/banana.py (Banana.getDataLastReceivedAt): same * foolscap/test/test_keepalive.py (Keepalives): test enabled case * foolscap/test/test_pb.py (TestReferenceable): test disabled case * foolscap/banana.py (Banana.initSend): stop importing the deprecated 'sets' module, to hush warnings when used with newer pythons, since we don't really need it here. We still import it from foolscap/slicers/set.py to manage compatibility with older code, but that's wrapped with a warning-suppressor. This is mainly to help fix Tahoe's #1329. 2010-12-28 Brian Warner * foolscap/_version.py: release Foolscap-0.6.0 * misc/*/debian/changelog: same * NEWS: update for upcoming release 2010-12-28 Brian Warner * foolscap/sslverify.py: hush pyflakes warnings * foolscap/referenceable.py (RemoteReference.callRemoteOnly): same 2010-12-06 Brian Warner * NEWS: start collecting items for the next release * foolscap/test/test_banana.py (InboundByteStream.testLong): add more tests. Thanks to Zarutian for the suggestions. 2010-12-05 Brian Warner * misc/*/debian/rules: include all *.xhtml files in the debian packages. Closes #131. * foolscap/referenceable.py (RemoteReferenceTracker._handleRefLost): guard against self.ref==None, in case that's the problem in #147. It doesn't seem likely, though. * foolscap/__init__.py: empty it, except for __version__. This finishes the transition to foolscap.api, and closes #122 . * foolscap/deprecated.py: deleted 2010-11-29 Brian Warner * foolscap/pb.py (Listener): fix Twisted-10.2 compatiblity: stop using strports.service() . Fixes #167. * foolscap/logging/web.py (WebViewer): same 2010-08-09 Brian Warner * foolscap/logging/log.py (FLOGTOTWISTED): honor FLOGLEVEL= when using FLOGTOTWISTED=1, just like FLOGFILE= does. Closes #154. * foolscap/logging/incident.py (IncidentReporter.TIME_FORMAT): change incident filenames to have timestamps in UTC (not localtime) and in a format almost like ISO-8601. They are now 2008-08-22--16:20:28Z . The real ISO-8601 would use "_" instead of "--", but I find the underscore to visually "bind" more tightly than the hyphen, making it hard to tell whether the 16 is part of the date side or the time side. This should close #111. * foolscap/logging/gatherer.py (GathererService): same for log-gatherer filenames. Also changed the from-vs-to separator to three hyphens now that we're using two hyphens in the timestamps. Filenames now look like this: from-2010-08-09--16:13:19Z---to-2010-08-09--22:13:19Z.flog.bz2 2010-08-08 Brian Warner * setup.py (setup_args packages=): use proper "foolscap.slicers" instead of incorrect "foolscap/slicers". This apparently caused problems with some external setuptools stuff. Closes #156. * foolscap/appserver/services.py (FileUploader.remote_putfile._err): fix error-handling code which would fail to unlink the tempfile upon client read() error. Fixes #157. * foolscap/test/test__versions.py (Versions.test_required): oops, tolerate Twisted versions like 10.1 (numeric compare instead of alphabetical), rather than thinking that 10.1 was older than the problematic 8.1.0 . * foolscap/_version.py: bump version while between releases * misc/*/debian/changelog: same 2010-03-25 Brian Warner * foolscap/_version.py: release Foolscap-0.5.1 * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: same 2010-03-25 Brian Warner * foolscap/banana.py, broker.py, copyable.py, pb.py * foolscap/appserver/cli.py, foolscap/logging/gatherer.py * foolscap/test/test_call.py, test_logging.py, test_observer.py, * test_promise.py, test_reference.py, test_registration.py, * test_schema.py, test_tub.py: clean up lots of pyflakes warnings, revealed by the new more-strict version of pyflakes * foolscap/banana.py (Banana.dataReceived): apply zooko's patch to use stringchain on the inbound data path, to fix the O(n^2) performance in large tokens (e.g. a 10MB string token). Thanks Zooko! Closes #149. * foolscap/test/bench_banana.py: also zooko's performance tests * foolscap/stringchain.py: Zooko's utility class to efficiently handle large strings split into several pieces, such as the inbound socket buffers that Banana.dataReceived() winds up with. * foolscap/test/test_stringchain.py: unit tests for the same. Zooko has given me permission to distribute both of these under Foolscap's MIT license. 2010-03-14 Brian Warner * foolscap/constraint.py: remove maxSize/maxDepth methods, and the related UnboundedSchema exception. As described in ticket #127, I'm giving up on resource-exhaustion defenses, which allows for a lot of code simplification. * foolscap/{copyable.py|removeinterface.py|schema.py}: same * foolscap/slicers/*.py: same * foolscap/test/test_schema.py: remove tests * doc/jobs.txt: remove TODO items around maxSize * foolscap/_version.py: bump version while between releases * misc/*/debian/changelog: same 2010-01-18 Brian Warner * foolscap/_version.py: release Foolscap-0.5.0 * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: same 2010-01-18 Brian Warner * NEWS: update for upcoming release 2010-01-11 Brian Warner * foolscap/logging/web.py (WebViewer.start): add --open, which uses stdlib's webbrowser.open() function to automatically open a browser window with the "flogtool web-viewer" contents. * foolscap/pb.py (Tub.getReferenceForName): for unknown names, only include the first two letters of the swissnum in the KeyError we return, to avoid revealing the whole thing in the caller's error logs. Closes #133. * foolscap/test/test_pb.py (TestCallable.testWrongSwiss): test it * foolscap/test/test_gifts.py (Bad.test_swissnum): same (Bad.testReturn_swissnum): same 2009-12-28 Brian Warner * doc/flappserver.xhtml: general improvements * foolscap/appserver/cli.py (Create.run): chmod the basedir go-rwx to keep the private key hidden from other users * foolscap/test/test_appserver.py (CLI.test_create): test it * doc/flappserver.xhtml: mention the chmod * foolscap/appserver/cli.py (Start.run): use the handy run() function from twisted.scripts.twistd instead of os.execvp to give control to twistd. This removes a lot of grossness and should remove the need to modify $PATH when running "bin/flappserver start" from an uninstalled source tree. * foolscap/logging/dumper.py (LogDumper.run): don't eat IOErrors when you point it at a missing file. Only eat the EPIPE that occurs when you run "flogtool dump FILE |less" and quit the pager early. * foolscap/logging/dumper.py: add --timestamps, which can be "short-local", "long-local" (ISO-8601 with timezone), or "utc" (ISO-8601 with "Z" suffix). Closes #100. * foolscap/logging/web.py: add --timestamps, "local" or "utc". This controls the default format. Added tooltips over the time strings to show the event time in local, iso-8601, and iso-8601-utc. Added links to the all-events page to reload it with various timestamp formats and sort options. * foolscap/test/test_logging.py (Dumper): update tests to match (Web): same 2009-10-04 Brian Warner * foolscap/slicers/set.py: suppress the DeprecationWarning that occurs when you import 'sets' on py2.6: seems like the best way to preserve compatibility with py2.3-compatible application code while still avoiding the noise when running with 2.6. Closes #124. * foolscap/appserver/client.py (Uploader.run): expanduser() the filename, so you can use 'flappclient upload-file ~/foo.txt' even without the shell expanding the filename for you (ClientOptions.read_furlfile): same, closes #134. 2009-09-17 Brian Warner * setup.py: use entry_points= on windows, to create executable .bat files for flogtool/flappserver/flappclient . Stick to scripts= on non-windows (and remember to install with --single-version-externally-managed) to get non-opaque flogtool scripts that don't use magic pkg_resources functions to import foolscap. This should close #109. 2009-07-30 Brian Warner * foolscap/pb.py (Tub.setLocationAutomatically): fix comment * foolscap/test/test_serialize.py: oops, fix pyflakes warnings * foolscap/test/test_reference.py: same * foolscap/test/test_gifts.py: same 2009-07-28 Brian Warner * foolscap/test/*: stop using old-style shouldFail (which would mask errors) 2009-06-24 Brian Warner * doc/listings/xfer-client.py, xfer-server.py, command-client.py, command-server.py: remove, obsoleted by flappserver * foolscap/pb.py (getRemoteURL_TCP): remove this function, I never liked it, and it wasn't very good anyway. * foolscap/api.py, foolscap/__init__/py, foolscap/deprecated.py: same * foolscap/test/test_pb.py: remove dead code * foolscap/deprecated.py: define wrappers that emit DeprecationWarning upon use, then wrap them around everything that used to be in foolscap/__init__.py . * foolscap/__init__.py: apply the wrappers. Please import from foolscap.api instead. See #122 for details. * foolscap/*: stop importing functions/classes directly from the top-level 'foolscap' module, instead import them from the 'foolscap.api' module (i.e. 'from foolscap.api import Tub' instead of 'from foolscap import Tub'). I'm planning to add deprecation wrappers around the things you can get from 'foolscap', and this is to make sure our own code doesn't run afoul of those warnings. 2009-06-22 Brian Warner * foolscap/ipb.py (DeadReferenceError): improve the string form by adding remote tubid, and information from the PendingRequest. Since DeadReferenceError frequently happens in an eventually() context, there's not usually a useful stack trace, so any piece of information will help. * foolscap/referenceable.py (RemoteReference._callRemote): give interfacename+methodname to PendingRequest, so it can give that information to DeadReferenceError.__str__ . Some day it'd be cool to include a stack trace here. * foolscap/call.py (PendingRequest.__init__): store it (PendingRequest.getMethodNameInfo): return it * foolscap/broker.py (Broker.abandonAllRequests): pass PendingRequest to DeadReferenceError * foolscap/test/test_call.py (TestCall.test_connection_lost_is_deadref): test it * foolscap/test/test_pb.py: update to match, remove dead code * doc/examples/git-remote-add-furl: fix shbang line * doc/examples/git-clone-furl: same 2009-06-20 Brian Warner * doc/examples/git-clone-furl: easier tool to set up FURL-based git repo * doc/examples/git-remote-add-furl: same * doc/examples/git-publish-with-furl: better name for git-setup-flappserver-repo * doc/examples/git-proxy-flappclient: mention those scripts * foolscap/appserver/client.py: improve --help strings * foolscap/appserver/cli.py (CreateOptions.opt_umask): capture the current umask at "flappserver create" time for use by the eventual server. This should reduce surprises. * doc/flappserver.xhtml: explain the new behavior * foolscap/appserver/server.py (AppServer): require the umask file to contain an octal string, don't use eval() * foolscap/test/test_appserver.py: update tests 2009-06-19 Brian Warner * doc/flappserver.xhtml: add --umask= to "flappserver create", since twisted forces it to 077 on startup, and the git-proxy-flappclient system was winding up with overly restrictive filemodes * foolscap/appserver/cli.py (CreateOptions): same * foolscap/appserver/server.py (AppServer): same * foolscap/test/test_appserver.py (CLI.test_create2): test it * doc/examples/git-setup-flappserver-repo: suggest --umask 2009-06-18 Brian Warner * doc/examples/git-proxy-flappclient: * doc/examples/git-setup-flappserver-repo: new pair of tools to run the Git remote-access protocol over a flappserver-based connection * MANIFEST.in: include them in the source tarball * foolscap/_version.py: bump version between releases * misc/*/debian/changelog: same 2009-06-16 Brian Warner * foolscap/_version.py: release Foolscap-0.4.2 * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: same 2009-06-16 Brian Warner * foolscap/test/test_gifts.py: remove all timeouts.. I think one buildslave took slightly too long, which flunked the test, and then screwed up every test after that point. * setup.py (packages): add foolscap/appserver * NEWS: update for upcoming release * setup.py (setup_args): include flappclient as a script * foolscap/negotiate.py (TubConnector.connect): if we have no location hints, set self.failureReason so we can error out cleanly. Closes #129. * foolscap/tokens.py (NoLocationHintsError): new exception for this purpose * foolscap/negotiate.py (TubConnector.failed): call tub.connectorFinished() too, so any pending Tub.stopService can be retired. * foolscap/pb.py (Tub.getReference): wrap in a maybeDeferred, so that unparseable FURLs give us errbacks instead of synchronous exceptions. This is the other half of #129. (Tub.connectorFinished): stop complaining (at log.WEIRD) if the connector wasn't in our table. I don't know what was causing this, and I don't really care. Closes #81. * foolscap/test/test_tub.py (BadLocationFURL): test #129 stuff * foolscap/pb.py (Tub.getReference): always return a Deferred, even if given an unparseable FURL. Addresses #129. * foolscap/test/test_tub.py (NoLocationFURL): test it * foolscap/appserver/client.py (UploadFile): accept multiple files * foolscap/test/test_appserver.py (Upload.test_run): test it * doc/flappserver.xhtml: update example to match * doc/flappserver.xhtml: rename tools: use "upload-file" and "run-command" instead of file-uploader and exe * doc/foolscap/appserver/client.py: same * doc/foolscap/appserver/services.py: same * foolscap/test/test_appserver.py: same * foolscap/test/test_appserver.py (RunCommand): improve coverage * foolscap/appserver/cli.py: move flappclient code out of cli.py .. * foolscap/appserver/client.py (run_flappclient): .. into client.py (Exec.dataReceived): don't spin up StandardIO until after the server tells us they want stdin. This also means we can stop buffering stdin. * foolscap/test/test_appserver.py: match client.run_flappclient change (RunCommand.test_run): first test of run-command/exec code * foolscap/broker.py (Broker.doNextCall): don't run any calls after we've been shut down. This fixes a racy bug in which two calls arrive and get queued in the same input hunk, the first one provokes a Tub.stopService, and the second one gets run after the Tub has been shutdown, leaving an "Unhandled Error in Deferred" lying around. This was occasionally flunking an appserver test. * foolscap/test/test_tub.py (CancelPendingDeliveries): test it * foolscap/appserver/cli.py (Restart): add 'flappserver restart' * foolscap/appserver/services.py (Exec): make run-command work, including optional stdin/stdout/stderr, logging of each, and return of exitcode. Still needs unit tests and a better name. * foolscap/appserver/client.py (Exec): same (parse_options): pass a 'stdio' object in separately, to let unit tests control stdin 2009-06-15 Brian Warner * foolscap/appserver/services.py (FileUploader.remote_putfile): use os.chmod, since Twisted-2.5 doesn't have FilePath.chmod * bin/flappclient: generic client for flappserver * foolscap/appserver/client.py: same * foolscap/test/test_appserver.py (Upload.test_run): test file uploading (Client.test_help): test --help and --version * foolscap/appserver/*: fixes to make file-upload work * foolscap/appserver/services.py (FileUploader): implement it for real, using twisted's FilePath. Not sure it actually works yet. * foolscap/test/test_appserver.py: begin unit tests for appserver * foolscap/appserver/*: clean up Create, add AppServer.when_ready() to handle setLocationAutomatically() better, display simple UsageError/BadServiceArguments exceptions better (no traceback) 2009-06-14 Brian Warner * foolscap/appserver/server.py (AppServer.__init__): improve startup message * foolscap/appserver/services.py (FileUploader.__init__): note that tub= might be None (such as in 'add' when we're merely validating the service arguments) * foolscap/appserver/cli.py: run Tub for 'create', but not for 'add' or 'list'. Stash the furl-prefix to support this. The idea will be to rewrite furl-prefix each time the server is started. This also allows 'add' and 'list' to be used while the server is already running. * doc/flappserver.xhtml: add example transcript * foolscap/pb.py (generateSwissnumber): pull this out to a seperate function, to be called externally (Tub.generateSwissnumber): same * doc/flappserver.xhtml: start to create an "application server": an easy tool to deploy pre-written foolscap-based services in a shared server process. This first step creates the bin/flappserver tool, and implements the "create", "add", "list", "start", and "stop" commands. The next step will be to write some basic tests, then implement some built-in services (file-upload and exec-command), then write some clients to drive those services. Eventually new services will be loaded with a plugin architecture. Tracked in ticket #82. * bin/flappserver: this tool is used to manipulate app servers * foolscap/appserver/*.py: the actual code * setup.py: bin/flappserver is a script too 2009-06-05 Brian Warner * foolscap/referenceable.py (RemoteReferenceOnly.getLocationHints): new method (RemoteReferenceOnly.getSturdyRef): remove "this is not secure" comment: getSturdyRef now *does* check the tubid against the connection (SturdyRef): factor out encode_furl/decode_furl into new functions * foolscap/test/test_pb.py (TestReferenceable): test both getLocationHints() and isConnected() (TestCallable.testGetSturdyRef): same * foolscap/test/test_gifts.py: update to use encode_furl 2009-06-03 Brian Warner * foolscap/referenceable.py (RemoteReferenceOnly.isConnected): new method, tests the same thing as notifyOnDisconnect but this one is immediate instead of callback-based * foolscap/test/test_call.py (TestCall.testNotifyOnDisconnect): test it * foolscap/test/test_call.py (ReferenceCounting.test_reference_counting): oops, turn off debugging noise which was accidentally left in back at 0.4.0 * foolscap/referenceable.py (TheirReferenceUnslicer): remove the 200-byte limit on FURLs which are passed as third-party "gifts". (ReferenceUnslicer): same, for inbound my-references * foolscap/_version.py: bump version between releases * misc/*/debian/changelog: same 2009-05-22 Brian Warner * foolscap/_version.py: release Foolscap-0.4.1 * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: same 2009-05-22 Brian Warner * NEWS: udpate for upcoming release * foolscap/tokens.py (RemoteException): override __str__, since we don't put arguments in .args . Otherwise, attempting to stringify a RemoteException will fail under python2.4 (but not 2.5). * foolscap/test/test_logging.py (Basic.testFailure): test it * foolscap/test/test_call.py (ExamineFailuresMixin._examine_raise): more thorough test * foolscap/_version.py: bump version between releases 2009-05-19 Brian Warner * foolscap/_version.py: release Foolscap-0.4.0 * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: same 2009-05-19 Brian Warner * NEWS: update for the next release * README: mention python2.6 compatibility * foolscap/test/test__versions.py (Versions.test_record): record versions of platform/python/twisted/foolscap in the test logs * foolscap/api.py: add some more symbols: ChoiceOf, IntegerConstraint * foolscap/banana.py (Banana.handleData): increment the inbound objectCounter as soon as we see the OPEN token, rather than doing it in handleOpen(), since the latter is skipped if we're dropping tokens due to a Violation. This avoids the loss-of-sync that occurred when the dropped tokens included more OPEN sequences (e.g. when a remote method call used a bad method name, and its arguments contained lists or tuples or other sequences). This fixes the bug in which subsequent method calls that used shared arguments saw their "reference" sequences point to the wrong value. Closes #104. * foolscap/test/test_call.py (ReferenceCounting): test it * foolscap/referenceable.py (RemoteReferenceTracker.__init__): require the proposed per-my-reference FURL to match the TubID that we're using for the parent Broker. This prevents the far end from sending a bogus FURL and thus breaking the security properties of rref.getSturdyRef(). (they couldn't confuse Tub.getReference, because the spoofed FURL would only be added to the per-Broker table, but they could confuse someone who relied upon the correctness of getSturdyRef). This closes #84. (RemoteReferenceTracker.__init__): oops, don't compare tubids when using an UnauthenticatedTub, since that fails the non-crypto tests * foolscap/test/test_reference.py (TubID): test it * foolscap/test/test_schema.py (Interfaces.test_remotereference): update to match * foolscap/hashutil.py: new file to smoothly use hashlib when available and fall back to sha1/md5 when not (i.e. python2.4). When we eventually drop py2.4 support, we'll get rid of this file. Thanks to Stephan Peijnik for the patch. Closes #118. * foolscap/sslverify.py: use hashutil instead of the md5 module * foolscap/vocab.py (hashVocabTable): use hashutil instead of the sha module * foolscap/schema.py (PolyConstraint.checkToken): remove unused code, thanks to Stephan Peijnik for the patch. * foolscap/test/test_banana.py (VocabTest1.test_table_hashes): make sure we don't change the pre-agreed vocab tables, or the hash algorithm that we use to confirm their contents. This is in preparation for the conditional switch to 'hashlib', from Stephan's patch in ticket #118 2009-05-18 Stephan Peijnik * foolscap/call.py, foolscap/constraint.py, foolscap/eventual.py, foolscap/observer.py, foolscap/promise.py, foolscap/reconnector.py, foolscap/remoteinterface.py, foolscap/slicer.py: Convert all old-style classes to new-style classes. Should close #96. * foolscap/reconnector.py, foolscap/referenceable.py, foolscap/remoteinterface.py: Import each module with a separate import statement. 2009-05-18 Brian Warner * foolscap/test/test_tub.py (TestCertFile.test_tubid): confirm that we get the TubID right when using a pre-recorded cert. If we were to accidentally change the TubID-computing hash function in the future, this test should catch it. (CERT_DATA): stash the contents of a pre-recorded cert * bin/flogtool: let 'base' default to abs("."), to help windows and systems where somebody has managed to copy it to /tmp/flogtool or /sbin/flogtool . Closes #108. 2009-05-13 Brian Warner * foolscap/test/test_call.py (ExamineFailuresMixin): rearrange code to try and fix a weird py2.4 failure * foolscap/api.py (RemoteException): hush pyflakes 2009-05-12 Brian Warner * foolscap/logging/dumper.py: emit PID and embedded versions, except when using --just-numbers or --verbose. Closes #97. * foolscap/logging/web.py (Welcome.render): same * foolscap/logging/log.py (LogFileObserver): store app_versions in the log file created by $FLOGFILE * foolscap/test/test_logging.py: test it all * foolscap/tokens.py (RemoteException): add a Tub option that causes remote exceptions to be reported with a special exception type, named foolscap.api.RemoteException, so that they won't be confused with locally-generated exceptions. Closes #105. * foolscap/api.py: make RemoteException available through api.py * doc/failures.xhtml: explain the new option * foolscap/pb.py (Tub.setOption): handle the new expose-remote-exception-types option, default remains True to retain the old behavior * foolscap/broker.py (Broker.setTub): copy the flag from the Tub into the Broker * foolscap/call.py (ErrorUnslicer.receiveClose): wrap exceptions if the mode says we should * foolscap/test/test_call.py (Failures): test RemoteException wrapping (TubFailures): test setOption() too * foolscap/test/common.py (ShouldFailMixin.shouldFail.done): return the Failure for further testing, but wrapped in an array to avoid triggering the errback 2009-05-11 Brian Warner * Makefile (LORE): parameterize "lore" binary * doc/stylesheet.css: update to new version from Lore * doc/schema.xhtml: fix mis-nested H3 header, to appease new Lore 2009-05-04 Brian Warner * foolscap/api.py: new preferred entry point. Application code should import symbols from "foolscap.api" instead of directly from foolscap/__init__.py . Importing from the __init__.py will be deprecated in the next major release, and removed in the subsequent one. Addresses #122 (but it won't be closed until we remove everything from __init__.py). * foolscap/__init__.py: add comment about preferring foolscap.api * doc/using-foolscap.xhtml: update examples to use e.g. "from foolscap.api import Tub" * doc/listings/*.py: update examples * foolscap/pb.py (Tub.setLogGathererFURL): allow users to call both Tub.setOption("log-gatherer-furl") and Tub.setOption("log-gatherer-furlfile") on the same Tub. Previously this was disallowed. Also avoid making multiple connections to the same gatherer. Closes #114. * foolscap/test/test_logging.py (Gatherer.test_log_gatherer_furlfile_multiple): test it * foolscap/test/test_logging.py (Publish.test_logpublisher_catchup): don't use an absolute delay, since it causes spurious test failures on slow systems. * foolscap/logging/web.py (WebViewer.start): put the welcome page at the root URL, in addition to /welcome . This simplifies the starting URL. Closes #120. (Reload.render_POST): make the "Reload Logfile" button point at the new URL too. * foolscap/logging/web.py (Welcome.render): display PID of logfile, if available. This will be present in incident records that were written out by the original process or gathered by a log gatherer, in the files written by "flogtool tail --save-to=", and in the file written by creating an explicit FileLogObserver (such as done by setting the FLOGFILE environment variable). This should close #80. * foolscap/logging/log.py (LogFileObserver): record PID in header * foolscap/test/test_logging.py (Web.test_basic): test PID * foolscap/test/test_banana.py (ThereAndBackAgain.test_decimal): use Decimal("Inf") instead of decimal.Inf, to unbreak python2.6.2 which privatized both Inf and Nan. Thanks to ivank for the patch. Closes #121. 2008-10-15 Brian Warner * misc/*/debian/rules (install/python-foolscap): include misc/classify_foolscap.py in the debian package, uncompressed, so it goes into /usr/share/doc/python-foolscap/classify_foolscap.py , so you can make a symlink to it from an incident-gatherer. * misc/classify_foolscap.py (TUBCON_RE): update to match foolscap-tubconnector messages for both old and new versions of Foolscap, and for python2.4 and 2.5 . * foolscap/logging/incident.py (IncidentClassifier): add --verbose option to 'flogtool classify-incident' to show the trigger dictionary for any unclassifiable incidents. This is useful when developing classification functions. * foolscap/test/test_logging.py (Incidents.test_classify): test it * Makefile (test-figleaf): oops, fix the test-figleaf-poll target, by making the test-figleaf target use $(TRIAL), not hardcoded 'trial'. * foolscap/_version.py: bump revision to 0.3.2+ while between releases * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: same 2008-10-14 Brian Warner * foolscap/_version.py: release Foolscap-0.3.2 * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: same * MANIFEST.in: add misc/classify_foolscap.py to source distribution * NEWS: update for the next release * Makefile (test-poll): add a convenience target, for running tests on a system with the broken combination of pyopenssl and twisted that requires the pollreactor (test-figleaf-poll): same 2008-10-14 Brian Warner * foolscap/logging/tail.py (LogTail): include the retrieved PID in the saved logfile. Part of #80. * foolscap/logging/incident.py (IncidentReporter.incident_declared): also include the PID in the incident report's header. * doc/specifications/logfiles.xhtml: document the ['versions'] and ['pid'] fields in incident reports and the 'flogtool tail' savefile. * foolscap/logging/web.py: mark any incident triggers in the logfile, and add links to them from the welcome page. Closes #79. * foolscap/logging/web.py (Reload): add a 'Reload Logfile' button to the web-viewer's Welcome page. Not automatic, but it means you don't have to get back to the shell and restart the viewer. Works well-enough to say Closes #103. * misc/classify_foolscap.py: plugin to classify some foolscap-internal incidents * foolscap/logging/cli.py: new "flogtool classify-incident" subcommand: given an incident, say what categories it falls into. Closes #102. * foolscap/logging/gatherer.py (IncidentGathererService): factor out the classification pieces into IncidentClassifierBase * foolscap/logging/incident.py (IncidentClassifierBase): same (IncidentClassifier.run): support for the new CLI command * foolscap/test/test_logging.py (Incidents.test_classify): test it * foolscap/logging/gatherer.py (IncidentGathererService.classify_incident): change the classifier function signature: now it just takes a single 'trigger' dict. * foolscap/test/test_logging.py: update to match * doc/logging.xhtml (gatherer): same * foolscap/logging/gatherer.py (IncidentGathererService.add_classify_files): make the incident gatherer look in its base directory for "classify_*.py" files, use them as plugins with classification functions. (INCIDENT_GATHERER_TACFILE): update example text to match * foolscap/test/test_logging.py (IncidentGatherer): test it (Gatherer.test_wrongdir): exercise another error case * doc/logging.xhtml (gatherer): document it 2008-10-13 Brian Warner * foolscap/logging/publish.py (IncidentSubscription.catch_up): Don't tell the gatherer about incidents that don't have triggers: these are malformed logfiles, such as the zero-length files that result from a process being terminated before it manages to write anything. * foolscap/logging/log.py (Count): replace itertools.count with a version that doesn't overflow at 2**31-1 (thanks to Zooko for the patch). Closes #99. (FoolscapLogger.__init__): use it * foolscap/pb.py (Listener.startFactory): add log facility identifiers (Listener.stopFactory): same (Listener.buildProtocol): same (Tub.getReference): same 2008-09-20 Brian Warner * doc/listings/xfer-client.py (_failure): do sys.exit(1) upon failure, so the caller can detect it via the exit code 2008-09-10 Brian Warner * foolscap/test/test__versions.py (Versions.test_required): check for bug #62 (openssl>0.7, twisted<=8.1.0, selectreactor) and print a warning if it is likely that the tests would fail, to remind the user to re-run with -r poll. (Versions.test_required): oops, guard import of OpenSSL on having crypto available, otherwise this test gets an error when OpenSSL is not installed. 2008-09-08 Brian Warner * doc/listings/xfer-client.py: use os.path.expanduser() on the filenames passed in by the user, to enable things like --furlfile ~/.upload.furl * foolscap/_version.py: bump revision to 0.3.1+ while between releases * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: same 2008-09-03 Brian Warner * foolscap/_version.py: release Foolscap-0.3.1 * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: same * NEWS: update for next release 2008-09-02 Brian Warner * foolscap/negotiate.py (TubConnector.__repr__): add more information to the repr, to help track down #81. Made TubConnector a new-style class in the process, to make upcalling easier. (TubConnector.checkForIdle): improve the log message * foolscap/pb.py (Tub._log_gatherer_connected): use callRemoteOnly to pass the logport to the gatherer: we don't need to hear about any problems it has. * foolscap/logging/gatherer.py (IncidentGathererService.classify_stored_incidents): reclassify everything that isn't already present in one of the classified/* files. This makes it a lot easier to iterate over the [start gatherer; see what is unknown; update classifiers; remove classified/unknown; repeat] loop. Also log classification events better. Closes #94. * foolscap/test/test_logging.py (IncidentGatherer.test_emit): test it * foolscap/test/common.py (PollMixin): replace the use of chained Deferreds with a task.LoopingCall-based version, from Tahoe. This avoids the weird and annoying maximum-recursion-depth-exceeded error that occurs when the check function is called more than about 600 times. Closes #95. 2008-08-29 Brian Warner * foolscap/logging/app_versions.py (versions): move the application version dict (which gets reported to remote subscribers, and copied into logfiles) to a separate module. It was causing circular import problems when it lived in an attribute of LogPublisher. (add_version): provide a setter method * foolscap/logging/publish.py (LogPublisher.versions): same (LogPublisher.remote_get_versions): same * foolscap/logging/incident.py (IncidentReporter.incident_declared): same * foolscap/test/test_logging.py (IncidentGatherer.test_emit): improve test shutdown a little bit 2008-08-28 Brian Warner * foolscap/logging/incident.py (IncidentReporter.incident_declared): put versions in the incident header too, also for #80. * foolscap/test/test_logging.py (Incidents.test_basic): test it * foolscap/logging/tail.py (LogPrinter.got_versions): record remote application versions to the --save-to file, in the header. Part of #80. (LogPrinter.got_versions): oops, control where stdout goes * foolscap/logging/dumper.py (LogDumper.start): show those versions with --verbose * foolscap/test/test_logging.py (Dumper.test_dump): update to match (Tail.test_logprinter): same, make sure got_versions is called * foolscap/logging/gatherer.py (IncidentObserver): only fetch one incident at a time, to limit the size of the sender's outbound queue. This should help close #85. (IncidentGathererService.new_incident): include the classification results in the per-incident log messages * foolscap/test/test_logging.py (IncidentGatherer.test_emit): tests * foolscap/logging/log.py (FoolscapLogger.declare_incident): combine overlapping incidents, by passing new triggers to an existing reporter instead of creating a new one. Helps with #85. * foolscap/logging/incident.py (IncidentReporter.new_trigger): new method to add subsequent triggers to an existing reporter. This doesn't do anything yet. It should be improved to record the other triggers in a trailer (since it's too late to add it to the header). Every triggering event will make it into an incident somewhere, but the report-file analysis tools may not know how to pay attention to the subsequent triggers. * foolscap/logging/interfaces.py (IIncidentReporter.new_trigger): same * foolscap/test/test_logging.py (Incidents.test_overlapping): test it * doc/logging.xhtml: mention incident-coalescing * foolscap/logging/log.py: add bridge-foolscap-logs-to-twisted functionality, set up by either calling bridgeLogsToTwisted(), or by setting the FLOGTOTWISTED environment variable (to anything). The default filter will exclude events below the OPERATIONAL severity level, and those generated by foolscap internals (i.e. facility.startswith("foolscap") ). Closes #93. * foolscap/pb.py (Tub.setOption): update to match * foolscap/test/test_logging.py (Bridge): tests for it (Publish): update to use new APIs * doc/logging.xhtml: docs 2008-08-25 Brian Warner * foolscap/broker.py (Broker.abandonAllRequests): map both ConnectionLost and ConnectionDone into DeadReferenceError, so that application code only needs to check for one exception type. * foolscap/negotiate.py (Negotiation.evaluateNegotiationVersion1): when an existing connection is dropped in favor of a new one, drop it with DeadReferenceError instead of ConnectionDone. The mapping in Broker.abandonAllRequests doesn't seem to quite catch everything in unit tests. (Negotiation.acceptDecisionVersion1): same, when we're the slave * foolscap/test/test_call.py (TestCall.test_connection_lost_is_deadref): test it (TestCall.test_connection_done_is_deadref): same (TestCall.testChoiceOf): switch to use ShouldFailMixin * foolscap/test/test_gifts.py (Gifts): remove ignoreConnectionDone, just look for DeadReferenceError 2008-08-21 Brian Warner * foolscap/_version.py (verstr): bump revision to 0.3.0+ while between releases * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: same 2008-08-04 Brian Warner * foolscap/_version.py (verstr): release Foolscap-0.3.0 * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: 2008-08-04 Brian Warner * foolscap/logging/interfaces.py (RILogPublisher.subscribe_to_incidents): small edit, make it more clear that since= is used for catch_up=True * NEWS: update for the upcoming release * foolscap/referenceable.py (RemoteReferenceOnly.getRemoteTubID): make this secure, by using the broker's .remote_tubref field, instead of the remote-side-controlled sturdyref. (RemoteReferenceOnly.getSturdyRef): add a note about the insecurity of this method * foolscap/test/test_pb.py (TestCallable.testGetSturdyRef): add a test for getRemoteTubID * doc/logging.xhtml: change filenames in the incident-gatherer to have fewer files starting with incident*, so tab-completion works better. * foolscap/logging/gatherer.py (IncidentGathererService): same (create_incident_gatherer): same * foolscap/test/test_logging.py (Filter): test 'flogtool filter' * foolscap/logging/filter.py: control stdout/stderr better * foolscap/test/test_logging.py (Dumper): test 'flogtool dump' * foolscap/logging/cli.py (dispatch): pass stdout/stderr through attributes on the subcommand's Options instance, rather than as separate function arguments.. makes testing easier. * foolscap/logging/gatherer.py (create_log_gatherer): same (create_incident_gatherer): same (CreateGatherOptions): same (CreateIncidentGatherOptions): same * foolscap/logging/dumper.py (DumpOptions): same 2008-08-01 Brian Warner * foolscap/logging/log.py (LogFileObserver): make these more useful, by not doing the addSystemEventTrigger in __init__ (LogFileObserver.stop_on_shutdown): do it here instead (FLOGFILE): when using $FLOGFILE, add call to stop_on_shutdown * foolscap/logging/gatherer.py (GathererService.do_rotate): oops, implement the precautions claimed by the comment in startService: test self._savefile before doing anything. (GathererService.__init__): set self._savefile to None * foolscap/test/test_logging.py (Gatherer.test_log_gatherer2): add a timed rotator, which caught the problem in do_rotate * foolscap/test/test_logging.py (IncidentGatherer.test_emit): add test coverage for re-classifying existing incidents (IncidentGatherer.test_emit._update_classifiers_again): and verify that leaving the classified/ directory in place properly inhibits reclassification at startup * foolscap/referenceable.py (RemoteReferenceOnly.getRemoteTubID): new method to extract the remote tubid from a RemoteReference. I'm not sure this is a good idea, but it fixes the immediate problem I'm dealing with. * foolscap/logging/gatherer.py: use "tubid_s" instead of "nodeid_s" (IncidentGathererService.remote_logport): use getRemoteTubID() instead of trying to sanitize the tubid we receive, since we use it as a directory name (IncidentGathererService.add_classifier): rename addClassifier to add_classifier, I'm more fond of the latter form these days. Also remove a pyflakes warning. * foolscap/test/test_logging.py (IncidentGatherer.test_emit): add test of incident generation, publish, recording, and default classification * foolscap/logging/gatherer.py (IncidentGathererService): get control over stdout, so we can exercise more code during tests (IncidentGathererService.startService): oops, this needs to be startService instead of start * foolscap/test/test_logging.py (IncidentGatherer): basic test of an incident gatherer, just setup and connection so far * foolscap/test/common.py (StallMixin): factor stall() out into a separate class (TargetMixin): same * foolscap/test/test_logging.py (LogfileReaderMixin): refactor (Gatherer.test_log_gatherer): turn on bzip=True, for more coverage * foolscap/test/test_logging.py (Gatherer._check_gatherer): ignore internal foolscap messages (like connection negotiation), since they occur at unpredictable times (specifically in test_log_gatherer_furlfile_multiple, which has establishes multiple connections) * foolscap/test/common.py (GoodEnoughTub): factor this and crypto_available out of all the other unit tests, make GoodEnoughTub(certFile=) work even if crypto is unavailable and we must therefore discard the certFile= argument. * foolscap/test/test_crypto.py: same * foolscap/test/test_gifts.py: same * foolscap/test/test_keepalive.py: same * foolscap/test/test_logging.py: same * foolscap/test/test_loopback.py: same * foolscap/test/test_negotiate.py: same * foolscap/test/test_pb.py: same * foolscap/test/test_serialize.py: same * foolscap/test/test_tub.py: same * foolscap/logging/gatherer.py (GatheringBase): rewrite Gatherers to make them easier to test: now they are a Service (with a subordinate Tub), meant to be run by a .tac file or manually. The intermediate class has been removed. All .tac files are unchanged: gatherers created by old versions of 'flogtool create-gatherer' will continue to work. (GathererService.do_rotate): return the name of the logfile that was just closed and/or compressed, so tests can know where to look. * foolscap/test/test_logging.py (Gatherer): update to match * foolscap/util.py (get_local_ip_for): update reactor comment 2008-07-30 Brian Warner * setup.py: update comment about bug #62 (pyopenssl problems) * foolscap/test/test_logging.py (CLI.test_create_gatherer): improve test coverage a little bit, by recording stdout (CLI.test_create_incident_gatherer): exercise the 'flogtool create-incident-gatherer' command * foolscap/logging/cli.py (run_flogtool): capture stdout+stderr (dispatch): same * foolscap/logging/gatherer.py (create_log_gatherer): same (create_incident_gatherer): same * trees/tahoe/Makefile (figleaf-output): exclude foolscap/test/ from the HTML results * doc/logging.xhtml (Running an Incident Gatherer): describe the Incident Gatherer, like the Log Gatherer but it only gathers incidents. It also does classification, and will eventually do reporting. No unit tests yet, but some manual system-level tests have been run. * foolscap/logging/gatherer.py (IncidentGatherer): implement it * foolscap/logging/cli.py (Options.subCommands): add the CLI command, named 'flogtool create-incident-gatherer' * foolscap/logging/incident.py (IncidentReporter.incident_declared): clean up incident naming * foolscap/logging/interfaces.py (RILogPublisher.list_incidents): same * foolscap/logging/log.py (FoolscapLogger.incident_recorded): same * foolscap/logging/publish.py (LogPublisher.list_incident_names): same * foolscap/test/test_logging.py (IncidentPublisher.test_list_incident_names): test the incident-naming cleanup 2008-07-29 Brian Warner * foolscap/logging/gatherer.py (GatheringBase): refactor the log gatherer to share code with the upcoming incident gatherer (LogGatherer.__init__): add a basedir= argument, rather than using os.getcwd * foolscap/test/test_logging.py (Gatherer): add basedir= argument * doc/logging.xhtml (Setting up the logport): New feature (well, it didn't work before, and now it does, so make it explicit): the log-gatherer FURL can be configured (but will not be connected) until after setLocation. This should resolve a crash I've seen in Tahoe (which runs a slow /sbin/ifconfig command to figure out the addresses to pass to setLocation) in which the app connects to the log gatherer before it figures out its own location, and then gets an exception during registerReference. Closes #55. * foolscap/pb.py (Tub._maybeConnectToGatherer): don't initiate the gatherer connection until locationHints is set (Tub.setLocation): call _maybeConnectToGatherer after the location is set. Also, don't let setLocation be called multiple times. * foolscap/test/test_logging.py (Publish.test_logport_furlfile2): test it (Gatherer.test_log_gatherer2): same (Gatherer.test_log_gatherer_furlfile2): same * doc/logging.xhtml (Setting up the logport): slight API restriction: the logport and its FURL are not available until after Tub.setLocation is called. This results in a better error message than the usual one inside registerReference. * foolscap/pb.py (Tub.getLogPort): enforce the API restriction by throwing an exception when it is violated (Tub.getLogPortFURL): same * foolscap/tokens.py (NoLocationError): new exception for it * foolscap/test/test_tub.py (SetLocation.test_set_location): test it * doc/logging.xhtml (Configuring a Log Gatherer): allow multiple log-gatherer furls in the log-gatherer-furlfile * foolscap/pb.py (Tub._maybeConnectToGatherer): same * foolscap/logging/gatherer.py (LogGatherer.remote_logport): return the subscribe_to_all Deferred, for testing * foolscap/test/test_logging.py (Gatherer.test_log_gatherer): refactor, to accomodate new test (Gatherer.test_log_gatherer_furlfile_multiple): test it 2008-07-28 Brian Warner * foolscap/logging/interfaces.py (RILogPublisher.list_incidents): change signature to remove tubid/incarnation from the response. (RILogPublisher.subscribe_to_incidents): add pubsub interface for incidents, including catch_up= and since= (RILogObserver.new_incident): same (RILogObserver.done_with_incident_catchup): same * foolscap/logging/publish.py (IncidentSubscription): same * foolscap/logging/incident.py (IncidentReporter.finished_recording): tell the logger about the incident name, so it can publish it. * foolscap/logging/log.py (FoolscapLogger.addImmediateIncidentObserver): same (FoolscapLogger.incident_recorded): same * foolscap/test/test_logging.py (IncidentPublisher._check_listed): same (IncidentPublisher.test_subscribe): test it * foolscap/logging/interfaces.py (RISubscription.unsubscribe): move the RILogPublisher.unsubscribe() method to the RISubscription object, since that's a better place for it. (RILogPublisher.unsubscribe): deprecate this one * foolscap/logging/publish.py (Subscription.remote_unsubscribe): same * foolscap/logging/publish.py (LogPublisher.remote_get_incident): reject invalid incident names * doc/logging.xhtml: add details about running a Log Gatherer (Python 'logging' module): comment out this section, it is wrong 2008-07-09 Brian Warner * doc/specifications/logfiles.xhtml: fix typos * doc/logging.xhtml (Remote log aggregation): same 2008-07-07 Brian Warner * foolscap/logging/dumper.py (LogDumper): when dumping an Incident Report, mark the triggering event with "[INCIDENT-TRIGGER]" * foolscap/_version.py (verstr): bump revision to 0.2.9+ while between releases * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: same 2008-07-02 Brian Warner * foolscap/_version.py (verstr): release Foolscap-0.2.9 * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: same * NEWS: update for the upcoming release * MANIFEST.in: add gutsy/hardy directories 2008-07-02 Brian Warner * foolscap/test/test_logging.py (Publish.test_logpublisher_overload): change poller condition to avoid spurious failures on slow systems * misc/*/debian/watch: update to point at foolscap.lothar.com * Makefile (debian-gutsy, debian-hardy): add .deb targets for gutsy and hardy. The rules are the same as for feisty. Closes #76. * misc/gutsy/*, misc/hardy/*: same * foolscap/logging/publish.py (Subscription.send): change the discard policy to discard-new instead of discard-random: in tests with a busy Tahoe node, this seems to give good enough behavior (probably since the busyness is bursty), although I can't say we fully understand what's really going on. discard-new lets us use a faster and simpler deque instead of requiring random access to the list. * foolscap/test/test_logging.py (Publish.test_logpublisher_overload): update to match 2008-07-02 Brian Warner * foolscap/logging/publish.py (Subscription.send): add a size-limited queue for sending messages to subscribers, with a default queue size of 2000 entries. This should probably keep the memory footprint bounded to perhaps 1MB. When the queue gets full, we randomly discard old messages, so recent messages are more likely to survive than earlier ones. We allow 10 messages to be outstanding on the wire at once, to pipeline them a bit and improve network utilization. Any errors during sending will cause the subscription to be dropped. This should close #72. * foolscap/logging/log.py (FoolscapLogger.addImmediateObserver): add a new kind of observer, so that the logport publisher can also avoid unboundedness in the eventual-send queue. If we can't throw away messages fast enough, callers to log.msg will block, slowing down the inlet rate. * foolscap/test/test_logging.py (Publish.test_logpublisher_overload): test it, by throwing 10k messages at log.msg and counting how many make it through. * foolscap/test/common.py (PollMixin): add a comment: this chained-Deferred pattern will run up against python's recursion limit if the check function is called more than about 300 times. 2008-07-01 Brian Warner * foolscap/logging/publish.py (Subscription): refactor a bit, in preparation for #72 limit log-publishing queue size 2008-07-01 Brian Warner * doc/logging.xhtml ("That Was Weird" Buttons): provide a simple example of giving the user a way to trigger Incident logging. Closes #75. * foolscap/logging/interfaces.py (RILogPublisher.get_pid): new interface, to retrieve the process ID through the logport * foolscap/test/test_logging.py (Publish.test_logpublisher_catchup._got_logport._check_pid): test it * foolscap/logging/publish.py (LogPublisher.remote_get_pid): implement it * foolscap/logging/tail.py (LogTail._got_logpublisher._announce): Use it, but tolerate old logports that don't offer it. Closes #71. * foolscap/slicers/decimal_slicer.py: handle Decimal objects, serializing them as a string. No constraints yet. Closes #50. * foolscap/slicers/allslicers.py: import decimal_slicer * foolscap/test/test_banana.py (ThereAndBackAgain.test_decimal): test it * foolscap/logging/tail.py (LogTail._got_logpublisher): exit if we get an error while connecting or subscribing. Closes #63. (LogTail._print_versions): print remote versions immediately after connecting to the logport. Closes #70. * foolscap/logging/log.py (FoolscapLogger.setLogDir): oops, allow the incident directory to be re-used: os.makedirs throws an exception if the directory already exists. * foolscap/test/test_logging.py (Incidents.test_basic): test it * foolscap/_version.py (verstr): bump revision to 0.2.8+ while between releases * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2008-06-04 Brian Warner * foolscap/_version.py (verstr): release Foolscap-0.2.8 * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same * NEWS: update for the upcoming release 2008-06-04 Brian Warner * foolscap/pb.py (Tub._tubsAreNotRestartable): fix args so that these methods actually get run (and produce a useful error message, instead of TypeError). Thanks to Brian Granger for the patch. Closes #65. * foolscap/test/test_tub.py (Shutdown.test_doublestop): test it * foolscap/test/common.py (ShouldFailMixin.shouldFail): add some support code * foolscap/referenceable.py (RemoteReferenceOnly.getPeer): new method to get the IP address and port number of the other end of a connection. This returns a twisted.internet.interfaces.IAddress provider. Loopback connections give foolscap.broker.LoopbackAddress instances. Real remote connections give twisted.internet.address.IPv4Address instances, so you can use rref.getPeer().host and rref.getPeer().port on them. Closes #45. * setup.py: add an "extras_require" clause to the setuptools-specific setup args, to declare that our "secure_connections" feature requires pyOpenSSL. This helps other packages, which can declare a dependency on "Foolscap[secure_connections]", rather than claiming to require pyOpenSSL themselves. Addresses #66. * foolscap/pb.py (Tub.registerReference): fix an exception that occurs if you call this with both name= and furlFile= and the furlFile already exists. Also prohibit attempts to change the name to something other than what is in the furlFile. Thanks to Brian Granger for the patch. Closes #64. * foolscap/_version.py (verstr): bump revision to 0.2.7+ while between releases * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2008-05-13 Brian Warner * foolscap/_version.py (verstr): release Foolscap-0.2.7 * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same * NEWS: update for new "OMG flogtool is broken" release. 2008-05-12 Brian Warner * foolscap/logging/cli.py (run_flogtool): fix use of sys.argv, the previous version was completely broken * foolscap/_version.py (verstr): bump revision to 0.2.6+ while between releases * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2008-05-06 Brian Warner * foolscap/_version.py (verstr): release Foolscap-0.2.6 * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same * NEWS: update for the upcoming release 2008-05-05 Brian Warner * foolscap/test/test_logging.py (Web): add tests for 'flogtool web-viewer' * foolscap/logging/web.py (WebViewer.start): make this class more amenable to being tested * foolscap/logging/interfaces.py: misc cleanups (RILogPublisher.list_incidents): provide an interface to retrieve stored incident reports from a logport. (RILogPublisher.get_incident): same * foolscap/logging/publish.py (LogPublisher): same * foolscap/test/test_logging.py (IncidentPublisher): test it * foolscap/pb.py (Tub.setup): make the Tub's logger a bit easier to override, for tests * foolscap/logging/incident.py (IncidentReporter): make the compressed logfile use a temporary name until we've closed it, so later observers don't get tricked into using an incomplete compressed logfile (they should use the more-complete uncompressed logfile instead). (IncidentReporter.incident_declared): make it easier to turn off the gather-trailing-events behavior (NonTrailingIncidentReporter): convenience subclass that does that * doc/logging.xhtml: update to match current reality, remove some TODO warnings * foolscap/logging/log.py (FoolscapLogger): rename setIncidentReporterClass to setIncidentReporterFactory * foolscap/test/test_logging.py (Incidents.test_customize): same * foolscap/logging/interfaces.py (RISubscription): remove spurious 'pass', figleaf thought it was real code * bin/flogtool: split out the CLI dispatcher to.. * foolscap/logging/cli.py (run_flogtool): here, to help with #51 (run_flogtool): make it possible to run with a wrapper, closes #51. * foolscap/test/test_logging.py (CLI.test_wrapper): test a wrapper, provide an example of how to build one * foolscap/logging/gatherer.py (CreateGatherOptions.optFlags): add a --quiet flag to make the unit test less noisy * misc/testutils/trial_figleaf.py: make this compatible with twisted-8.0.x: the earlier version didn't write out any coverage data on newer twisteds * foolscap/logging/log.py (FoolscapLogger.setLogDir): wire up Incident handling. We still need incident publishing. This gets us most of the way to #61. (setIncidentQualifier): this customizes the qualification function (setIncidentReporterClass): this customizes the reporter/recorder * foolscap/logging/levels.py: move log.WEIRD and friends here to avoid circular import problems elsewhere. They are still available from log.py . * foolscap/logging/incident.py (IncidentReporter.incident_declared): flush the uncompressed logfile just before we switch into gather-trailing-events mode (IncidentReporter.stop_recording): oops, fix typo * foolscap/test/test_logging.py (Incidents): test most incident-handling functionality 2008-05-02 Brian Warner * foolscap/logging/incident.py: start to implement Incident handling, for ticket #61. Not complete yet, might be completely broken, needs tests and to be wired up. * foolscap/logging/log.py (FoolscapLogger.setIncidentQualifier): start to add the new interfaces, not complete yetxo * foolscap/logging/interfaces.py (IIncidentReporter): same * doc/logging.xhtml (Incidents): document the new features 2008-05-01 Brian Warner * doc/logging.xhtml: fix discussion of format=, since the instructions and examples were simply wrong * doc/specifications/logfiles.xhtml (Logfile Headers): define "headers", a separate dictionary at the start of the logfile that contains metadata. The specific use for this will be the "Triggering Event" that gets put into incident reports, once we implement those. This induces a backwards compatibility break: logfiles produced after this change will probably not be tolerated by tools like 'flogtool dump' from before this point. The other direction is ok: newer tools can handle either format. * foolscap/logging/gatherer.py (LogGatherer._open_savefile): emit a header, with type="gatherer", and a starting timestamp in "start". Refactor a bit to make this easier. * foolscap/logging/tail.py (LogPrinter): emit a header, with type="tail" * foolscap/logging/log.py (LogFileObserver): emit a header, with type="log-file-observer", and a "threshold" key * foolscap/logging/dumper.py (LogDumper.start): tolerate headers * foolscap/logging/filter.py (Filter.run): same * foolscap/logging/web.py (WebViewer.process_logfiles): same * foolscap/test/test_logging.py: update tests to match * doc/specifications/logfiles.xhtml: document the current saved logfile format (event dictionaries, pickled wrapper dicts) * foolscap/referenceable.py (SturdyRef): accept multiple connection hints in unauthenticated FURLs. This fixes a test failure induced by the #60 changes when pyOpenSSL is unavailable. * doc/using-foolscap.xhtml: document the new feature. * foolscap/referenceable.py (SturdyRef): tolerate extensions in tubid and location-hints fields, change FURL parsing, store structured hints in the SturdyRef instead of just strings. This should give us some wiggle room in the future to gracefully transition applications to using new features while retaining backwards compatibilty. Many thanks to Zooko for the suggestion. Closes #60. * foolscap/negotiate.py (TubConnector): same * foolscap/test/test_sturdyref.py (URL.testTubIDExtensions): new tests for it (URL.testLocationHintExtensions): same * foolscap/test/test_tub.py (SetLocation.test_set_location): update to match * foolscap/test/test_gifts.py (Bad): same * foolscap/test/test_negotiate.py (Versus.testVersusHTTPServerAuthenticated): same * foolscap/base32.py (is_base32): new utility function * foolscap/test/test_util.py (Base32.test_is_base32): test it 2008-04-22 Brian Warner * doc/listings/command-client.py: don't emit extra newlines, use /usr/bin/env on shbang line * doc/listings/command-server.py: use /usr/bin/env on shbang line 2008-04-22 Brian Warner * foolscap/logging/filter.py: add "--above UNUSUAL" option (to discard events below the given level", and "--from [TUBID]" (to discard events that weren't recorded by the given tubid). * foolscap/logging/dumper.py (LogDumper.print_event): use --rx-time to show both event-generation time and event-receive time in the logs, useful if you suspect the application is getting bogged down and events are being delivered slowly. (LogDumper.print_event): Use a different format to display failure tracebacks. * foolscap/logging/web.py (WebViewer.get_events): catch ValueError, mention it as a possible truncated pickle file * foolscap/logging/log.py (FoolscapLogger.msg): if something goes wrong, print both str() and repr() in case it helps figure out the problem * foolscap/logging/dumper.py (LogDumper.print_event): include failure tracebacks in 'flogtool dump' output 2008-04-09 Brian Warner * foolscap/constraint.py (Constraint.checkOpentype): always accept ('reference',) sequences. The per-token constraint checking system is a defense against resource-exhaustion attacks, and shared reference don't consume any more memory or stack frames than any other object. The check-all-args that CallUnslicer does just before delivering the arguments is responsible for making sure the final (resolved) types all match the constraint. By allowing shared references here, we fix a bug in which a schema violation was raised when python combined two equivalent tuples into a single object. * foolscap/test/test_call.py (TestCall.testMega3): test it * foolscap/test/common.py (MegaSchema3): same * setup.py: finally remove zip_safe=False * foolscap/test/test_interfaces.py (TestInterface.testStack): allow the test to pass even if the source code isn't available (i.e. it's locked away inside an egg). This ought to remove the need for zip_safe=False. * foolscap/test/test_logging.py (Tail.test_logprinter): make the test pass on python2.5: I think twisted now emits type(exception) instead of str(exception), or something: the printed form changed. * foolscap/schema.py (PolyConstraint): improve error messages * foolscap/test/test_logging.py (Tail.test_logprinter): improve error messages, to figure out why this fails on python2.5 * foolscap/__init__.py (_unused): add __version__, to hush pyflakes * foolscap/test/test_logging.py (Publish.test_logpublisher): make the test work with twisted-8.0.x, which puts more information in log.err Failures than earlier versions did. * foolscap/__init__.py (__version__): move version from here.. * foolscap/_version.py (verstr): .. to here * foolscap/setup.py: merge zooko's patch to get foolscap version by scanning foolscap/_version.py instead of importing foolscap. This makes setuptools a lot happier: it doesn't need to have Twisted installed (so foolscap can be imported) while it's building foolscap as part of automatic dependency satisfaction. Also change the shebang line to use /usr/bin/env, and import (but do not use) setuptools if it is available. We set 'zip_safe=False' in the extra setuptools arguments because two of the unit tests assert that their stack traces have source code lines in them, and that doesn't happen if foolscap is living inside an egg. We will probably change those tests soon and allow zipped eggs. 2008-03-31 Brian Warner * foolscap/test/test_logging.py (Tail): basic tests for logging.tail * foolscap/logging/tail.py (LogPrinter.remote_msg): parameterize the output filehandle so we can test it * foolscap/logging/gatherer.py (LogGatherer): remove the "Z" suffix from the from- and to- timestamps that we put into filenames, since we're actually using localtime. If and when we switch to use UTC, we'll bring the Z back. Also remove the hyphens between the time portions: from-2008-03-31-161721 instead of from-2008-03-31-16-17-21. I'm still looking for a format that feels readable, scannable, and clear. * foolscap/logging/tail.py (LogPrinter.formatted_print): remove spurious "0" from timestamps: i.e. print "08:09:12.345" instead of "08:09:120.345" * foolscap/logging/web.py (LogEvent.to_html): same * foolscap/__init__.py: bump revision to 0.2.5+ while between releases * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2008-03-25 Brian Warner * foolscap/__init__.py: release Foolscap-0.2.5 * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2008-03-25 Brian Warner * MANIFEST.in: add LICENSE * NEWS: update for the upcoming release * bin/flogtool (Options.synopsis): update with all commands. There's still something broken with --help output, though. * doc/listings/command-server.py: * doc/listings/command-client.py: new sample programs, a client/server pair which lets the client trigger a specific command to be run on the server. Like xfer-server.py, but for running commands instead of transferring files. * foolscap/test/common.py (PollMixin): refactor a bit, to extract the poll() method for use by other tests * foolscap/test/test_logging.py (Publish.test_logpublisher): improve the timing a bit, by waiting until the observer has heard only silence for a full second, instead of starting the verify pass one second after subscribing. (Publish.test_logpublisher_catchup): same. It makes a bigger difference here, because catch_up=True means that we'll be seeing several hundred messages, which may take a non-trivial amount of time to receive. I was seeing intermittent test failures with the one-second-from-subscribe stall. 2008-03-24 Brian Warner * foolscap/logging/interfaces.py (RILogPublisher.subscribe_to_all): add a catch_up= argument, which causes the publisher to dump all its stored messages just after adding the subscriber. Closes #49. * foolscap/logging/log.py (FoolscapLogger.get_buffered_events): same (LogFileObserver.msg): catch+print exceptions while pickling events * foolscap/logging/publish.py (LogPublisher): add catch_up=, make sure that any log events arrive after subscribe_to_all() has returned, to make event sequencing easier on the subscriber. Also make sure that any catch-up events arrive before subsequent log events. * foolscap/logging/tail.py: add --catch-up option, make sure we remain compatible with <=0.2.4 publishers (as long as you don't use --catch-up) * foolscap/test/test_logging.py (Publish.test_logpublisher_catchup): test it (Publish.test_logpublisher_catchup): make test more reliable * LICENSE: make it clear that Foolscap ships under the MIT license, the same as Twisted uses. Closes #47. * README: same * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/copyright: same 2008-03-24 Brian Warner * foolscap/logging/dumper.py (LogDumper.print_event): event dicts store the printable tubid, not a binary form * foolscap/logging/gatherer.py (Observer.__init__): same (LogGatherer.remote_logport): same * foolscap/logging/interfaces.py (TubID): same * foolscap/logging/web.py (LogEvent.__init__): same * foolscap/logging/gatherer.py (LogGatherer.__init__): make bzip= argument optional, for unit tests * foolscap/logging/tail.py: fix --save-to, also put exception information into the twistd.log when we can't pickle the event * foolscap/logging/gatherer.py: add --rotate and --bzip options to 'flogtool create-gatherer': rotate the logfile every N seconds, and optionally compress the results. Each logfile gets named like from-2008-03-24-20-31-46Z--to-2008-03-24-20-31-56Z.flog.bz2 , and the open one is named from-2008-03-24-20-32-16Z--to-present.flog . Sending SIGHUP to the gatherer will force a rotation. Closes #48. * foolscap/logging/tail.py: add --save-to option to 'flogtool tail', which saves the log events to a file (in addition to printing them to stdout). 'flogtool dump' can be used on the saved file later. Also refactor things a bit to let us grab the tubid from the target furl, since this gets recorded in the save file format. * foolscap/logging/gatherer.py (LogSaver): move to tail.py * foolscap/logging/tail.py (LogPrinter): print formatted event lines by default. Use the new --verbose option to dump raw event dictionaries. Closes #43. * bin/flogtool (dispatch): same 2008-02-17 Brian Warner * doc/using-foolscap.xhtml: fix href, thanks to Stephen Waterbury for the catch. Note that the .xhtml points to .xhtml, and lore converts the target of the link to .html in the .html output. * doc/copyable.xhtml: same * foolscap/logging/gatherer.py: make 'flogtool create-gatherer' to build a .tac file (which can be launched with twistd) rather than starting a gatherer right away. This is much more useable in practice. * bin/flogtool: same 2008-01-31 Brian Warner * foolscap/test/test_tub.py: hush pyflakes, make it work without crypto too * doc/listings/xfer-server.py: * doc/listings/xfer-client.py: new sample programs, a client/server pair which allow the client to put files in the server's directory. Useful as a replacement for restricted-command passphraseless ssh key arrangements. * foolscap/pb.py (Tub.setLocationAutomatically): new method that guesses an externally-visible IP address and uses it to call setLocation(). * foolscap/test/test_tub.py (SetLocation.test_set_location): test for it * foolscap/util.py (get_local_ip_for): make get_local_ip_for() more generally available, moving out of gatherer.py * foolscap/logging/gatherer.py: same * foolscap/negotiate.py (TubConnector.connectToAll): fix log message * foolscap/logging/web.py (LogEvent.__init__): include the incarnation number in the anchor index, so that logfiles which contain events from multiple incarnations of the same Tub will not suffer from href collisions * foolscap/broker.py: switch to foolscap.logging (Broker.connectionLost): log this event (Broker.freeYourReference): same, at log.UNUSUAL (Broker._callFinished): same, at log.UNUSUAL * foolscap/__init__.py: bump revision to 0.2.4+ while between releases * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2008-01-28 Brian Warner * foolscap/__init__.py: release Foolscap-0.2.4 * misc/{sid|sarge|dapper|edgy|feisty}/debian/changelog: same 2008-01-28 Brian Warner * NEWS: update for the upcoming release * foolscap/logging/filter.py (FilterOptions.parseArgs): make it possible to filter the event file in place, by using 'flogtool filter filename' instead of 'flogtool filter infile outfile' * foolscap/negotiate.py (Negotiation.evaluateNegotiationVersion1): include tubid in error message * foolscap/call.py (CopiedFailure.__getstate__): don't use reflect.qual on a string. This might improve behavior when we copy failures around, particularly in logging. 2008-01-18 Brian Warner * foolscap/negotiate.py: put all negotiation flog messages in facility "foolscap.negotiation", to make it easier to strip them out with 'flogtool filter --strip-facility foolscap' * foolscap/call.py (PendingRequest.fail): switch remote-exception logging over to flogging (InboundDelivery.logFailure): switch local-exception logging to flogging too * foolscap/test/test_pb.py (TestCallable.testLogLocalFailure): match it (TestCallable.testLogRemoteFailure): same * foolscap/logging/log.py (FoolscapLogger.__init__): remove unused .buffer attribute (FoolscapLogger.msg): avoid pickling application-specific exception classes when using failure=; use CopiedFailure to avoid it. This allows the emitted log pickle to be loaded on systems that do not have the original source code around. * foolscap/logging/filter.py: add --strip-facility option, to remove events that pertain to a given facility or its children 2008-01-17 Brian Warner * foolscap/logging/web.py (EventView.render): add a sort= query argument. If set to 'nested' (the default), you get the default nested view, in which the root events (those without a parent) are shown in chronological order, and the child events of each node are shown in chronological order underneath it. If sort=time, then no nesting is used, and all events are shown in chronological order. If sort=number, then all events are shown in numerical order, which is nominally better for single-process logfiles and coarse timestamps, but practically speaking is no better than sort=time. Using a non-nested mode can make it easier to spot events that happen in different areas but at about the same time; the nested display puts these events further apart (EventView._emit_events): minor formatting change 2008-01-10 Brian Warner * bin/flogtool (Options.opt_version): make 'flogtool --version' show the foolscap version. * foolscap/logging/web.py (SummaryView): add summaries to the 'flogtool web-viewer' tool: show counts of events by severity level, show lists of those events, with each event hyperlinked to the correct line in the full display. The full display also has anchor tags which let you construct bookmarks to specific lines. * foolscap/logging/filter.py (Filter): add 'flogtool filter' subcommand, to take one large eventlog pickle file and produce a smaller one with just a subset of the events. Currently this only allows you to filter by timestamp, and requires timestamps be provided as seconds since epoch. * bin/flogtool (dispatch): same * foolscap/logging/log.py (FoolscapLogger.err): add log.err, which behaves just like twisted's log.err: it accepts an exception or Failure object, or it can be used inside an except: clause to log the current exception. * foolscap/test/test_logging.py (Publish): test it * foolscap/logging/gatherer.py (LogSaver.remote_msg): if we can't pickle something, complain to stdout, rather than causing an error message to be sent back to the sender's Tub. * foolscap/call.py (CopiedFailure.__getstate__): make CopiedFailures pickleable, and make sure they come back looking just like they started. The issue was that we play games with the .type attribute to make .trap/.check work. * foolscap/test/test_copyable.py (Copyable._testFailure1_1): test it (Copyable._testFailure2_1): same * Makefile (pyflakes): new pyflakes doesn't uniqueify its output * NEWS: fix misspelling * foolscap/__init__.py: bump revision to 0.2.3+ while between releases * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2007-12-24 Brian Warner * foolscap/__init__.py (__version__): release 0.2.3 * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2007-12-24 Brian Warner * setup.py: point url= at the trac page, instead of the root, since I keep forgetting to update the tarball links on the root * NEWS: update for the upcoming release * foolscap/logging/log.py (LogFileObserver.__init__): open the flogfile with mode "wb" instead of "a", so that both compressed and uncompressed files work the same way. This truncates the file on each run instead of appending to it. * foolscap/broker.py (Broker.getRemoteInterfaceByName): remove duplicate definition of method, detected by pyflakes * foolscap/test/test_banana.py: change import of names from foolscap.tokens to appease the new (stricter) pyflakes * foolscap/logging/gatherer.py (get_local_ip_for): move imports up to module level, remove duplicates * foolscap/pb.py (Tub.setLogPortFURLFile): if the application has done setOption("logport-furlfile"), make sure the logport furlfile is created as soon as possible (i.e. when setLocation is called). Allow these two calls to occur in either order. The setOption call must still occur before doing getLogPortFURL(), or before the tub connects to the log gatherer; otherwise the furlfile will be ignored. Closes #38. * foolscap/test/test_logging.py (Publish.test_logport_furlfile1): test it * doc/logging.xhtml: update to match, fix a few typos * foolscap/referenceable.py (RemoteReferenceOnly.getSturdyRef): fix this, it had bitrotted. Closes #35. * foolscap/test/test_pb.py (TestCallable.testGetSturdyRef): test it * foolscap/logging/log.py (FoolscapLogger.msg): only record e['args'] if there were any, to avoid spurious formatting attempts * foolscap/negotiate.py: update log.msg calls to use format= * foolscap/logging/log.py (format_message): refactor event formatting into a separate function. Switch to using Twisted's style of format= argument to indicate that we want to use keyword-argument formatting. Closes #39. (FoolscapLogger.msg): same, use format_message() to test whether the event is stringifiable or not. (TwistedLogBridge._old_twisted_log_observer): copy dict directly if there's a format= kwarg, or stringify the message portion if there isn't. I *think* this ought to match what twisted does. * foolscap/logging/dumper.py (LogDumper.print_event): same * foolscap/logging/web.py (LogEvent.to_html): use format_message * foolscap/test/test_logging.py (Basic.testLog): improve tests 2007-12-23 Brian Warner * foolscap/negotiate.py (Negotiation.__init__): log everything under the 'foolscap.negotiation' facility, and use NOISY by default. (Negotiation.log): same * foolscap/logging/log.py (NOISY): log levels are now defined in terms of their stdlib 'logging' counterparts, so they're ints from a larger scale (10 to 40). This removes the need for 'levelmap', and allows log levels to be compared better. (FoolscapLogger.msg): all messages get a ['level'] key (which defaults to OPERATIONAL), so that observers and viewers don't have to define a default or handle a missing key. (FoolscapLogger.add_event): observers are now a simple callable, and they're always invoked with eventually(). log-to-file is now done with an observer, as are remote subscribers. (FoolscapLogger.logTo): remove this, log-to-file is now done with an observer (LogFileObserver): new class to implement log-to-file. This accepts a minimum level to pay attention to. Pickling is done at level '2'. It will compress the output if the filename ends in .bz2, and will use twisted's reactor.addSystemEventTrigger to try and close the file at shutdown. This is good enough for trial test cases, since trial helpfully fires this trigger for us after all test cases have been run. (FLOGFILE): if the $FLOGFILE environment variable is set, write all log events of $FLOGLEVEL or higher to the named file (opened at import time). Set FLOGLEVEL=1 to include NOISY debug messages. Set FLOGTWISTED=1 to get twisted.log events in the same file. Use all three when running trial tests to see what foolscap is saying as the tests run, but note that you can't do FLOGFILE=_trial_temp/flog.out.bz2 because foolscap is usually imported before trial creates (or clears) that directory, thus deleting the newly-created flogfile. Applications are expected to use a better API to control log-to-file, but it doesn't really exist yet. * foolscap/logging/dumper.py (LogDumper): accept .bz2 files * foolscap/logging/web.py (WebViewer.get_events): same * foolscap/logging/publish.py (LogPublisher): observers are now simple callables, so build a wrapper to do the callRemote for us. Track both the wrapper and the subscriber, since we need to use them both at unsubscribe time. * foolscap/test/test_logging.py (Basic.testLog): make sure we can tolerate '%' in log messages. (this needs to be changed, see #39). Also test the generation threshold, and using num= (Advanced.testObserver): test the new observer-as-callable scheme (Advanced.testPriorities): facility names are now dot-separated 2007-12-21 Brian Warner * doc/logging.xhtml: define facilities as dot-separated instead of slash-separated, to match stdlib logging package * foolscap/__init__.py: bump revision to 0.2.2+ while between releases * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2007-12-12 Brian Warner * foolscap/__init__.py (__version__): release 0.2.2 * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2007-12-12 Brian Warner * NEWS: update for new release * foolscap/negotiate.py (Negotiation): fix duplicate-connection handling logic. Simplify by removing the attempt-id information, and thus reject any connections with old seqnums (to prevent connection flap for parallel (connection-hint) offers). Reject connections from master_IR=None. Send the 'my-incarnation' field with all offers (both client and server), so that it will be available to whomever the master happens to be. * foolscap/broker.py (Broker.__init__): remove attempt-id * foolscap/test/test_negotiate.py (Replacement): update to match * foolscap/negotiate.py: covert logging to flogging (i.e. call foolscap.logging.log.msg instead of twisted.log.msg). This gives us parent/child structure, more useful severity levels, improves display of Failure instances, and will give us more data to work with in the future as the foolscap.logging tools mature. * foolscap/pb.py: same * foolscap/test/test_negotiate.py: same * bin/flogtool: move all usage.Options into the file that contains the implementations: TailOptions are moved to foolscap.logging.tail, etc. * foolscap/logging/gatherer.py: same * foolscap/logging/tail.py (TailOptions): same * foolscap/logging/dumper.py: same (DumpOptions.optFlags): add --verbose (show all event keys), --just-numbers. (LogDumper.print_event): change event printing to handle funny messages * foolscap/logging/web.py (WebViewerOptions.optParameters): same (FLOG_CSS): colorize the background of unusual messages (LogEvent): improve stringification, add better timestamps, event numbers, escape HTML better, display Failure instances better, (WebViewer.process_logfiles): event numbers use 'num', not 'number' * foolscap/logging/log.py (FoolscapLogger.set_generation_threshold): implement generation thresholds: don't even record messages if they fall below this threshold. Still needs a lot of usability work. The default is level>=NOISY, which records everything. (FoolscapLogger.logTo): give this a filename, and all log messages will be pickled and written to the given file. Messages are written with the same dict wrapper as 'flogtool gather' uses, so they are displayable by 'flogtool dump' or 'flogtool web-view'. Still needs work, ideally this would be a file to which we dump buffered messages once a problem is detected. To enable this from, say, trial, set os.environ['FLOGFILE'] to a filename. Also, set FLOGTWISTED to enable a twisted.log-to-flog bridge. (FoolscapLogger.msg): implement generation_threshold, also change stringification to survive things like "100%" in the log message. This needs work too, the 'except ValueError' clause is icky. * foolscap/pb.py (Tub.brokerAttached): use an eventual-send when informing everyone in waitingForBrokers, to match the asynchronicity of disconnection notifications delivered in Broker.finish() 2007-12-11 Brian Warner * foolscap/call.py (PendingRequest.fail): tolerate pass-then-fail, which is just as weird of an error case but shouldn't cause an exception. The fact that this is necessary indicates significant problems in the new connection-management code. Sigh. * foolscap/banana.py (Banana.handleError): loseConnection takes a Failure, not an exception. Addresses #36. * foolscap/negotiate.py (Negotiation.acceptDecisionVersion1): same * foolscap/test/test_call.py: same * foolscap/test/test_reconnector.py (Reconnector._got_ref): same * foolscap/broker.py (Broker.shutdown): assert that we get a Failure * foolscap/test/common.py (Loopback.loseConnection): same * foolscap/__init__.py: bump revision to 0.2.1+ while between releases * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2007-12-10 Brian Warner * foolscap/__init__.py: release 0.2.1 * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same * foolscap/broker.py (Broker.connectionTimedOut): sigh, brown paper bag bug: broker.shutdown() requires a Failure, not an exception. Unfortunately the unit tests didn't catch this. * foolscap/pb.py (Tub.stopService): same * foolscap/negotiate.py (Negotiation.evaluateNegotiationVersion1): same * foolscap/logging/interfaces.py: give fully-qualified __remote_name__ strings to all RemoteInterfaces, to avoid collision with other code that might use these names. 2007-12-10 Brian Warner * foolscap/__init__.py: release 0.2.0 * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2007-12-10 Brian Warner * setup.py: add the foolscap/logging package * NEWS: update for upcoming release * foolscap/negotiate.py (Negotiation.handle_old): finally implement this: if enabled, "old" brokers (older than 60 seconds) will be replaced by new offers, but "new" ones are not. The default timeout is 60 seconds, but you can set it to something else by calling tub.setOption("handle-old-duplicate-connections", 120) instead of using 'True'. Closes #34. * foolscap/pb.py (Tub.setOption): same * foolscap/broker.py (Broker.__init__): record creation timestamp * foolscap/test/test_negotiate.py (Replacement.testAncientClientWorkaround): test it 2007-12-10 Brian Warner * doc/logging.xhtml: updates * foolscap/reconnector.py (Reconnector.reset): add a utility method to manually force a Reconnector to reconnect, most useful from a manhole or other in-process eval loop. Closes #30. (Reconnector.getDelayUntilNextAttempt): another to query the delay until the next connection attempt will be made. Returns None if no attempt is currently scheduled. (Reconnector.getLastFailure): and another to provide the last failure, useful if you want to find out why it keeps reconnecting. * foolscap/negotiate.py (Negotiation.compareOfferAndExisting): if the seqnum is too old, accept the offer anyways: this handles the lost-a-decision-message case better at the expense of worse handling of offers-delivered-out-of-order case (which is far less likely to occur than a lost decision message) * foolscap/test/test_negotiate.py (Replacement): more tests (Replacement.testBouncedClient_Reverse): same * foolscap/logging/log.py (FoolscapLogger.msg): handle two kinds of argument formatting: log.msg("%d>%d", a, b), and log.msg("%(foo)s!", foo=something). Also handle the twisted_log equivalent of the dict case: log.msg(format="%(foo)s", foo=blah). * foolscap/test/test_logging.py (Basic.testLog): test it (Publish.test_logpublisher._got_logport._check_observer): same 2007-12-07 Brian Warner * foolscap/negotiate.py (Negotiation.compareOfferAndExisting): new duplicate-connection-handling, for ticket #28. Details in the ticket, and need to be put in the docs. Each end stores and sends more information to give the master a better chance of detecting race conditions correctly. The end result is that silently-lost TCP connections (due to NAT timeouts or laptops being yanked from one network to another) should not cause 35-minute reconnector delays any longer. Many many thanks to Rob Kinninmont and Zooko for their invaluable help in finding a good solution to this problem. Hopefully this closes #28. (Negotiation.handle_old): placeholder method for the code to handle <=0.1.7 clients, not yet written. (Negotiation.acceptDecisionVersion1): record information from the decision in the Broker, so we can compare against it later * foolscap/broker.py (Broker.__init__): same * foolscap/pb.py (Tub.setup): record an "Incarnation Record", just a unique (random) string, so that remote servers can tell if we ought to remember them or not. (Tub.setOption): add new 'handle-old-duplicate-connections' option, for the <=0.1.7-client-handler code (not yet written). This option name might change. * foolscap/test/test_negotiate.py (Replacement): test much of the new code, probably covers 60% of the code paths. Still need to test the opposite direction (client==master) to make sure I didn't break anything. * Makefile (.figleaf.el): copy more tools from Tahoe: convert figleaf coverage data into an emacs-lisp -parseable format * misc/testutils/figleaf.el: elisp code to highlight uncovered lines * misc/testutils/figleaf_htmlizer.py: emit lines *not* covered in a separate column, and sort by that: this makes it a lot easier to pay attention to the places that need work. Code copied over from Tahoe. 2007-12-06 Brian Warner * foolscap/broker.py (Broker.shutdown): refactor shutdown code, to make it safer to replace one connection with another. (Broker.abandonAllRequests): use an eventual-send for each flunking, to make sure the connection is completely gone by the time application code gets to run in the errbacks. * foolscap/pb.py (Tub.stopService): give a reason for the shutdown. Keep it a ConnectionDone (so we don't need to update other code that wouldn't expect a new exception type), but mention Tub.stopService in the arguments. (Tub.brokerAttached): add assert self.running.. this should never be called on a Tub that's already been shutdown. (Tub.brokerDetached): fix last-broker-detached detection, since the sequencing changed a bit. Make sure we don't fire the observerlist twice, by only firing it when we actually remove a broker and it was the last one (i.e. we started with some brokers, and finished with none). (Tub.__repr__): add the TubID to the repr * foolscap/call.py (FailureSlicer.getStateToCopy): truncate .value, .type, and .parents to the lengths defined in FailureConstraint. Unfortunately we don't seem to impose these constraints during serialization, so if some exception happens to have a really long name or arguments, it's the recipient of the CopiedFailure who will complain, making it hard to figure out where the real problem lies. The fact that the Violation doesn't seem to specify which attribute was in violation only adds confusion: it said ..??? for a 'token too large: 1819>1000', in RemoteCopyUnslicer.checkToken, which was probably f.value since that's the only 1000byte constraint. 2007-11-29 Brian Warner * foolscap/pb.py (Tub._log_gatherer_connected): for UnauthenticatedTubs, send tubid="" instead of tubid=None, since RILogGatherer.logport is expecting a string. Ideally I'd like this to be ChoiceOf(str,None), but ChoiceOf doesn't work. In the long run UnauthenticatedTubs will acquire normal (but distinct) tubids, so this issue will go away. * foolscap/logging/interfaces.py: minor cleanup * foolscap/test/test_logging.py (Publish._test_gatherer._check): update to match * foolscap/constraint.py: improve various Violation messages * foolscap/logging/log.py: get logport/gatherers working the way they're described in the docs * foolscap/logging/publish.py: merge into log.py * foolscap/logging/interfaces.py: improve imports * foolscap/pb.py: fix API for access to this stuff * foolscap/test/test_logging.py: fix up tests to match * doc/logging.xhtml: more cleanup 2007-11-28 Brian Warner * Makefile: misc new targets, parameterize $(TRIAL) * doc/logging.xhtml: document new logging scheme, some of this is still speculative, unimplemented, or just plain wrong * foolscap/logging/log.py: frontend for logging scheme * foolscap/pb.py: partial API support for logging * foolscap/test/test_logging.py: tests for new code, partially adapted from Petmail log tests 2007-11-27 Brian Warner * foolscap/logging/*: new logging support, including hierarchical logging, remote publishing, gathering tools * foolscap/test/test_logging.py: minimal tests for it * bin/flogtool: command-line tool to view/gather log events * setup.py (scripts=): include flogtool 2007-10-16 Brian Warner * foolscap/test/test_gifts.py (Bad.test_location._introduce): use 127.0.0.1:2 as the bogus port, instead of 127.0.0.47:1 . Connecting to .47 causes a long delay on OS-X (probably because of some sort of personal firewall), whereas .1 fails right away. Port 2 is not listed in my copy of /etc/services and seems unlikely to have a real service running on it. 2007-09-24 Brian Warner * doc/serializing.xhtml: oops, fix closing tags * foolscap/__init__.py: bump revision to 0.1.7+ while between releases * misc/*/debian/changelog: same 2007-09-24 Brian Warner * foolscap/__init__.py: release 0.1.7 * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2007-09-24 Brian Warner * setup.py: remove download_url, I think url= is sufficient. * NEWS: update for the upcoming release * all: remove default size limits on Constraints. Users who wish to enforce size limits should provide a maxLength= argument to their Constraint constructors. These limits turned out to be more surprising than helpful. Closes #26. * foolscap/constraint.py (everythingTaster): remove the default SIZE_LIMIT on STRING tokens. (ByteStringConstraint.__init__): and on StringConstraint * foolscap/slicers/unicode.py (UnicodeConstraint.__init__): and here * foolscap/slicers/list.py (ListConstraint.__init__): and here * foolscap/slicers/dict.py (DictConstraint.__init__): and here * foolscap/slicers/set.py (SetConstraint.__init__): and here * foolscap/test/test_schema.py (CreateTest.testMakeConstraint): update tests to match * foolscap/test/test_copyable.py (MyRemoteCopy4.stateSchema): same 2007-09-16 Brian Warner * foolscap/pb.py (Tub.debug_listBrokers): add a debugging method that tells you about all the connected brokers, and what methods are outstanding (both inbound and outbound) for each. If you see any method is sitting in one of these lists for a long time, there might be a problem in the execution of that method. * foolscap/broker.py (Broker.doNextCall): don't allow slow remote_foo methods to stall subsequent calls. This fixes a major message-delivery bug. Closes #25. * foolscap/test/test_call.py (TestCall.testStallOrdering): test it * all: merge in serialization-refactoring branch, adding Tub.(un)serialize (which can handle Referenceables) and moving us slightly closer to Sealers/Unsealers. The regular storage.serialize can now handle Copyables and objects for which you've registered an ISlicer adapter. Eventually the default unserialization interface will be 'safe' (meaning it won't create instances of arbitrary classes), and you'll have to give it additional arguments to enable 'unsafe' behavior. * doc/serializing.xhtml: document it * foolscap/banana.py: create slicer/unslicer at connectionMade, not __init__. Also change the way that errors are handled, and remove the use of 'types'. * foolscap/broker.py (PBRootUnslicer.open): refactor, move code to slicers.root.RootUnslicer (Broker.use_remote_broker): new attribute, True for connected brokers, False for the non-connected one that Tub.serialize uses. This is used by ReferenceableSlicer to decide whether to emit a my-reference sequence or a their-reference sequence. (StorageBroker): new subclass of Broker, uses new StorageBrokerRoot(un)Slicer. This broker is specialized to accept exactly one object, and hand it off to a waiting Deferred. There's still some useful refactoring to do, to mix in the behavior of ScopedRootUnslicer better. * foolscap/debug.py: delete this, having it around made the refactoring too difficult * foolscap/ipb.py (IBroker): use this to distinguish between a regular Banana instance and a Broker. The PB-specific slicers use it to assert that their .protocol is really a Broker. * foolscap/referenceable.py (ReferenceableSlicer): if the Referenceable is being sliced for storage, emit a their-reference sequence instead of a (useless) my-reference. Use giftID=0 to indicate that we don't want to do reference counting. (TheirReferenceUnslicer.ackGift): giftID=0 means don't ack * foolscap/slicers/root.py: RootSlicer refactoring (ScopedRootSlicer): new class for refactoring (ScopedRootUnslicer): same * foolscap/storage.py: same. Rewrite serialize/unserialize, they now return Deferreds and let you override the banana and root (un)slicer class to use, as well as the IO stream. * foolscap/test/test_banana.py: massive hacking to make it work. The storage classes changed a lot, requiring this cleanup. Also all banana/broker objects must have a transport and their connectionMade() method needs to be called before you can use them. * foolscap/test/test_pb.py: same * foolscap/test/test_serialize.py: test all the new functionality, including how serialized data cannot keep a Referenceable alive, and how you you can only serialize Referenceables with Tub.serialize(), not with foolscap.serialize(). 2007-09-11 Brian Warner * foolscap/call.py (PendingRequest.fail): improve logRemoteFailures by adding a source+dest TubID to the log message. Closes #23. (InboundDelivery.logFailure): same for logLocalFailures (CallUnslicer.receiveClose): tell the InboundDelivery about its broker * foolscap/pb.py (Tub.getTubID, .getShortTubID): utility methods (UnauthenticatedTub.getTubID, .getShortTubID): same * foolscap/referenceable.py (TubRef.getShortTubID): same (NoAuthTubRef.getShortTubID): same (RemoteReference._callRemote): tell the PendingRequest about the interface name it is using, so it can log a fully-qualified remote method name * foolscap/broker.py (Broker): tell the Broker about the remote TubRef it is attached to (Broker.doNextCall): make sure to log.err any problems that occur during callFailed, rather than discarding them. This catches errors in InboundDelivery.logFailure * foolscap/negotiate.py (Negotiation.switchToBanana): pass the remote TubRef to the newly-created Broker * foolscap/test/test_pb.py (TestCallable.testLogLocalFailure): test it (TestCallable.testLogLocalFailure): reduce timeout (TestCallable.testLogRemoteFailure._check): test it (TestCallable.testLogRemoteFailure): reduce timeout * foolscap/test/test_call.py (TestCall.testFailStringException): same (TestCall.testCopiedFailure): same * foolscap/test/test_gifts.py (Gifts.testGift): same * foolscap/test/common.py (TargetMixin.setupBrokers): provide tubref 2007-09-08 Brian Warner * foolscap/pb.py (Tub.registerReference): add furlFile= argument, to make it easy to persist unguessable FURLs in a file on disk. * foolscap/test/test_tub.py (FurlFile): test it * foolscap/tokens.py (WrongTubIdError): new exception for it * doc/using-foolscap.xhtml (Using a Persistent FURL): document it. Also replace all use of 'url' with 'furl'. * foolscap/__init__.py: bump revision to 0.1.6+ while between releases * misc/*/debian/changelog: same 2007-09-02 Brian Warner * foolscap/__init__.py: release 0.1.6 * misc/{dapper|edgy|feisty|sarge|sid}/debian/changelog: same * misc/etch: copy sid packaging for etch * Makefile (debian-etch): new target for etch .debs * setup.py: update download_url for 0.1.6 * MANIFEST.in: include etch files in the source release * NEWS: update for the upcoming release * Makefile (api-docs): new target to run 'epydoc' and generate API documentation. Addresses #16. (docs): fix the URLs of API doc targets to match * foolscap/negotiate.py: fix some docstrings * foolscap/pb.py: same * foolscap/promise.py: same 2007-08-31 Brian Warner * foolscap/schema.py (PolyConstraint.checkToken): we need to override this method as well as checkObject, otherwise inbound tokens may get rejected by the token-checking phase. robk noticed this with a ChoiceOf(StringConstraint(maxLength=3000, None)), which refused to accept a 2000 byte string (since the default Constraint.checkToken uses the everythingTaster which only accepts 1000-byte strings). Closes #13. * foolscap/test/test_call.py (TestCall.testChoiceOf): test it 2007-08-21 Brian Warner * foolscap/call.py (FailureSlicer.getStateToCopy): if we can't fit the whole traceback, elide the middle rather than truncate the end, since the end is usually the most interesting part. * foolscap/pb.py (Tub.setupEncryptionFile): don't use os.path.exists to decide if the certFile exists or not, just try to open it and deal with the exception if it happens. This avoids a race condition, not a big deal here but a good pattern to get in the habit of using everywhere. * foolscap/slicers/unicode.py (UnicodeConstraint.checkObject): improve error message 2007-08-17 Brian Warner * doc/using-foolscap.xhtml: fix more typos, as reported by arch_o_median. Closes #15. * .hgignore: ignore generated .html files in doc/ 2007-08-11 Brian Warner * doc/using-foolscap.xhtml: replace the phrase "private-key certificate" with the more accurate and more widely-used "public-key certificate". Thanks to Zooko for the patch. 2007-08-09 Brian Warner * doc/using-foolscap.xhtml: fix typos, thanks to David Ripton for the catch. * setup.py: add classifiers= and platforms=, update metadata, so that I can use 'setup.py register' for the next release. Closes #7. * README: update references to the home page. Closes #9. 2007-08-08 Brian Warner * misc/{sid|sarge|dapper|edgy|feisty}/debian/rules: fix references to renamed docs/ files. Closes #8. * foolscap/__init__.py: bump revision to 0.1.5+ while between releases * misc/{sid|sarge|dapper|edgy|feisty}/debian/changelog: same 2007-08-07 Brian Warner * foolscap/__init__.py: release Foolscap-0.1.5 * misc/{sid|sarge|dapper|edgy|feisty}/debian/changelog: same 2007-08-07 Brian Warner * NEWS: update for the upcoming release * foolscap/pb.py (Tub.registerNameLookupHandler): new function to augment Tub.registerReference(). This allows names to be looked up at request time, rather than requiring all Referenceables be pre-registered with registerReference(). The chief use of this would be for FURLs which point at objects that live on disk in some persistent state until they are needed. Closes #6. (Tub.unregisterNameLookupHandler): allow handlers to be removed (Tub.getReferenceForName): use the handler during lookup * foolscap/test/test_tub.py (NameLookup): test it 2007-07-27 Brian Warner * foolscap/referenceable.py (LocalReferenceable): implement an adapter that allows code to do IRemoteReference(t).callRemote(...) and have it work for both RemoteReferences and local Referenceables. You might want to do this if you're getting back introductions to a variety of remote Referenceables, some of which might actually be on your local system, and you want to treat all of the, the same way. Local Referenceables will be wrapped with a class that implements callRemote() and makes it behave like an actual remote callRemote() would. Closes ticket #1. * foolscap/test/test_reference.py (LocalReference): test it 2007-07-26 Brian Warner * foolscap/call.py (AnswerUnslicer.receiveChild): accept a ready_deferred, to accomodate Gifts in return values. Closes #5. (AnswerUnslicer.receiveClose): .. and don't fire the response until any such Gifts resolve * foolscap/test/test_gifts.py (Gifts.testReturn): test it (Gifts.testReturnInContainer): same (Bad.testReturn_swissnum): and test the failure case too * foolscap/test/test_pb.py (TestAnswer.testAccept1): fix a test which wasn't calling start() properly and was broken by that change (TestAnswer.testAccept2): same * foolscap/test/test_gifts.py (Bad.setUp): disable these tests when we don't have crypto, since TubIDs are not mangleable in the same way without crypto. * foolscap/slicer.py (BaseUnslicer.receiveChild): new convention: Unslicers should accumulate their children's ready_deferreds into an AsyncAND, and pass it to the parent. If something goes wrong, the ready_deferred should errback, which will abandon the method call that contains it. * foolscap/slicers/dict.py (DictUnslicer.receiveClose): same * foolscap/slicers/tuple.py (TupleUnslicer.receiveClose): same (TupleUnslicer.complete): same * foolscap/slicers/set.py (SetUnslicer.receiveClose): same * foolscap/slicers/list.py (ListUnslicer.receiveClose): same * foolscap/call.py (CallUnslicer.receiveClose): same * foolscap/referenceable.py (TheirReferenceUnslicer.receiveClose): use our ready_deferred to signal whether the gift resolves correctly or not. If it fails, errback ready_deferred (to prevent the message from being delivered without the resolved gift), but callback obj_deferred with a placeholder to avoid causing too much distress to the container. * foolscap/broker.py (PBRootUnslicer.receiveChild): accept ready_deferred in the InboundDelivery, stash both of them in the broker. (Broker.scheduleCall): rewrite inbound delivery handling: use a self._call_is_running flag to prevent concurrent deliveries, and wait for the ready_deferred before delivering the top-most message. If the ready_deferred errbacks, that gets routed to self.callFailed so the caller hears about the problem. This closes ticket #2. * foolscap/call.py (InboundDelivery): remove whenRunnable, relying upon the ready_deferred to let the Broker know when the message can be delivered. (ArgumentUnslicer): significant cleanup, using ready_deferred. Remove isReady and whenReady. * foolscap/test/test_gifts.py (Base): factor setup code out (Base.createCharacters): registerReference(tubname), for debugging (Bad): add a bunch of tests to make sure that gifts which fail to resolve (for various reasons) will inform the caller about the problem, via an errback on the original callRemote()'s Deferred. 2007-07-25 Brian Warner * foolscap/util.py (AsyncAND): new utility class, which is like DeferredList but is specifically for control flow rather than data flow. * foolscap/test/test_util.py: test it * foolscap/call.py (CopiedFailure.setCopyableState): set .type to a class that behaves (as least as far as reflect.qual() is concerned) just like the original exception class. This improves the behavior of derived Failure objects, as well as trial's handling of CopiedFailures that get handed to log.err(). CopiedFailures are now a bit more like actual Failures. See ticket #4 (http://foolscap.lothar.com/trac/ticket/4) for more details. (CopiedFailureSlicer): make sure that CopiedFailures can be serialized, so that A-calls-B-calls-C can return a failure all the way back. * foolscap/test/test_call.py (TestCall.testCopiedFailure): test it * foolscap/test/test_copyable.py: update to match, now we must compare reflect.qual(f.type) against some extension classname, rather than just f.type. * foolscap/test/test_pb.py: same * foolscap/test/common.py: same 2007-07-15 Brian Warner * foolscap/test/test_interfaces.py (TestInterface.testStack): don't look for a '/' in the stacktrace, since it won't be there under windows. Thanks to 'strank'. Closes Twisted#2731. 2007-06-29 Brian Warner * foolscap/__init__.py: bump revision to 0.1.4+ while between releases * misc/{sid|sarge|dapper|edgy|feisty}/debian/changelog: same 2007-05-14 Brian Warner * foolscap/__init__.py: release Foolscap-0.1.4 * misc/{sid|sarge|dapper|edgy|feisty}/debian/changelog: same, also remove a bunch of old between-release version numbers 2007-05-14 Brian Warner * NEWS: update for the upcoming release * doc/using-foolscap.xhtml: rename from doc/using-pb.xhtml * doc/using-pb.xhtml: replace all uses of 'PB URL' with 'FURL' * foolscap/pb.py (Tub.getReference): if getReference() is called before Tub.startService(), queue the request until startup. (Tub.connectTo): same for connectTo(). (Tub.startService): launch pending getReference() and connectTo() requests. There are all fired with eventual-sends. * foolscap/reconnector.py (Reconnector): don't automatically start the Reconnector in __init__, rather wait for the Tub to start it. * foolscap/test/test_tub.py (QueuedStartup): test it * doc/using-pb.xhtml: update docs to match * foolscap/test/test_call.py (TestCall.testCall1): replace an arbitrary delay with a polling loop, to make the test more reliable under load * foolscap/referenceable.py (SturdyRef.asLiveRef): remove a method that was never used, didn't work, and is of dubious utility anyways. (_AsLiveRef): remove this too * misc/testutils/figleaf.py (CodeTracer.start): remove leftover debug logging * foolscap/remoteinterface.py (RemoteInterfaceConstraint): accept gifts too: allow sending of RemoteReferences on the outbound side, and accept their-reference sequences on the inbound side. * foolscap/test/test_gifts.py (Gifts.test_constraint): test it * foolscap/test/test_schema.py (Interfaces.test_remotereference): update test, since now we allow RemoteReferences to be sent on the outbound side * foolscap/remoteinterface.py (getRemoteInterface): improve the error message reported when a Referenceable class implements multiple RemoteInterfaces * foolscap/remoteinterface.py (RemoteMethodSchema.initFromMethod): properly handle methods like 'def foo(nodefault)' that are missing *all* default values. Previously this resulted in an unhelpful exception (since typeList==None), now it gives a sensible InvalidRemoteInterface exception. * foolscap/test/test_schema.py (Arguments.test_bad_arguments): test it 2007-05-11 Brian Warner * foolscap/slicers/set.py (FrozenSetSlicer): finally acknowledge our dependence on python2.4 or newer, by using the built-in 'set' and 'frozenset' types by default. We'll serialize the old sets.Set and sets.ImmutableSet too, but they'll emerge as a set/frozenset. This will cause problems for code that was written to be compatible with python2.3 (by using sets.Set) and wasn't changed when moved to 2.4, if it tries to mingle sets.Set with the data coming out of Foolscap. Unfortunate, but doing it this way preserves both sanity and good behavior for modern 2.4-or-later apps. (SetUnslicer): fix handling of children that were unreferenceable during construction, fix handling of children that are not ready for use (i.e. gifts). (FrozenSetUnslicer): base this off of TupleUnslicer, since previously the cycle-handling logic was completely broken. I'm not entirely sure this is necessary, since I think the contents of sets must be transitively immutable (or at least transitively hashable), but it good to review and clean it up anyways. * foolscap/slicers/allslicers.py: match name change * foolscap/slicers/tuple.py (TupleUnslicer.receiveClose): fix handling of unready children (i.e. gifts), previously gifts inside containers were completely broken. * foolscap/slicers/list.py (ListUnslicer.receiveClose): same * foolscap/slicers/dict.py (DictUnslicer.receiveClose): same * foolscap/call.py: add debug log messages (disabled) * foolscap/referenceable.py (TheirReferenceUnslicer.receiveClose): gifts must declare themselves 'unready' until the RemoteReference resolves, since we might be inside a container of some sort. Without this fix, methods would be invoked too early, before the RemoteReference was really available. * foolscap/test/test_banana.py (ThereAndBackAgain.test_set): match new set/sets.Set behavior (ThereAndBackAgain.test_cycles_1): test some of the cycles (ThereAndBackAgain.test_cycles_3): add (disabled) test for checking cycles that involve sets. I think these tests are non-sensical, since sets can't really participate in the sorts of cycles we worry about, but I left the (disabled) test code in place in case it becomes useful again. * foolscap/test/test_gifts.py (Gifts.testContainers): validate that gifts can appear in all sorts of containers successfully. 2007-05-11 Brian Warner * foolscap/__init__.py: bump revision to 0.1.3+ while between releases * misc/{sid|sarge|dapper|edgy|feisty}/debian/changelog: same 2007-05-02 Brian Warner * foolscap/__init__.py: release Foolscap-0.1.3 * misc/{sid|sarge|dapper|edgy|feisty}/debian/changelog: same 2007-05-02 Brian Warner * MANIFEST.in: include some recently-added files to the source tarball * NEWS: update for the upcoming release * foolscap/reconnector.py (Reconnector._failed): simplify log/no-log logic * foolscap/slicers/unicode.py (UnicodeConstraint): add a new constraint that only accepts unicode objects. It isn't complete: I've forgotten how the innards of Constraints work, and as a result this one is too permissive: it will probably accept too many tokens over the wire before raising a Violation (although the post-receive just-before-the-method-is-called check should still be enforced, so application code shouldn't notice the issue). * foolscap/test/test_schema.py (ConformTest.testUnicode): test it (CreateTest.testMakeConstraint): check the typemap too * foolscap/test/test_call.py (TestCall.testMegaSchema): test in a call * foolscap/test/common.py: same * foolscap/constraint.py (ByteStringConstraint): rename StringConstraint to ByteStringConstraint, to more accurately describe its function. This constraint will *not* accept unicode objects. * foolscap/call.py, foolscap/copyable.py, foolscap/referenceable.py: * foolscap/slicers/vocab.py: same * foolscap/schema.py (AnyStringConstraint): add a new constraint to accept either bytestrings or unicode objects. I don't think it actually works yet, particularly when used inside containers. (constraintMap): map 'str' to ByteStringConstraint for now. Maybe someday it should be mapped to AnyStringConstraint, but not today. Map 'unicode' to UnicodeConstraint. * foolscap/pb.py (Tub.getReference): assert that the Tub is already running, either because someone called Tub.startService(), or because we've been attached (with tub.setServiceParent) to a running service. This requirement appeared with the connector-tracking code, and I hope to relax it at some point (such that any pre-startService getReferences will be queued and serviced when the Tub is finally started), but for this release it is a requirement to start the service before trying to use it. (Tub.connectTo): same * doc/using-pb.xhtml: document it * doc/listings/pb1client.py: update example to match * doc/listings/pb2client.py: update example to match * doc/listings/pb3client.py: update example to match * foolscap/pb.py (Tub.connectorFinished): if, for some reason, we're removing the same connector twice, log and ignore rather than explode. I can't find a code path that would allow this, but I *have* seen it occur in practice, and the results aren't pretty. Since the whole connection-tracking thing is really for the benefit of unit tests anyways (who want to know when Tub.stopService is done), I think it's more important to keep application code running. * foolscap/negotiate.py (TubConnector.shutdown): clear out self.remainingLocations too, in case it helps to shut things down faster. Add some comments. * foolscap/negotiate.py (Negotiation): improve error-message delivery, by keeping track of what state the receiver is in (i.e. whether we should send them an HTTP error block, an rfc822-style error-block, or a banana ERROR token). (Negotiation.switchToBanana): empty self.buffer, to make sure that any extra data is passed entirely to the new Banana protocol and none of it gets passed back to ourselves (Negotiation.dataReceived): same, only recurse if there's something still in self.buffer. In other situtations we recurse here because we might have somehow received data for two separate phases in a single packet. * foolscap/banana.py (Banana.sendError): rather than explode when trying to send an overly-long error message, just truncate it. 2007-04-30 Brian Warner * foolscap/broker.py (Broker.notifyOnDisconnect): if the RemoteReference is already dead, notify the callback right away. Previously we would never notify them, which was a problem. (Broker.dontNotifyOnDisconnect): be tolerant of attempts to unregister callbacks that have already fired. I think this makes it easier to write correct code, but on the other hand it loses the assertion feedback if somebody tries to unregister something that was never registered in the first place. * foolscap/test/test_call.py (TestCall.testNotifyOnDisconnect): test this new tolerance (TestCall.testNotifyOnDisconnect_unregister): same (TestCall.testNotifyOnDisconnect_already): test that a handler fires when the reference was already broken * foolscap/call.py (InboundDelivery.logFailure): don't use f.getTraceback() on string exceptions: twisted explodes (FailureSlicer.getStateToCopy): same * foolscap/test/test_call.py (TestCall.testFailStringException): skip the test on python2.5, since string exceptions are deprecated anyways and I don't want the warning message to clutter the test logs * doc/using-pb.xhtml (RemoteInterfaces): document the fact that the default name is *not* fully-qualified, necessitating the use of __remote_name__ to distinguish between foo.RIBar and baz.RIBar * foolscap/remoteinterface.py: same * foolscap/call.py (FailureSlicer.getStateToCopy): handle string exceptions without exploding, annoying as they are. * foolscap/test/test_call.py (TestCall.testFail4): test them 2007-04-27 Brian Warner * foolscap/broker.py (Broker.freeYourReference._ignore_loss): change the way we ignore DeadReferenceError and friends, since f.trap is not suitable for direct use as an errback * foolscap/referenceable.py (SturdyRef.__init__): log the repr of the unparseable FURL, rather than just the str, in case there are weird control characters in it * foolscap/banana.py (Banana.handleData): rewrite the typebyte scanning loop, to remove the redundant pos<64 check. Also, if we get an overlong prefix, log it so we can figure out what's going wrong. * foolscap/test/test_banana.py: update to match * foolscap/negotiate.py (Negotiation.dataReceived): if a non-NegotiationError exception occurs, log it, since it indicates a foolscap coding failure rather than some disagreement with the remote end. Log it with 'log.msg' for now, since some of the unit tests seem to trigger startTLS errors that flunk tests which should normally pass. I suspect some problems with error handling in twisted's TLS implementation, but I'll have to investigate it later. Eventually this will turn into a log.err. * foolscap/pb.py (Tub.keepaliveTimeout): set the default keepalive timer to 4 minutes. This means that at most 8 minutes will go by without any traffic at all, which should be a reasonable value to keep NAT table entries alive. PINGs are only sent if no other traffic was received, and they are only one byte long, so the traffic overhead should be minimal. Note that we are not turning on disconnectTimeout by default: if you want quietly broken connections to be disconnected before TCP notices a problem you'll need to do tub.setOption("disconnectTimeout", 10*60) or something. * foolscap/pb.py: remove an unused import * foolscap/pb.py (Tub.generateSwissnumber): always use os.urandom to generate the unguessable identifier. Previously we used either PyCrypto or fell back to the stdlib 'random' module (which of course isn't very random at all). I did it this way originally to provide compatibility with python2.3 (which lacked os.urandom): now that we require python2.4 or newer, os.urandom is a far better source (it uses /dev/random or equivalent). * doc/using-pb.xhtml: don't mention PyCrypto now that we aren't using it at all. * foolscap/negotiate.py (Negotiation.minVersion): bump both min and max version to '3', since we've added PING and PONG tokens that weren't present before. It would be feasible to accomodate v2 peers (by adding a Banana flag that refrains from ever sending PINGs), but there aren't enough 0.1.2 installations present to make this seem like a good idea just now. (Negotiation.maxVersion): same (Negotiation.evaluateNegotiationVersion3): same (Negotiation.acceptDecisionVersion3): same * foolscap/test/test_negotiate.py (Future): same * foolscap/banana.py (Banana): add keepalives and idle-disconnect. The first timeout value says that if we haven't received any data for this long, poke the other side by sending them a PING message. The other end is obligated to respond with a PONG (both PING and PONG are otherwise ignored). If we still haven't heard anything from them by the time the second timeout expires, we drop the connection. (Banana.dataReceived): if we're using keepalives, update the dataLastReceivedAt timestamp on every inbound message. (Banana.sendPING, sendPONG): new messages and handlers. Both are ignored, and serve only to update dataLastReceivedAt. * foolscap/tokens.py: add PING and PONG tokens * doc/specifications/banana.xhtml: document PING and PONG * foolscap/broker.py (Broker.__init__): add keepaliveTimeout and disconnectTimeout arguments. Both default to 'None' to disable keepalives and disconnects. * foolscap/negotiate.py (Negotiation.switchToBanana): copy timeouts from the Tub into the new Banana/Broker instance * foolscap/pb.py (Tub.setOption): accept 'keepaliveTimeout' and 'disconnectTimeout' options to enable this stuff. * foolscap/test/test_keepalive.py: test it * foolscap/pb.py (Tub.brokerClass): parameterize the kind of Broker that this Tub will create, to make certain unit tests easier to write (allowing them to substitute a custom Broker subclass). * foolscap/negotiate.py (Negotiation.brokerClass): same (Negotiation.initClient): capture the brokerClass here for clients (Negotiation.handlePLAINTEXTServer): and here for listeners (Negotiation.switchToBanana): use it 2007-04-26 Brian Warner * README (DEPENDENCIES, INSTALLATION): add docs 2007-04-16 Brian Warner * foolscap/remoteinterface.py (RemoteInterfaceConstraint.checkObject): string-format the object inside a tuple, to avoid an annoying logging failure when the object in question is actually a tuple * foolscap/test/test_gifts.py (ignoreConnectionDone): trap both ConnectionDone and ConnectionLost, since it appears that windows signals ConnectionLost. Hopefully this will make the unit tests pass under windows. * foolscap/banana.py (Banana.handleData): when the token prefix is too long, log and emit the repr of the prefix string, so somebody can figure out where it came from. * foolscap/test/test_banana.py (InboundByteStream.testString): update to match 2007-04-13 Brian Warner * foolscap/copyable.py (CopyableSlicer.slice): set self.streamable before yielding any tokens, otherwise contained elements that use streaming will trigger an exception. Many thanks to Ricky (iacovou-AT-gmail.com) for trying out advanced features of Foolscap and discovering this problem, I would never have stumbled over this one on my own. TODO: we still need unit tests to exercise this sort of thing on a regular basis. (Copyable2): same thing * foolscap/schema.py (_tupleConstraintMaker): redefine what tuples mean in constraint specifications. They used to indicate an alternative: (int,str) meant accept either an int *or* a string. Now tuples indicate actual tuples, so (int,str) means a 2-element tuple in which the first element is an int, and the second is a string. I don't know what I was thinking back then. If you really want to use alternatives, use schema.ChoiceOf instead. * foolscap/test/test_schema.py (CreateTest.testMakeConstraint): test that tuples mean tuples * foolscap/reconnector.py (Reconnector._failed): the old f.trap() could sometimes cause the reconnector to stop trying forever. Remove that. Thanks to Rob Kinninmont for finding the problem. Add new code to log the failure if f.check() indicates that it is a NegotiationError, since that's the sort of weird thing that users will probably want to see. * foolscap/test/test_reconnector.py: add lots of new tests * misc/testutils: add tools to do figleaf-based code-coverage checks while running unit tests. We have 89.2% coverage! Use 'make test-figleaf figleaf-output' to see the results. * Makefile: new targets for figleaf (test): enable 'make test TEST=foolscap.test.test_call' to work (test-figleaf): same * foolscap/__init__.py: bump revision to 0.1.2+ while between releases * misc/{sid|sarge|dapper|edgy|feisty}/debian/changelog: same 2007-04-04 Brian Warner * foolscap/__init__.py: release Foolscap-0.1.2 * misc/{sid|sarge|dapper|edgy|feisty}/debian/changelog: same * NEWS: update for new release 2007-04-04 Brian Warner * misc/feisty/debian/*: add debian packaging support for the Ubuntu 'feisty' distribution * Makefile: and a way to invoke it * misc/edgy/debian/*: same for the 'edgy' distribution * MANIFEST.in: include the edgy/feisty files in the source tarball * foolscap/test/test_call.py (TestCall.testMegaSchema): add a new test to exercise lots of constraint code * foolscap/test/common.py: support code for it * foolscap/slicers/set.py (SetUnslicer.setConstraint): fix bugs discovered as a result (SetConstraint.__init__): same * foolscap/__init__.py: bump revision to 0.1.1+ while between releases * misc/{sid|sarge|dapper}/debian/changelog: same 2007-04-03 Brian Warner * foolscap/__init__.py: release Foolscap-0.1.1 * misc/{sid|sarge|dapper}/debian/changelog: same * NEWS: update for new release 2007-04-03 Brian Warner * foolscap/negotiate.py (Negotiation): bump both minVersion and maxVersion to 2, indicating that this release is not compatible with 0.1.0, since the reqID=0 change will cause the first method call in either direction (probably a getYourReferenceByName) to never receive a response. The handler functions were rearranged a bit too. * foolscap/test/test_negotiate.py (Future): update to match * NEWS: get ready for release * foolscap/test/test_pb.py (TestCallable.testLogLocalFailure): validate that Tub.setOption("logLocalFailures") actually works (TestCallable.testLogRemoteFailure): same * foolscap/remoteinterface.py (UnconstrainedMethod): add a "constraint" that can be used to mark a method as accepting anything and returning anything. This might be useful if you have RemoteInterface for most of your application, but there are still one or two methods which should not enforce a schema of any sort. This mostly defeats the purpose of schemas in the first place, but offering UnconstrainedMethod means developers can make the schema-or-not decision differently for individual methods, rather than for a whole class at a time. * foolscap/constraint.py (IRemoteMethodConstraint): document the requirements on IRemoteMethodConstraint-providing classes, now that there are two of them. * foolscap/test/test_call.py (TestCall.testUnconstrainedMethod): test it * foolscap/test/common.py: add some support code for the test * foolscap/referenceable.py (RemoteReferenceTracker._handleRefLost): refrain from sending decref messages with count=0 * foolscap/negotiate.py (TubConnectorClientFactory.__repr__): include both the origin and the target of the factory * foolscap/test/*.py (tearDown): insure that all tests use the now-standard stopService+flushEventualQueue teardown procedure, to avoid trial complaints about leftover timers and selectables. * foolscap/test/test_pb.py (GoodEnoughTub): when crypto is not available, skip some tests that really require it. Modify others to not really require it. * foolscap/test/test_crypto.py: same * foolscap/test/test_gifts.py: same * foolscap/test/test_loopback.py: same * foolscap/test/test_negotiate.py: same * foolscap/test/test_*.py (localhost): replace all use of "localhost" with "127.0.0.1" to avoid invoking the address resolver, which sometimes leaves a cleanup timer running. I think the root problem is there's no clean way to interrupt a connection attempt which still in the address resolution phase. You can stop it, but there's no way to wait for the resolver's cleanup timer to finish, which is what we'd need to make Trial happy. tcp.BaseClient.resolveAddress does not keep a handle to the resolver, so failIfNotConnected cannot halt its timer. * foolscap/test/test_zz_resolve.py: removed this test * foolscap/crypto.py (_ssl): import SSL goo in a different way to appease pyflakes * all: fix some pyflakes warnings by checking for the importability of foolscap.crypto in a different way * foolscap/pb.py (Tub.stopService): when shutting down the Tub, make sure all outstanding connections are shut down as well. By the time stopService's deferred fires, all of our TCP transports should have had their 'connectionLost' methods fired. This is specifically to help unit tests that use Trial, which insists upon having a clean reactor between tests. With this change, test suites should use a tearDown() method that looks like: 'd = tub.stopService(); d.addCallback(flushEventualQueue); return d', and trial shouldn't complain about foolscap selectables or timers being left over. (Tub.stopService): also, since Tubs are not currently restartable, modify some entry points at shutdown to make sure nobody gets confused about why their getReference() doesn't work anymore. Be aware that at some point soon, we'll start enforcing the rule that the Tub must be started before you can get any connections out of it, at which point getReference() will queue requests until startService() is called. The idea is that the Tub will not use the network at all unless it is running. * foolscap/broker.py: drop the connection when shutdown() is called * foolscap/negotiate.py (Negotiate): rearrange error reporting and connection shutdown. Now errors are stashed and loseConnection() is called, but the errors are not reported to negotiationFailed() until connectionLost() is fired (which will be after any remaining data gets sent out over the wire). (TubConnector): the TubConnector reports success once the first connection has passed negotiation, but now lives until all of the connections are finally closed. It then informs the Tub that it is done, so the Tub can forget about it (and possibly notify stopService that it can finally complete). * foolscap/observer.py (OneShotObserverList): eventual-send -using event distributor, following the pattern espoused by Mark Miller's "Concurrency Among Strangers" paper. Many thanks to AllMyData.com for contributing this class. * foolscap/test/test_observer.py: tests for it 2007-03-22 Brian Warner * foolscap/constraint.py (StringConstraint): add a regexp= argument * foolscap/test/test_schema.py (ConformTest.testString): test it * foolscap/test/test_banana.py (TestBananaMixin.shouldDropConnection): fix a pyflakes warning * foolscap/call.py: same, don't fall back to plain StringIO if cStringIO is unavailable * foolscap/debug.py: same * foolscap/storage.py: same * foolscap/slicers/list.py (ListConstraint): add a minLength= argument, fix maxLength=None * foolscap/test/test_schema.py (ConformTest.testList): test it * foolscap/constraint.py (StringConstraint): add a minLength= argument * foolscap/test/test_schema.py (ConformTest.testString): test it * foolscap/slicers/set.py (BuiltinFrozenSetSlicer): add slicer for the builtin 'frozenset' type that appeared in python2.4 (SetConstraint): provide a constraint for sets * foolscap/schema.py (SetOf): add an alias * foolscap/test/test_schema.py (ConformTest.testSet): test it 2007-03-20 Brian Warner * foolscap/banana.py (Banana.outgoingVocabTableWasReplaced): remove verbose debug message, not really needed anymore * foolscap/ipb.py (IRemoteReference.callRemoteOnly): new method to invoke a remote method without waiting for a response. Useful for certain messages where we really don't care whether the far end receives them or not. * foolscap/referenceable.py (RemoteReference.callRemoteOnly): implement it (TheirReferenceUnslicer.ackGift): use it * foolscap/broker.py (Broker.initBroker): use reqID=0 to mean "we don't want a response". Note that this is a compatibility barrier: older endpoints which use reqID=0 for the first message will not get a response. All subsequent messages will be ok, though. (Broker._callFinished): don't send a response if reqID=0 (Broker.callFailed): don't send an error if reqID=0 * foolscap/call.py (InboundDelivery.logFailure): fix arg logging (CallUnslicer.receiveChild): don't create an activeLocalCalls entry if reqID=0 * foolscap/test/test_call.py (TestCallOnly.testCallOnly): test it (TestCall._testFailWrongReturnLocal_1): update expectations, now that reqIDs start at 1 instead of 0 * foolscap/test/common.py (TargetMixin.poll): new support code * foolscap/referenceable.py: add Referenceable to schema.constraintMap, so that RemoteInterfaces can use 'return Referenceable' to indicate that they return a Referenceable of any sort. This is like using 'return RIFoo' to indicate that the method returns a Referenceable that implements RIFoo, but without the specific interface requirement. * foolscap/remoteinterface.py (RemoteInterfaceConstraint): support this by skipping the interface check if self.interface=None * foolscap/test/test_schema.py (CreateTest): test it * foolscap/test/test_interfaces.py (Types): update test to match, since the error messages changed * foolscap/test/common.py: more test support changes 2007-03-19 Brian Warner * foolscap/ipb.py (IRemoteReference): new interface .. * foolscap/referenceable.py (RemoteReferenceOnly): .. implemented here * foolscap/remoteinterface.py (RemoteInterfaceConstraint.checkObject): remove a circular import by using IRemoteReference to detect RemoteReference instances, rather than using isinstance(). * foolscap/test/test_schema.py (Interfaces): test it * everything: massive Constraint refactoring. Primitive constraints (StringConstraint, IntegerConstraint, etc) are now in foolscap/constraint.py, while opentype-specific constraints like ListConstraint and BooleanConstraint are in the same module that defines the associated Slicer. Remote method constraints are in remoteinterface.py and copyable.py, FailureConstraint is in call.py . A new foolscap/constraint.py module contains the base classes but is careful not to import much else. foolscap/schema.py contains a reference to all constraints, so that user code can get at them conveniently. Tests were updated to import from the new places. Some circular imports were resolved. zope.interface adaptation has been used to assist with the conversion from the "shorthand" forms of constraint specification into the full form (i.e. converting x=str into x=StringConstraint()), specifically IConstraint(shorthand) will return a Constraint instance. * foolscap/__init__.py: bump revision to 0.1.0+ while between releases * misc/{sid|sarge|dapper}/debian/changelog: same 2007-03-15 Brian Warner * foolscap/__init__.py: release Foolscap-0.1.0 * misc/{sid|sarge|dapper}/debian/changelog: same * README: update for new release * NEWS: update for new release 2007-02-16 Brian Warner * foolscap/eventual.py (_SimpleCallQueue._turn): retire all pending eventual-send messages before returning control to the reactor, rather than doing exactly one event per reactor turn. This seems likely to help avoid starvation, as we now finish as much work as possible before accepting IO (which might cause more work to be added to our queue), and probably makes the interaction between eventual-send and DelayedCalls a bit more consistent. Thanks to Rob Kinninmont for the suggestion. 2007-02-08 Brian Warner * foolscap/test/test_pb.py: move TestCall out to.. * foolscap/test/test_call.py: .. a new test file * foolscap/test/test_negotiate.py (BaseMixin.tearDown): add a 100ms stall between shutting down all the Tubs and actually finishing the test. This seems to be enough to stop the occasional test failures that probably occur because TCP connections that we've dropped haven't finished signalling the other end (also in our process) that they've been closed. 2007-01-30 Brian Warner * foolscap/pb.py (Tub): add certFile= argument, to allow the Tub to manage its own certificates. This argument provides a filename where the Tub should read or write its certificate. If the file exists, the Tub will read the certificate data from there. If not, the Tub will generate a new certificate and write it to the file. * foolscap/test/test_tub.py: test it * doc/using-pb.xhtml: document certFile= * doc/listings/pb2server.py: use certFile= in the example 2007-01-24 Brian Warner * foolscap/crypto.py (MyOptions._makeContext.alwaysValidate): add code to ignore two additional OpenSSL certificate validation errors: X509_V_ERR_CERT_NOT_YET_VALID (9) and X509_V_ERR_CERT_HAS_EXPIRED (10). Foolscap uses certificates very differently than web sites, and it is exceedingly common to start using a cert mere seconds after creating it. If there is any significant clock skew between the two systems, then insisting that the cert's "valid after X" time is actually in the past will cause a lot of false errors. 2007-01-22 Brian Warner * .darcs-boringfile: ignore files that are generated by distutils when we make a source release (dist/*) and when making a debian package (build/* and the debian install directory). * foolscap/__init__.py: bump revision to 0.0.7+ while between releases * misc/{sid|sarge|dapper}/debian/changelog: same 2007-01-16 Brian Warner * foolscap/__init__.py: release Foolscap-0.0.7 * misc/{sid|sarge|dapper}/debian/changelog: same * NEWS: update for 0.0.7 2007-01-16 Brian Warner * foolscap/pb.py (Tub.getBrokerForTubRef): special-case an attempt to connect to a tub with an ID equal to our own, by attaching a Broker to a special LoopbackTransport that delivers serialized data directly to a peer without going through a socket. * foolscap/broker.py (LoopbackTransport): same (Broker.setTub): refactor some code out of negotiate.py * foolscap/negotiate.py (Negotiation.switchToBanana): same (Negotiation.loopbackDecision): new method to determine params for a loopback connection * foolscap/test/test_loopback.py: enable all tests, add a check to make sure we can connect to ourselves twice * foolscap/referenceable.py (RemoteReferenceTracker.getRef): the weakref this holds may have become stale, so check that we both have self.ref *and* that self.ref() is not None to decide whether we must re-create the RemoteReference. This fixes a bug in which two calls to Tub.getReference() for the same URL would result in the second call getting None. (RemoteReferenceTracker._handleRefLost): only send a decref message if we haven't already re-created the RemoteReference * foolscap/test/test_pb.py (TestService.testConnect3): modify this test to validate the 'call Tub.getReference() twice' bug is fixed 2007-01-15 Brian Warner * foolscap/test/test_loopback.py (ConnectToSelf): Add tests to validate that we can connect to our own Tub properly. This test does not yet pass for authenticated Tubs: the negotiation hangs until the 30 second timeout is reached. To fix this requires special-casing such connections to use a different kind of Broker, one that wires transport.write to eventual(rcvr.dataReceived) and skips negotiation completely. 2007-01-10 Brian Warner * doc/using-pb.xhtml: fix some "pb" references to mention "Foolscap" instead * doc/schema.xhtml: same 2007-01-09 Brian Warner * foolscap/pb.py (Listener.removeTub): disownServiceParent is not guaranteed to return a Deferred, so don't try to make removeTub do so either. I saw a failure related to this, but was unable to analzye it well enough to reproduce it or write a test case. (Tub.stopListeningOn): tolerate removeTub returning synchronously (Tub.stopService): same 2007-01-04 Brian Warner * foolscap/negotiate.py (Negotiation.dataReceived): when sending an error message to the far end inside the decision block, make sure the error text itself has no newlines, since that would break the format of the block, and probably cause all sorts of confusion. * foolscap/ipb.py (IRemotelyCallable.doRemoteCall): remote calls now accept positional args 2007-01-04 Brian Warner * foolscap/__init__.py: bump revision to 0.0.6+ while between releases * misc/{sid|sarge|dapper}/debian/changelog: same 2006-12-18 Brian Warner * foolscap/__init__.py: release Foolscap-0.0.6 * misc/{sid|sarge|dapper}/debian/changelog: same 2006-12-18 Brian Warner * misc/{sid|sarge|dapper}/debian/rules: include copyable.xhtml * NEWS: update for 0.0.6 * foolscap/negotiate.py (Negotiation): Send less data. When sending a range (both for the fundamental banana negotiation version and for the initial vocab table index), send it in a single line with "0 1" rather than two separate min and max lines. This brings the hello message down to about 105 bytes and improves the benefit of using a negotiated initial-vocab-table-range rather than an early (but post-negotiation) SET-VOCAB banana sequence. * foolscap/schema.py: add RemoteInterfaceConstraints. This works by declaring an argument as, e.g., RIFoo, which means that this argument must be passed a RemoteReference that is connected to a remote Referenceable which implements RIFoo. This works as a return value constraint too. (Constraint.checkObject): add inbound= argument to this method, so RemoteInterfaceConstraint can work properly (InterfaceConstraint): split this into local and remote forms (LocalInterfaceConstraint): only check real local objects, not RemoteReferences. This isn't really useful yet, but eventually schemas will turn into proper local Guards and then it will be. (RemoteInterfaceConstraint): only check RemoteReferences. The check performed must be different on inbound and outbound (since we'll see a RemoteReference when inbound=True, and a Referenceable when inbound=False). (makeConstraint): distinguish between Interfaces and RemoteInterfaces, so we can figure out whether to use LocalInterfaceConstraint or RemoteInterfaceConstraint (callable): get rid of this, the functionality has been absorbed into RemoteMethodSchema.initFromMethod * foolscap/broker.py (Broker._doCall): use inbound= argument (Broker._callFinished): same * foolscap/referenceable.py (RemoteReference._callRemote): same * foolscap/slicer.py (ReferenceUnslicer.receiveChild): same * foolscap/test/test_schema.py: same * foolscap/test/test_interfaces.py: rearrange, add tests for RemoteInterfaces. (LocalTypes): there are tests for Interfaces too, but without local Guards they're disabled for now. * foolscap/test/common.py: refactoring * foolscap/schema.py (makeConstraint): map None to Nothing(), which only accepts None. This is pretty handy for methods which are always supposed to return None. * foolscap/schema.py (RemoteMethodSchema.checkResults): don't annotate any Violations here.. leave that up to the caller * foolscap/broker.py (Broker._callFinished): update the annotation * foolscap/test/test_pb.py: update to match * foolscap/tokens.py (Violation.prependLocation): add new methods to Violations for easier annotation of where they occurred (Violation.appendLocation): same (Violation.__str__): remove the "at" from the location text * foolscap/test/test_pb.py: update to match * foolscap/broker.py (Broker._callFinished): if the outbound return value violates the schema, annotate the Violation to indicate the object and method that was responsible. 2006-12-15 Brian Warner * foolscap/call.py (CopiedFailure): clean up a bit, make it match the current Failure class better * doc/using-pb.xhtml: document positional arguments * foolscap/call.py: pass both positional and keyword arguments to remote methods. Previously only keyword arguments were accepted. This is a pretty far-reaching change, and introduces a compatibility barrier. (ArgumentSlicer): send both positional args and kwargs in a separate container (CallSlicer): move arg-sending out of CallSlicer (InboundDelivery.isRunnable): make InboundDelivery itself responsible for determining when it is runnable, instead of leaving that up to the CallUnslicer. The InboundDelivery is always referenceable, making the resulting object delivery simpler. (ArgumentUnslicer): move arg-receiving out of CallUnslicer. All of the schema-checking takes place here. Simplify the are-we-ready tests. (CallUnslicer): most of the code has moved out. The (call) sequence is now ('call', reqID, objID, methname, (args)). * foolscap/broker.py (Broker.scheduleCall): simplify, allow posargs * foolscap/referenceable.py (Referenceable.doRemoteCall): deliver posargs to the target method as well as kwargs (RemoteReference): same, stop trying to pre-map posargs into kwargs, no longer require a RemoteInterface to use posargs * foolscap/vocab.py (vocab_v1): add 'arguments' to the v1 vocab list. This is a compatibilty barriers, and changes like this are only allowed between releases. Once 0.0.6 is out we should leave the v1 list alone and make any additions to v2 instead. * foolscap/schema.py (RemoteMethodSchema): allow posargs, deal correctly with a mixture of posargs and kwargs * foolscap/test/test_schema.py (Arguments): test the RemoteMethodSchema class * foolscap/test/test_pb.py (TestCall.testCall1a): new tests of posargs and mixed posargs/kwargs (TestService.tearDown): use flushEventualQueue for cleanup * foolscap/test/test_interfaces.py (TestInterface): change a few things to match RemoteMethodSchema's new interfaces * foolscap/eventual.py (flushEventualQueue): allow this method to accept a single argument, which it ignores. This enables it to be used easily as a Deferred callback/errback, such as in a Trial tearDown method. The recommended usage is: d = clean_stuff(); d.addBoth(flushEventualQueue); return d 2006-12-11 Brian Warner * foolscap/vocab.py: add code to negotiate an initial set of words with which to pre-fill the VOCAB token list. Each side proposes a range and they use the highest common index (and they exchange a short hash of the list itself to guard against disagreements). This serve to compress the protocol traffic by maybe 50% over the longer run. * foolscap/negotiate.py: send the 'initial-vocab-table-min' and '-max' keys in the offer, and 'initial-vocab-table-index' in the decision (and in the Banana params) * foolscap/broker.py (Broker.__init__): populate the table * foolscap/banana.py (Banana.populateVocabTable): new method * foolscap/test/test_banana.py (Sliceable.testAdapter): todo items * foolscap/referenceable.py (RemoteReferenceOnly.notifyOnDisconnect): document this method. * foolscap/broker.py (Broker.shutdown): cancel all disconnect watchers upon shutdown * foolscap/pb.py (Tub.stopService): same * foolscap/negotiate.py (Negotiation.evaluateHello): if we spot an <=0.0.5 peer, mention that fact in our error message, to distinguish this case from some completely non-Foolscapish protocol trying to talk to us. 2006-12-10 Brian Warner * foolscap/negotiate.py (TubConnectorClientFactory.__repr__): annotate the string form to include which Tub we're connecting to. This makes the default factory's "BlahFactory Starting" log messages more interesting to look at. * foolscap/referenceable.py (TubRef.getTubID): support method (NoAuthTubRef.getTubID): same 2006-12-01 Brian Warner * foolscap/referenceable.py (RemoteReference.callRemote): use defer.maybeDeferred to rearrange and simplify. Clarify the comments about the various phases of commitment. * foolscap/call.py (AnswerUnslicer.checkToken): when re-raising an exception, use bareword 'raise' rather than explicitly re-raising the same exception instance with 'raise v'. Both forms get the right instance, but the latter loses the earlier stack trace. * foolscap/schema.py (RemoteMethodSchema.checkResults): same (RemoteMethodSchema.checkAllArgs): same * foolscap/referenceable.py (RemoteReference.callRemote): same * foolscap/test/test_interfaces.py (TestInterface.testStack): new test to verify that the Failure you get when you violate outbound method arguments actually includes the call to callRemote. * foolscap/schema.py (StringConstraint.checkObject): make the Violation message more useful (InterfaceConstraint.checkObject): same, by printing the repr() of the object that didn't meet the constraint. I'm not sure if this could be considered to leak sensitive information or not. (ClassConstraint.checkObject): same (RemoteMethodSchema.checkAllArgs): record which argument caused the problem in the Violation * foolscap/referenceable.py (RemoteReference.callRemote): add RemoteInterface and method name to the Violation when a caller violates their outbound constraint * foolscap/tokens.py (Violation.setLocation,getLocation): make it easier to modify an existing location value * foolscap/test/test_interfaces.py (TestInterface.testFail): verify that RemoteFailures pass a StringConstraint schema * foolscap/test/test_copyable.py: remove unused imports, from pyflakes * foolscap/test/test_pb.py: same * foolscap/test/test_reconnector.py: same * foolscap/test/test_registration.py: same * foolscap/test/test_interfaces.py: split the RemoteInterface tests out to a separate file * foolscap/test/test_pb.py: split them from here * foolscap/test/common.py: factor out some common utility classes 2006-11-30 Brian Warner * foolscap/negotiate.py (Negotiation.dataReceived): when sending an error block, set banana-decision-version to '1' so the recipient knows that it's safe to interpret the 'error' key. Thanks to Rob Kinninmont for the catch. 2006-11-27 Brian Warner * foolscap/negotiate.py (Negotiation._evaluateNegotiationVersion1): ignore extra keys in the offer, since a real v2 (and beyond) offer will have all sorts of extra keys. * foolscap/test/test_negotiate.py (NegotiationV2): test it by putting extra keys in the offer 2006-11-26 Brian Warner * foolscap/negotiate.py (Negotiation): change negotiation protocol: now each end sends a minVersion/maxVersion pair, using banana-negotiation-min-version and banana-negotiation-max-version, indicating that they can handle all versions between those numbers, inclusive. The deciding end finds the highest version number that fits in the ranges of both ends, and includes it in the banana-decision-version key of the decision block. This is an incompatible protocol change, but should make it easier (i.e. possible) to have compatible protocol changes in the future. Thanks to Zooko for suggesting this approach. (Negotiation.evaluateNegotiationVersion1): each negotiation version gets is own methods (Negotiation.acceptDecisionVersion1): same (TubConnectorClientFactory.buildProtocol): allow the Tub to make us use other Negotiation classes, for testing * foolscap/pb.py (Listener.__init__): same, use the class from the Tub that first caused the Listener to be created * foolscap/broker.py (Broker.__init__): record the banana-decision-version value, so tests can check it * foolscap/test/test_negotiate.py (Future): test it 2006-11-17 Brian Warner * foolscap/pb.py: remove unused and dodgy urlparse stuff * doc/using-pb.xhtml: move and expand the section on Copyable and other pass-by-copy things into a new file * doc/copyable.xhtml: new document. Thanks to Ricky Iacovou for the registerCopier examples. * doc/listings/copyable-{receive|send}.py: new examples * doc/stylesheet.css, doc/stylesheet-unprocessed.css * doc/template.tpl: docs utilities * Makefile: add 'make docs' target * foolscap/__init__.py: export registerCopier and registerRemoteCopyFactory * foolscap/copyable.py (Copyable): The new preferred Copyable usage is to have a class-level attribute named "typeToCopy" which holds the unique string. This must match the class-level "copytype" attribute of the corresponding RemoteCopy class. Copyable subclasses (or ICopyable adapters) may still implement getTypeToCopy(), but the default just returns self.typeToCopy . Most significantly, we no longer automatically use the fully-qualified classname: instead we *require* that the class definition include "typeToCopy". Feel free to use any stable and globally-unique string here. (RemoteCopyClass): Require that RemoteCopy subclasses set their "copytype" attribute, and use it for auto-registration. These subclasses can still use "copytype=None" to inhibit auto-registration. They no longer auto-register with the fully-qualified classname. * foolscap/referenceable.py (SturdyRef): match this change * foolscap/test/test_copyable.py: same 2006-11-16 Brian Warner * foolscap/negotiate.py (Negotiation.dataReceived): include the error message in the '500 Internal Server Error' string. (Negotiation.handlePLAINTEXTClient): include the full '500 Internal Server Error' string in the reported exception. These two changes make it easier to spot mismatched TubIDs. Thanks to Rob Kinninmont for the suggestion. 2006-11-14 Brian Warner * foolscap/__init__.py: bump revision to 0.0.5+ while between releases * misc/{sid|sarge|dapper}/debian/changelog: same 2006-11-04 Brian Warner * NEWS: update for 0.0.5 * foolscap/__init__.py: release Foolscap-0.0.5 * misc/{sid|sarge|dapper}/debian/changelog: same * MANIFEST.in: add debian packaging files to source tarball 2006-11-01 Brian Warner * foolscap/pb.py (Tub.setOption): new API to set options. Added logRemoteFailures and logLocalFailures, which cause failed callRemotes to be sent to the twisted log via log.msg . The defaults are False, which means that failures are only reported through the caller's Deferred.errback . Setting logRemoteFailures to True means that the client's log will contain a record of every callRemote that it sent to someone else that failed on the far side. This can be implemented on a per-callRemote basis by just doing d.addErrback(log.err) everywhere, but often there are reasons (like debugging) for logging failures that are completely independent of the desired error-handling path. These log messages have a REMOTE: prefix to make it very clear that the stack trace being shown is *not* occurring on the local system, but rather on some remote one. Setting logLocalFailures to True means that the server's log will contain a record of every callRemote that someone sent to it which failed on that server. This cannot be implemented with addErrbacks, since normally the server does not care about the methods it is running for other people's benefit. This option is purely for debugging purposes. These log messages have a LOCAL: prefix to make it clear that the stack trace is happening locally, but on behalf of some remote caller. * foolscap/call.py (PendingRequest.fail): improve the logging, make it conditional on logRemoteFailures, add the REMOTE: prefix (InboundDelivery): put more information into the InboundDelivery, move logLocalFailures logging into it (CallUnslicer.receiveClose): put the .runnable flag on the InboundDelivery object instead of on the CallUnslicer * foolscap/broker.py (Broker): pass the InboundDelivery around instead of the CallUnslicer that it points to. (Broker.callFailed): Add logLocalFailures checking here. * foolscap/reconnector.py: oops, add missing import that would break any actual reconnection attempts 2006-10-31 Brian Warner * misc/sarge/debian/control: add sarge packaging * misc/dapper/debian/control: update dependencies, add Recommends on pyopenssl * misc/sid/debian/control: same * Makefile: add 'debian-sarge' target * misc/dapper/debian: move debian packaging up a level * misc/sid/debian: same * Makefile: same * foolscap/__init__.py (__version__): bump to 0.0.4+ while between releases * misc/debs/sid/debian/changelog: same * misc/debs/dapper/debian/changelog: same 2006-10-26 Brian Warner * foolscap/__init__.py: release Foolscap-0.0.4 * misc/debs/sid/debian/changelog: same * misc/debs/dapper/debian/changelog: same 2006-10-26 Brian Warner * setup.py: fix project URL * MANIFEST.in: include misc/debs/* in the source tarball * NEWS: update for 0.0.4 * foolscap/test/test_reconnector.py: verify that the Reconnector's callbacks are properly interleaved with any notifyOnDisconnect callbacks the user might have registered. A Reconnector cb that uses notifyOnDisconnect should see a strictly-ordered sequence of connect, disconnect, connect, disconnect. 2006-10-25 Brian Warner * foolscap/referenceable.py (RemoteReferenceOnly.notifyOnDisconnect): accept args/kwargs to pass to the callback. Return a marker that can be passed to dontNotifyOnDisconnect() to de-register the callback. * foolscap/broker.py (Broker.notifyOnDisconnect): same (Broker.connectionLost): fire notifyOnDisconnect callbacks in a separate turn, using eventually(), so that problems or side-effects in one call cannot affect other calls or the connectionLost process * foolscap/test/test_pb.py (TestCall.testDisconnect4): test it * foolscap/pb.py (Tub.registerReference): undo that, make registerReference *always* create a strongref to the target, but split some of the work out to an internal function which makes the weakrefs. Tub.registerReference() is the API that application code uses to publish an object (make it reachable) *and* have the Tub keep it alive for you. I'm not sure I can think of a use case for making it reachable but *not* wanting the Tub to keep it alive. If you want to make it reachable but still ephemeral, just pass it over the wire. (Tub._assignName): new method to make weakrefs and assign names. (Tub.getOrCreateURLForReference): renamed from getURLForReference. Changed to assign a name if possible and one didn't already exist. BEHAVIOR CHANGE: This causes *all* objects passed over the wire, whether explicitly registered or just implicitly passed along, to be shareable as gifts (assuming the Tub is reachable and has a location, of course). * foolscap/referenceable.py (ReferenceableTracker.getURL): update * foolscap/test/test_registration.py (Registration.testWeak): use _assignName instead of registerReference * foolscap/test/test_gifts.py (Gifts.testOrdering): test it 2006-10-25 Brian Warner * foolscap/pb.py (Tub.registerReference): add a strong= argument which means the Tub should keep the registered object alive. If strong=False, the tub uses a weakref, so that when the application and all remote peers forget about the object, the Tub will too. strong= defaults to True to match the previous behavior, but this might change in the future, and/or it might become a property to be set on the Tub. * foolscap/test/test_registration.py: new tests for it * foolscap/test/test_pb.py (TestService.testStatic): disable this test, since static data (like tuples) are not weakreffable. The registration of static data is an outstanding issue. * foolscap/pb.py (Tub.connectTo): provide a new method, sets up a repeating connection to a given url (with randomized exponential backoff) that will keep firing a callback each time a new connection is made. This is the foolscap equivalent of ReconnectingClientFactory, and is the repeating form of getReference(). Thanks to AllMyData.com for sponsoring this work. * foolscap/reconnector.py (Reconnector): implement it here * foolscap/test/test_reconnector.py: test it * doc/using-pb.xhtml: update to reflect that we now have secure PBURLs and TubIDs, and that methods are delivered in-order (at least within a Tub-to-Tub connection) even in the face of gifts. * misc/debs/dapper/debian/rules (binary-indep): remove obsolete reference to the old python-twisted-pb2 package * foolscap/referenceable.py (YourReferenceSlicer.slice): assert that we actually have a URL to give out, since otherwise the error will confusingly show up on the far end (as a Violation). This occurs when we (as Alice) try to introduce Carol to a Bob that was not explicitly registered in Bob's Tub, such that Bob does not have a URL to give out. * foolscap/pb.py (Tub): tubID is no longer a parameter to Tub, since it is always computed from the certificate (UnauthenticatedTub): but it *is* a parameter here, since there is no certificate * foolscap/broker.py (Broker.getMyReferenceByCLID): relax the assertion to (int,long), since eventually clids will overrun a 31-bit integer. Thanks to Rob Kinninmont for the catch. (Broker.remote_decref): same 2006-10-10 Brian Warner * misc/debs: add some debian packaging, separate directories for sid and dapper because sid has pycentral and dapper is still in the versioned-python-package era * Makefile: simple Makefile to remind me how to create .debs 2006-10-05 Brian Warner * foolscap/__init__.py: bump to 0.0.3+ while between releases 2006-10-05 Brian Warner * foolscap/__init__.py: release Foolscap-0.0.3 * NEWS: update for 0.0.3 release 2006-10-05 Brian Warner * foolscap/test/test_gifts.py (Gifts): split out the Introduction tests from test_pb.py (Gifts.testOrdering): test the ordering of messages around a gift. Doing [send(1), send(2, carol), send(3)] should result in Bob seeing [1, (2,carol), 3] in that order. Before the recent ordering fix, the presence of the gift would delay message delivery, resulting in something like [1, 3, (2,carol)] * foolscap/test/test_pb.py: same * foolscap/call.py (CallUnslicer): fix ordering of message delivery in the face of Gifts. Each inbound method call gets unserialized into an InboundDelivery/CallUnslicer pair, which gets put on a queue. Messages get pulled off the queue in order, but only when the head of the queue is ready (i.e. all of its arguments are available, which means any pending Gifts have been retrieved). (InboundDelivery): same (CallUnslicer.describe): stop losing useful information * foolscap/broker.py (Broker.doNextCall): add inboundDeliveryQueue to implement all this * foolscap/test/test_pb.py (TestCall.testFailWrongArgsRemote1): match the change to CallUnslicer.describe * foolscap/referenceable.py (TheirReferenceUnslicer.receiveClose): don't bother returning ready_deferred, since we're returning an unreferenceable Deferred anyway. * foolscap/test/test_pb.py (Test3Way): put off the check that Alice's gift table is empty until we're sure she's received the 'decgift' message. Add a note about a race condition that we have to work around in a weird way to avoid spurious test failures until I implement sendOnly (aka callRemoteOnly). 2006-10-04 Brian Warner * foolscap/test/test_banana.py (ThereAndBackAgain.testIdentity): use an actual tuple. Obviously I wasn't thinking when I first wrote this and tried to use "(x)" to construct a one-item tuple. 2006-10-02 Brian Warner * everything: fix most of the pyflakes warnings. Some of the remaining ones are actual bugs where I need to finish implementing something. * foolscap/slicers/*.py: move most Slicers/Unslicers out to separate files, since slicer.py was way too big * foolscap/slicers/allslicers.py: new module to pull them all in. banana.py imports this to make sure all the auto-registration hooks get triggered. * everything: rearrange imports to match * setup.py: add new sub-package 2006-10-01 Brian Warner * foolscap/slicer.py: rearrange the internals, putting the corresponding Slicer and Unslicer for each type next to each other * foolscap/slicer.py: move all "unsafe" Slicers and Unslicers out to storage.py where it belongs * foolscap/storage.py: same * foolscap/test/test_banana.py: fix some imports to match * foolscap/test/test_pb.py: same * foolscap/slicer.py (ReplaceVocabSlicer): clean up VOCAB handling: add the ('add-vocab') sequence to incrementally add to the receiving end's incomingVocabulary table, fix the race condition that would have caused problems for strings that were serialized after the setOutgoingVocabulary() call was made but before the ('set-vocab') sequence was actually emitted. Lay the groundwork for adaptive tokenization and negotiated vocab table presets. Other classes involved are AddVocabSlicer, AddVocabUnslicer, and ReplaceVocabUnslicer. (BananaUnslicerRegistry): handle the add-vocab and set-vocab sequences with a registry rather than special-casing them. * foolscap/storage.py (UnsafeRootUnslicer): same, add the BananaUnslicerRegistry * foolscap/banana.py (setOutgoingVocabulary): make it safe to call this function at any time, as it merely schedules an update. Change the signature to accept a list of strings that should be tokenized rather than expecting the caller to choose the index values as well. (addToOutgoingVocabulary): new function to tokenize a single string, also safe to call at any time (outgoingVocabTableWasReplaced): (allocateEntryInOutgoingVocabTable): (outgoingVocabTableWasAmended): new functions for use by the Slicers that are sending the 'set-vocab' and 'add-vocab' sequences (Banana.maybeVocabizeString): reserve a place for adaptize tokenizing * foolscap/test/test_banana.py: match the changes * foolscap/broker.py: s/topRegistry/topRegistries/, since it is actually a list of Registries. Same for openRegistry and openRegistries * foolscap/slicer.py: same * foolscap/storage.py: same * foolscap/test/test_banana.py: same * foolscap/slicer.py (BuiltinSetSlicer): use a different test to look for python < 2.4, one which doesn't make pyflakes complain about using __builtins__ 2006-09-30 Brian Warner * foolscap/promise.py (Promise): implement a simpler syntax, at the encouragement of Zooko and others: now p.foo(args) does an eventual-send. This is a simpler form of send(p).foo(args) . Added _then and _except methods to do simple callback/errback handling. You can still do send() and sendOnly() on either immediate values or Promises: this shortcut only helps with send() on a Promise. You can still do when() on a Promise, which is more flexible because it returns a Deferred. The new syntax gives you a more dataflow-ish style of coding, which might be confusing in some ways but can also make the overall code much much easier to read. * foolscap/test/test_promise.py: update tests * foolscap/test/common.py (HelperTarget.remote_defer): replace callLater(0) with fireEventually() * foolscap/test/test_banana.py (ErrorfulSlicer.next): same (EncodeFailureTest.tearDown): use flushEventualQueue() for cleanup * foolscap/crypto.py (CertificateError): In Twisted >2.5, this exception is defined in twisted.internet.error, and it is sometimes raised by the SSL transport (in getPeerCertificate), and we need to catch it. In older versions, we define it ourselves even though it will never be raised, so that the code which catches it doesn't have to have weird conditionals. * foolscap/negotiate.py (Negotiation.handleENCRYPTED): catch the CertificateError exception (which indicates that we have an encrypted but unauthenticated connection: the other end did not supply a certificate). In older versions of twisted's SSL code, this was just indicated by having getPeerCertificate() return None. * foolscap/test/test_negotiate.py: re-enable all negotiation tests * foolscap/pb.py (UnauthenticatedTub): change the API and docs to refer to "Unauthenticated" tubs rather than "Unencrypted" ones, since that's really the choice you get to make. We use encrypted connections whenever possible; what you get to control is whether we use keys to provide secure identification and introduction. * foolscap/__init__.py: same, export UnauthenticatedTub instead of UnencryptedTub * foolscap/negotiate.py: same * foolscap/referenceable.py: same * foolscap/test/test_negotiate.py: same * doc/listings/pb1server.py: update examples to match * doc/using-pb.xhtml: same 2006-09-26 Brian Warner * foolscap/pb.py (Tub): rename PBService to Tub, make it always be encrypted (UnencryptedTub): new class for unencrypted tubs * all: fix everything else (code, docs, tests) to match * foolscap/ipb.py (ITub): new interface to mark a Tub 2006-09-24 Brian Warner * foolscap/referenceable.py (RemoteReferenceTracker._refLost): now that we have eventually(), use it to avoid the ugly bug-inducing indeterminacies that result from weakref callbacks being fired in the middle of other operations. * foolscap/promise.py (Promise._resolve): I think I figured out chained Promises. In the process, I made it illegal to call _break after the Promise has already been resolved. This also means that _resolve() can only be called once. We'll figure out breakable Far references later. * foolscap/test/test_promise.py (Chained): tests for them * foolscap/broker.py (Broker.getRemoteInterfaceByName): fix a bunch of typos caught by pyflakes. Left a couple of ones in there that I haven't figured out how to fix yet. * foolscap/slicer.py (InstanceUnslicer.receiveChild): same * foolscap/schema.py (RemoteMethodSchema.initFromMethod): same * foolscap/pb.py (Listener.addTub): same * foolscap/debug.py (TokenBanana.reportReceiveError): same * foolscap/copyable.py: same * foolscap/test/common.py: same * foolscap/test/test_pb.py (TestReferenceable.NOTtestRemoteRef1): same * foolscap/eventual.py: move eventual-send handling out to a separate file. This module now provides eventually(cb), d=fireEventually(), and d=flushEventualQueue() (for use by unit tests, not user code). * foolscap/negotiate.py: update to match * foolscap/test/common.py: same * foolscap/test/test_pb.py: same * foolscap/test/test_eventual.py: new tests for eventually() * foolscap/promise.py: rework Promise handling, now it behaves like I want it to (although chained Promises aren't working yet) * foolscap/test/test_promise.py: rework tests 2006-09-16 Brian Warner * foolscap/crypto.py: fall back to using our own sslverify.py if Twisted doesn't provide one (i.e. Twisted-2.4.x). * foolscap/sslverify.py: copy from the Divmod tree 2006-09-14 Brian Warner * foolscap/banana.py: remove #! line from non-script * foolscap/remoteinterface.py: same * foolscap/tokens.py: same * foolscap/test/test_schema.py: same * foolscap/__init__.py: bump to 0.0.2+ while between releases 2006-09-14 Brian Warner * foolscap/__init__.py: release Foolscap-0.0.2 2006-09-14 Brian Warner * doc/using-pb.xhtml: update pb3 example to match current usage, show an example of using encrypted Tubs * doc/listings/pb3calculator.py: same * doc/listings/pb3user.py: same * foolscap/__init__.py: rearrange the API: now 'import foolscap' is the preferred entry point, rather than 'from foolscap import pb'. * foolscap/pb.py: stop importing things just to make them available to people who import foolscap.pb * all: same, update docs, examples, tests * all: rename newpb to 'Foolscap' * setup.py: fix packages= to get tests too 2006-05-15 Brian Warner * test_zz_resolve.py: rename test file, I'd like to sit at the end of the tests rather than at the beginning. This is to investigate ticket #1390. * test_negotiate.py (Crossfire): oops, a cut-and-paste error resulted in two CrossfireReverse tests and zero Crossfire tests. Fixed this to enable the possibly-never-run real CrossfireReverse test case. (top): disable all negotiation tests unless NEWPB_TEST_NEGOTIATION is set in the environment, since they are sensitive to system load and the intermittent buildbot failures are annoying. 2006-05-05 Brian Warner * release-twisted: add 'pb' subproject * twisted/python/dist.py: same * twisted/pb/__init__.py: set version to 0.0.1 * twisted/pb/topfiles/setup.py: fix subproject name, set version 0.0.1 2006-04-29 Brian Warner * topfiles/README, topfiles/NEWS: prepare for 0.0.1 release * setup.py: fix up description, project name * test_ZZresolve.py: add some instrumentation to try and debug the occasional all-connection-related-tests-fail problem, which I suspect involves the threadpool being broken. 2006-02-28 Brian Warner * sslverify.py: update to latest version (r5075) from Vertex SVN, to fix a problem reported on OS-X with python2.4 . Removed the test-case-name tag to prevent troubles with buildbot on systems that don't also have vertex installed. I need to find a better solution for this in the long run: I don't want newpb to depend upon Vertex, but I also don't want to duplicate code. 2006-02-27 Brian Warner * debug.py (encodeTokens): return a Deferred rather than use deferredResult 2006-02-02 Brian Warner * test_negotiate.py: skip pb-vs-web tests when we don't have twisted.web, thanks to for the patch. 2006-01-26 Brian Warner * test/test_banana.py (ErrorfulSlicer.next): don't use callLater() with non-zero timeout * test/test_promise.py (TestPromise.test2): same * test/common.py (HelperTarget.remote_defer): same 2006-01-25 Brian Warner * copyable.py: refactor ICopyable and IRemoteCopy to make it possible to register adapters for third-party classes. (RemoteCopy): allow RemoteCopy to auto-register with the fully-qualified classname. This is only useful if you inherit from both pb.Copyable and pb.RemoteCopy at the same time, otherwise the sender and receiver will be using different names so they won't match up. * broker.py (PBRootUnslicer.open): now that registerRemoteCopy is done purely in terms of Unslicers, remove all the special-case code that handled IRemoteCopy (PBRootSlicer.slicerForObject): since zope.interface won't do transitive adaptation, manually handle the ThirdPartyClass -> ICopyable -> ISlicer case * test/test_copyable.py: clean up, improve comments (MyRemoteCopy3Unslicer): update to match new RemoteCopyUnslicer behavior. This needs to be documented and made easier. Also switch from pb.registerRemoteCopy to registerRemoteCopyUnslicerFactory, which is a mouthful. (Registration): split this out, update to match new debug tables in copyable.py (Adaptation): test ICopyable adapters 2006-01-23 Brian Warner * common.py: remove missing test_gift from the test-case-name tag, not sure how that got in there * test/test_copyable.py: split Copyable tests out of test_pb.py * test/common.py: factor out some more common test utility pieces * copyable.py: add suitable test-case-name tag * base32.py: rename Base32.py to base32.py, to match Twisted naming conventions * crypto.py: same * pb.py: same 2006-01-02 Brian Warner * negotiate.py (eventually): add glyph's eventual-send operator, based upon a queue cranked by callLater(0). (flushEventualQueue): provide a way to flush that queue, so tests know when to finish. * test/test_pb.py: switch to negotiate.eventually * test/__init__.py: add test-case-name tag 2005-12-31 Brian Warner * test_gift.py (TestOrderedGifts.testGift): verify that the presence of a gift (a third-party reference) in the arguments of a method does not cause that method to be run out-of-order. Marked TODO because at the moment they *are* run out-of-order. * common.py (RIHelper.append): new method * referenceable.py (TheirReferenceUnslicer.ackGift): ignore errors that involve losing the connection, since if these happen, the giver will decref the gift reference anyway. This removes some spurious log.errs and makes the unit tests happier. 2005-12-30 Brian Warner * test_negotiate.py (Versus.testVersusHTTPServerEncrypted): stall for a second after the test completes, to give the HTTP server a moment to tear down its socket. Otherwise trial flunks the test because of the lingering socket. I don't care for the arbitrary 1.0-second delay, but twisted.web doesn't give me any convenient way to wait for it to shut down. (this test was only failing under the gtk2 reactor, but I think this was an unlucky timing thing). (Versus.testVersusHTTPServerUnencrypted): same * negotiate.py (eventually): add an eventual-send operator (Negotiation.negotiationFailed): fire connector.negotiationFailed through eventually(), to give us a chance to loseConnection beforehand. This helps the unit tests clean up better. * negotiation.py (eventually): change the eventual-send operator to (ab)use reactor.callFromThread instead of callLater(0). exarkun warned me, but I didn't listen: callLater(0) does not guarantee relative ordering of sequentially-scheduled calls, and the windows reactors in fact execute them in random order. Obviously I'd like the reactor to provide a clearly-defined method for this purpose. * test_pb.py (eventually): same (Loopback.write): same. It was the reordering of these _write calls that was breaking the unit tests on windows so badly. (Loopback.loseConnection): same 2005-12-29 Brian Warner * test_pb.py (Loopback): fix plan-coordination bug by deferring all write() and loseConnection() calls until the next reactor turn, using reactor.callLater(0) as an 'eventual send' operator. This avoids an infinite-mutual-recursion hang that confuses certain test failures. Tests which use this Loopback must call flush() and wait on the returned Deferred before finishing. (TargetMixin): do proper setup/teardown of Loopback (TestCall.testDisconnect2): use proper CONNECTION_LOST exception (TestCall.testDisconnect3): same (TestReferenceable.testArgs1): rename some tests (TestReferenceable.testArgs2): test sending shared objects in multiple arguments of a single method call (TestReferenceable.testAnswer1): test shared objects in the return value of a method call (TestReferenceable.testAnswer2): another test for return values * call.py (CallUnslicer): inherit from ScopedUnslicer, so arguments that reference shared objects will accurately reproduce the object graph (AnswerUnslicer): same, for answers that have shared objects (ErrorUnslicer): same, just in case serialized Failures do too * slicer.py (ImmutableSetSlicer): set trackReferences=False, since immutable objects are never shared, so don't require reference tracking * banana.py (Banana.sendError): do loseConnection() in sendError rather than inside dataReceived. 2005-12-26 Brian Warner * slicer.py (ScopedSlicer.registerReference): track references with a (obj,refid) pair instead of just refid. This insures that the object being tracked stays alive until the scope is retired, preventing some ugly bugs that result from dead object id() values being reused. These bugs would only happen if the object graph changes during serialization (which you aren't supposed to do), but this is a cheap fix that limits the damage that could happen. In particular, it should fix a test failure on the OS-X buildslave that results from a unit test that is violating this object-graph -shouldn't-change prohibition. * banana.py (StorageBanana): refactor storage-related things, moving them from banana.py and slicer.py into the new storage.py . This includes UnsafeRootSlicer, StorageRootSlicer, UnsafeRootUnslicer, and StorageRootUnslicer. Also provide a simple serialize()/unserialize() pair in twisted.pb.storage, which will be the primary interface for simple pickle.dumps()-like serialization. 2005-12-24 Brian Warner * slicer.py: remove #!, add test-case-name (SetSlicer): define this unconditionally, now that python2.2 is no longer supported. (BuiltinSetSlicer): just like SetSlicer, used when there is a builtin 'set' type (python2.4 and higher) (ImmutableSetSlicer): define this unconditionally (SetUnslicer): same (ImmutableSetUnslicer): same * test_banana.py (TestBananaMixin.looptest): make it easier to test roundtrip encode/decode pairs that don't *quite* re-create the original object (TestBananaMixin.loop): clear the token stream for each test (ThereAndBackAgain.test_set): verify that python2.4's builtin 'set' type is serialized as a sets.Set * all: drop python2.2 compatibility, now that Twisted no longer supports it 2005-12-22 Brian Warner * pb.py (Listener.getPortnum): more python2.2 fixes, str in str (PBService.__init__): same, bool issues * test/test_banana.py: same, use failUnlessSubstring * test/test_negotiate.py: same * test/test_pb.py: same * negotiate.py: same, str in str stuff * broker.py: don't import itertools, for python2.2 compatibility * sslverify.py: same 2005-12-20 Brian Warner * test/test_banana.py: remove all remaining uses of deferredResult/deferredError * test/test_pb.py: same 2005-12-09 Brian Warner * pb.py (PBService.__init__): switch to SHA-1 for TubID digests * negotiate.py (Negotiation.evaluateHello): same * crypto.py (digest32): same 2005-12-08 Brian Warner * pb.py (PBService): allow all Tubs to share the same RandomPool 2005-10-10 Brian Warner * lots: overhaul negotiation, add lots of new tests. Implement shared Listeners, correct handling of both encrypted and non-encrypted Tubs, follow multiple locationHints correctly. More docs, update encrypted-tub examples to match new usage. 2005-09-15 Brian Warner * test_pb.py: remove some uses of deferredResult/deferredError 2005-09-14 Brian Warner * pb.py (PBService.generateSwissnumber): use PyCrypto RNG if available, otherwise use the stdlib 'random' module. Create a 160-bit swissnumber by default, this can be changed by the NAMEBITS class attribute. (PBService.__init__): use a random 32-bit number as a TubID when we aren't using crypto and an SSL certificate * Base32.py: copy module from the waterken.org Web-Calculus python implementation * test/test_crypto.py (TestService.getRef): let it register a random swissnumber instead of a well-known name * crypto.py: Implement encrypted PB connections, so PB-URLs are closer to being secure capabilities. This file contains utility functions. * sslverify.py: some pyOpenSSL wrappers, copied from Divmod's Vertex/vertex/sslverify.py * test/test_crypto.py: test case for encrypted connections * pb.py (PBServerFactory.buildProtocol): accomodate missing tubID, this needs to be re-thought when I do the "what if we aren't using crypto" pass. (PBServerFactory.clientConnectionMade): get the remote_tubid from a .theirTubID attribute, not the negotiated connection parameters, which won't include tub IDs anyway) (PBClientFactory.buildProtocol): if we're using crypto, tell the other side we want an encrypted connection (PBService.__init__): add useCrypto= parameter, currently defaults to False. This should switch to =True soon. (PBService.createCertificate): if useCrypto=True, create an SSL certificate for the Tub. * ipb.py (DeadReferenceError): actually define it somewhere * broker.py (Broker.handleNegotiation_v1): cleanup, make the different negotiation-parameter dictionaries distinct, track the ['my-tub-id'] field of each end more carefully. Start a TLS session when both ends want it. (Broker.startTLS): method to actually start the TLS session. This is called on both sides (client and server), the t.i.ssl subclasses figure out which is which and inform SSL appropriately. (Broker.acceptNegotiation): Make a PB-specific form. Start TLS if the server tells us to. When the second (encrypted) negotiation block arrives, verify that the TubID we're looking for matches both what they claim and what their SSL certificate contains. (Broker.freeYourReference): ignore DeadReferenceErrors too * banana.py (Banana.__init__): each instance must have its own copy of self.negotiationOffer, rather than setting it at the class level (Banana.negotiationDataReceived): let both handleNegotiation() and acceptNegotiation() return a 'done' flag, if False then the negotiation is re-started (Banana.handleNegotiation): make handleNegotiation_v1 responsible for setting self.negotiationResults (Banana.handleNegotiation_v1): same (Banana.acceptNegotiation): same 2005-09-09 Brian Warner * broker.py: big sanity-cleanup of RemoteInterface usage. Only allow a single RemoteInterface on any given pb.Referenceable. Tub.getReference() now only takes a string-form method name, so the rr.callRemote(RIFoo['bar'], *args) form is gone, and the one RemoteInterface associated with the RemoteReference (is available) will be checked. Tub.getReference() no longer takes an interface name: you request an object, and then later find out what it implements (rather than specifying your expectations ahead of time). Gifts (i.e. 'their-reference' sequences) no longer have an interfacename.. that is left up to the actual owner of the reference, who will provide it in the 'my-reference' sequence. * call.py, pb.py, referenceable.py, remoteinterface.py: same * test/test_pb.py: update to match, still needs some cleanup 2005-09-08 Brian Warner * setup.py, twisted/pb/topfiles: add "PB" sub-project * banana.py (Banana.sendFailed): oops, loseConnection() doesn't take an argument * copyable.py (RemoteCopyClass): make it possible to disable auto-registration of RemoteCopy classes * test/test_pb.py (TestCopyable.testRegistration): test it * referenceable.py (CallableSlicer): make it possible to publish callables (bound methods in particular) as secure capabilities. They are handled very much like pb.Referenceable, but with a negative CLID number and a slightly different callRemote() codepath. * broker.py (Broker.getTrackerForMyCall): same (Broker.getTrackerForYourReference): same, use a RemoteMethodReferenceTracker for negative CLID values (Broker.doCall): callables are distinguished by having a methodname of 'None', and are dispatched differently * call.py (CallUnslicer.checkToken): accept INT/NEG for the object ID (the CLID), but not string (leftover from old scheme) (CallUnslicer.receiveChild): handle negative CLIDs specially * test/test_pb.py (TestCallable): tests for it all (TestService.getRef): refactor (TestService.testStatic): verify that we can register static data too, at least stuff that can be hashed. We need to decide whether it would be useful to publish non-hashable static data too. 2005-09-05 Brian Warner * pb.py (PBService): move to using tubIDs as the primary identity key for a Tub, replacing the baseURL with a .location attribute. Look up references by name instead of by URL, and start using SturdyRefs locally instead of URLs whenever possible. (PBService.getReference): accept either a SturdyRef or a URL (RemoteTub.__init__): take a list of locationHints instead of a single location. The try-all-of-them code is not yet written, nor is the optional redirect-following. (RemoteTub.getReference): change the inter-Tub protocol to pass a name over the wire instead of a full URL. The Broker is already connected to a specific Tub (multiple Tubs sharing the same port will require separate Brokers), and by this point the location hints have already served their purpose, so the name is the only appropriate thing left to send. * broker.py (RIBroker.getReferenceByName): match that change to the inter-Tub protocol: pass name over the wire, not URL (Broker.getYourReferenceByName): same (Broker.remote_getReferenceByName): same * referenceable.py (RemoteReferenceOnly): replace getURL with getSturdyRef, since the SturdyRef can be stringified into a URL if necessary (SturdyRef): new class. When these are sent over the wire, they appear at the far end as an identical SturdyRef; if you want them to appear as a live reference, send sr.asLiveRef() instead. * test/test_pb.py (TestService.testRegister): match changes (Test3Way.setUp): same (HelperTarget.__init__): add some debugging annotations * test/test_sturdyref.py: new test * doc/pb/using-pb.xhtml: update to match new usage, explain PB URLs and secure identifiers * doc/pb/listings/pb1server.py: same * doc/pb/listings/pb1client.py: same * doc/pb/listings/pb2calculator.py: same * doc/pb/listings/pb2user.py: same 2005-05-12 Brian Warner * doc/pb/using-pb.xhtml: document RemoteInterface, Constraints, most of Copyable (still need examples), Introductions (third-party references). * doc/pb/listings/pb2calculator.py, pb2user.py: demostrate bidirectional references, using service.Application 2005-05-10 Brian Warner * broker.py (Broker.freeYourReference): also ignore ConnectionLost errors * doc/pb/listings/pb1client.py, pb1server.py: use reactor.run() * doc/pb/using-pb.xhtml: add shell output for examples * doc/pb/using-pb.xhtml: started writing usage docs * banana.py (Banana.dataReceived): add .connectionAbandoned, don't accept inbound data if it has been set. I don't trust .loseConnection to work right away, and sending multiple negotiation error messages is bad. (Banana.negotiationDataReceived): split out negotiation stuff to a separate method. Improve failure-reporting code to make sure we either report a problem with a negotation block, or with an ERROR token, not both, and not with multiple ERROR tokens. Catch errors in the upper-level bananaVersionNegotiated() call. Make sure we only send a response if we're the server. Report negotiation errors with NegotiationError, not BananaError. (Banana.reportReceiveError): rearrange a bit, accept a Failure object. Don't do transport.loseConnection here, do it in whatever calls reportReceiveError * debug.py (TokenBanana.reportReceiveError): match signature change (TokenStorageBanana.reportReceiveError): same * test/test_banana.py: match changes * tokens.py (NegotiationError): new exception * broker.py (Broker.handleNegotiation_v1): use the negotiation block to exchange TubIDs. (Broker.connectionFailed): tell the factory if negotiation failed (Broker.freeYourReference): ignore lost-connection errors, call freeYourReferenceTracker even if the connection was lost, since in that case the reference has gone away anyway. (Broker.freeYourReferenceTracker): don't explode if the keys were already deleted, since .connectionLost will clear everything before the decref-ack mechanism gets a chance to delete them. * referenceable.py (RemoteReferenceTracker.__repr__): stringify these with more useful information. * pb.py (PBServerFactory.buildProtocol): copy .debugBanana flag into the new Broker (both .debugSend and .debugReceive) (PBServerFactory.clientConnectionMade): survive a missing TubID (PBClientFactory.negotiationFailed): notify all onConnect watchers 2005-05-08 Brian Warner * test_pb.py (TestService): test the use of PBService without RemoteInterfaces too 2005-05-04 Brian Warner * broker.py (Broker): add tables to track gifts (third-party references) (PBOpenRegistry): add their-reference entry (RIBroker.decgift): new method to release pending gifts * call.py (PendingRequest): add some debugging hints (CallUnslicer): accept deferred arguments, don't invoke the method until all arguments are available * pb.py (PBService.listenOn): return the Service, for testing (PBService.generateUnguessableName): at least make them unique, if not actually unguessable (top): remove old URL code, all is now PBService * referenceable.py (RemoteReferenceOnly.__repr__): include the URL, if available (RemoteReference.callRemote): set .methodName on the PendingRequest, to make debugging easier (YourReferenceSlicer.slice): handle third-party references (TheirReferenceUnslicer): accept third-party references * schema.py (Nothing): a constraint which only accepts None * test/test_pb.py (Test3Way): validate third-party reference gifts 2005-04-28 Brian Warner * tokens.py (IReferenceable): move to flavors.py * flavors.py (IReferenceable): add it, mark Referenceable as implementing it. * pb.py (PBServerFactory): make root= optional (PBService): new class. In the future, all PB uses will go through this service, rather than using factories and connectTCPs directly. The service uses urlparse to map PB URLs to target hosts. * test_pb.py (TestService): start adding tests for PBService 2005-04-26 Brian Warner * banana.py: add preliminary newpb connection negotiation * test_banana.py: start on tests for negotiation, at least verify that newpb-newpb works, and that newpb-http and http-newpb fail. 2005-04-16 Brian Warner * banana.py (Banana.handleData): handle -2**31 properly * test_banana.py (ThereAndBackAgain.test_bigint): test it properly * flavors.py: python2.2 compatibility: __future__.generators * pb.py: same * schema.py (TupleConstraint.maxSize): don't use sum() (AttributeDictConstraint.maxSize): same (makeConstraint): in 2.2, 'bool' is a function, not a type, and there is no types.BooleanType * slicer.py: __future__.generators, and the 'sets' module might not be available (SetSlicer): only define it if 'sets' is available (SetUnslicer): same * test_banana.py: __future__.generators, 'sets' might not exist, (EncodeFailureTest.failUnlessIn): 2.2 can't do 'str in str', only 'char in str', so use str.find() instead (InboundByteStream2.testConstrainedBool): skip bool constraints unless we have a real BooleanType (ThereAndBackAgain.test_set): skip sets unless they're supported * test_schema.py (ConformTest.testBool): skip on 2.2 (CreateTest.testMakeConstraint): same * test_pb.py: __future__.generators, use str.find() * test_banana.py (DecodeTest.test_ref2): accomodate python2.4, which doesn't try to be quite as clever as python2.3 when comparing complex object graphs with == (DecodeTest.test_ref5): same. Do the comparison by hand. (DecodeTest.test_ref6): same, big gnarly validation phase * test_pb.py (TestReferenceUnslicer.testNoInterfaces): update to new signature for receiveClose() (TestReferenceUnslicer.testInterfaces): same (TestCall.testFail1): deferredError doesn't seem to like CopiedFailure all that much. Use retrial's return-a-deferred support instead. (MyRemoteCopy3Unslicer.receiveClose): same (TestCall.testFail2): same (TestCall.testFail3): same (TestFactory): clean up both server and client sockets, to avoid the "unclean reactor" warning from trial (Test3Way.tearDown): clean up client sockets * tokens.py (receiveClose): fix documentation * pb.py (CopiedFailure): make CopiedFailure old-style, since you can't raise new-style instances as exceptions, and CopiedFailure may have its .trap() method invoked, which does 'raise self'. (CopiedFailure.__str__): make it clear that this is a CopiedFailure, not a normal Failure. (callRemoteURL_TCP): Add a _gotReferenceCallback argument, to allow test cases to clean up their client connections. * flavors.py (RemoteCopyOldStyle): add an old-style base class, so CopiedFailure can be old-style. Make RemoteCopy a new-style derivative. * test_banana.py (DecodeTest.test_instance): fix the manually-constructed class names to reflect their new location in the tree (test_banana to twisted.pb.test.test_banana) (EncodeFailureTest.test_instance_unsafe): same * twisted/pb/*: move newpb from Sandbox/warner into the 'newpb' branch, distributed out in twisted/pb/ and doc/pb/ * twisted/pb: add __init__.py files to make it a real module * twisted/pb/test/test_*.py: fix up import statements 2005-03-22 Brian Warner * flavors.py: implement new signature * pb.py: same * test_pb.py: same * test_banana.py (BrokenDictUnslicer.receiveClose): new signature (ErrorfulUnslicer.receiveChild): same (ErrorfulUnslicer.receiveClose): same (FailingUnslicer.receiveChild): same * slicer.py: implement new receiveChild/receiveClose signature. Require that ready_deferred == None for now. (ListUnslicer.receiveChild): put "placeholder" in the list instead of the Deferred (TupleUnslicer.start): change the way we keep track of not-yet-constructable tuples, using a counter of unreferenceable children instead of counting the Deferred placeholders in the list (TupleUnslicer.receiveChild): put "placeholder" in the list instead of the Deferred * banana.py (Banana.reportReceiveError): when debugging, log the exception in a way that doesn't cause trial to think the test failed. (Banana.handleToken): implement new receiveChild signature (Banana.handleClose): same * debug.py (LoggingBananaMixin.handleToken): same * tokens.py (IUnslicer.receiveChild): new signature for receiveClose and receiveChild, they now pass a pair of (obj, ready_deferred), where obj is still object-or-deferred, but ready_deferred is non-None when the object will not be ready to use until some other event takes place (like a "slow" global reference is established). # Local Variables: # add-log-time-format: add-log-iso8601-time-string # End: foolscap-0.13.1/doc/0000755000076500000240000000000013204747603014576 5ustar warnerstaff00000000000000foolscap-0.13.1/doc/connection-handlers.rst0000644000076500000240000003200413204160675021262 0ustar warnerstaff00000000000000Connection Handlers =================== Each FURL contains 0 or more "connection hints", each of which tells the client Tub how it might connect to the target Tub. Each hint has a "type", and the usual value is "tcp". For example, "tcp:example.org:12345" means that the client should make a TCP connection to "example.org", port 12345, and then start a TLS-based Foolscap connection. Plugins can be used to enable alternate transport mechanisms. For example, a Foolscap Tub which lives behind a Tor "hidden service" might advertise a connection hint of "tor:abc123.onion:80". This hint is not usable by an unenhanced Foolscap application, since onion services cannot be reached through normal TCP connections. But applications which have installed a "Connection Handler" that recognizes the "tor:" hint type can make these connections. A Tor-capable handler would probably replace the usual TCP connection with one to a locally-configured Tor daemon's SOCKS proxy. These handlers are given the connection hint, and are expected to return an "Endpoint" object. Endpoints are a Twisted concept: they implement the IStreamClientEndpoint interface, and have a "connect()" method. Handlers are registered for specific hint types. If the handler is unable to parse the hint it was given (or is otherwise unable to produce a suitable Endpoint), it should raise InvalidHintError, and the Tub will ignore the hint. Adding New Connection Handlers ------------------------------ Connection handlers can be added to the Tub with `addConnectionHintHandler`: .. code-block:: python tub = Tub() tub.addConnectionHintHandler("tor", tor.socks_port("127.0.0.1", 9050)) Note that each Tub has a separate list of handlers, so if your application uses multiple Tubs, you must add the handler to all of them. Handlers are stored in a dictionary, with "tcp:" hints handled by the built-in `tcp.default` handler. Recommended Connection-Hint Types --------------------------------- Connection handlers allow for arbitrary hint types, and applications can put whatever they want into `Tub.setLocation()`, so this list is not exhaustive. But to improve interoperability, applications are encouraged to converge on at least the following hint types: * `tcp:HOSTNAME:PORT` : This is the standard hint type. It indicates that clients should perform DNS resolution on `HOSTNAME` (if it isn't already a dotted-quad IPv4 address), make a simple TCP connection to the IPv4 or IPv6 addresses that result, and perform negotiation with anything that answers. (in the future, this will accept `tcp:[IPv6:COLON:HEX]:PORT`, see ticket #155) * (legacy) `HOSTNAME:PORT` : Older applications used this form to indicate standard TCP hints. If `HOSTNAME` and `PORT` are of the expected form, this is converted (by prepending `tcp:`) before being delivered to the `tcp:` handler. New applications should not emit this form. * `tor:HOSTNAME:PORT` : This indicates the client should connect to `HOSTNAME:PORT` via a Tor proxy. The only meaningful reason for putting a `tor:` hint in your FURL is if `HOSTNAME` ends in `.onion`, indicating that the Tub is listening on a Tor "onion service" (aka "hidden service"). * `i2p:ADDR` : Like `tor:`, but use an I2P proxy. `i2p:ADDR:PORT` is also legal, although I2P services do not generally use port numbers. Built-In Connection Handlers ---------------------------- Foolscap includes connection handlers that know how to use SOCKS5 and Tor proxies. They live in their own modules, which must be imported separately. These functions are not in `foolscap.api`, because they depend upon additional libraries (`txsocksx` and `txtorcon`) which Foolscap does not automatically depend upon. Your application can declare a dependency upon Foolscap with "extras" to include these additional libraries, e.g. your `setup.py` would contain `install_requires=["foolscap[tor]"]` to enable `from foolscap.connections import tor`. All handlers live in modules under in the `foolscap.connections` package, so e.g.: .. code-block:: python from foolscap.connections import tcp handler = tcp.default() tub.addConnectionHintHandler("tcp", handler) Foolscap's built-in connection handlers are: * `tcp.default()` : This is the basic TCP handler which all Tubs use for `tcp:` hints by default. * `socks.socks_endpoint(proxy_endpoint)` : This routes connections to a SOCKS5 server at the given endpoint. * `tor.default_socks()` : This attempts a SOCKS connection to `localhost` port 9050, which is the Tor default SOCKS port. If that fails, it tries port 9150, which is where the Tor Browser Bundle runs a SOCKS port. This should work if either Tor or the TBB are running on the current host. * `tor.socks_endpoint(tor_socks_endpoint)` : This makes a SOCKS connection to an arbitrary endpoint. When using TCP endpoints, callers are strongly encouraged to use `host="localhost"`, rather than allowing traffic to travel off-box. * `tor.control_endpoint(tor_control_endpoint)` : This connects to a pre-existing Tor daemon via it's "Control Port", which allows the handler to query Tor for its current SOCKS port (as well as control Tor entirely). On debian systems, the control port lives at `unix:/var/run/tor/control`, but the user must be a member of the `debian-tor` unix group to access it. The handler only makes one attempt to connect to the control port (when the first hint is processed), and uses that connection for all subsequent hints. * `tor.control_endpoint_maker(tor_control_endpoint_maker)` : This is like `tor_control_endpoint()`, but instead of providing the endpoint directly, you provide a function which will return the endpoint on-demand. The function won't be called until a `tor:` hint is encountered, so you can avoid doing work until needed, and it can return a Deferred, so the handler won't try to connect to Tor until your function says it's ready. The function is given one argument: the reactor to use. This handler exists to support a use case where the application wants to launch Tor on it's own, so it can set up an onion-service listener, but wants to use the same Tor for outbound connections too (`tor.launch()` doesn't expose enough txtorcon internals to support this, and `tor.control_endpoint()` needs to know the endpoint too early, and also connects to Tor too early). * `tor.launch(data_directory=None, tor_binary=None)` : This launches a new copy of Tor (once, when the first hint is processed). `tor_binary=` points to the exact executable to be run, otherwise it will search $PATH for the `tor` executable. If `data_directory=` is provided, it will be used for Tor's persistent state: this allows Tor to cache the "microdescriptor list" and can speed up the second invocation of the program considerably. If not provided, a ephemeral temporary directory is used (and deleted at shutdown). * `i2p.default(reactor)` : This uses the "SAM" protocol over the default I2P daemon port (localhost:7656) to reach an I2P server. Most I2P daemons are listening on this port. * `i2p.sam_endpoint(endpoint)` : This uses SAM on an alternate port to reach the I2P daemon. * (future) `i2p.local_i2p(configdir=None)` : When implemented, this will contact an already-running I2P daemon by reading it's configuration to find a contact method. * (future) `i2p.launch(configdir=None, binary=None)` : When implemented, this will launch a new I2P daemon (with arguments similar to `tor.launch`). Applications which want to enable as many connection-hint types as possible should simply install the `tor.default_socks()` and `i2p.default()` handlers if they can be imported. This will Just Work(tm) if the most common deployments of Tor/I2P are installed+running on the local machine. If not, those connection hints will be ignored. .. code-block:: python try: from foolscap.connections import tor tub.addConnectionHintHandler("tor", tor.default_socks()) except ImportError: pass # we're missing txtorcon, oh well try: from foolscap.connections import i2p tub.addConnectionHintHandler("i2p", i2p.default(reactor)) except ImportError: pass # we're missing txi2p Configuring Endpoints for Connection Handlers --------------------------------------------- Some of these handlers require an Endpoint to reference a proxy server. The easiest way to obtain a Client Endpoint that reaches a TCP service is like this: .. code-block:: python from twisted.internet imports endpoints proxy_endpoint = endpoints.HostnameEndpoint(reactor, "localhost", 8080) Applications can use a string from their config file to specify the Endpoint to use. This gives end users a lot of flexibility to control the application's behavior. Twisted's `clientFromString` function parses a string and returns an endpoint: .. code-block:: python from twisted.internet import reactor, endpoints config = "tcp:localhost:8080" proxy_endpoint = endpoints.clientFromString(reactor, config) Disabling Built-In TCP Processing --------------------------------- Normal "tcp" hints are handled by a built-in connection handler named `tcp.default`. This handles "tcp:example.org:12345". It also handles the backwards-compatible "example.org:12345" format (still in common use), because all such hints are translated into the modern "tcp:example.org:12345" format before the handler lookup takes place. You might want to disable the `tcp.default` handler, for example to run a client strictly behind Tor. In this configuration, *all* outbound connections must be made through the Tor SOCKS proxy (since any direct TCP connections would expose the client's IP address). Any "tcp:" hints must be routed through a Tor-capable connection handler. To accomplish this, you would use `Tub.removeAllConnectionHintHandlers()` to remove the `tcp.default` handler, then you would add a Tor-aware "tcp:" handler. You might also add a "tor:" handler, to handle hints that point at hidden services. .. code-block:: python from foolscap.connections import tor tub.removeAllConnectionHintHandlers() handler = tor.default_socks() tub.addConnectionHintHandler("tcp", handler) tub.addConnectionHintHandler("tor", handler) Writing Handlers (IConnectionHintHandler) ----------------------------------------- The handler is required to implement `foolscap.ipb.IConnectionHintHandler`, and to provide a method named `hint_to_endpoint()`. This method takes three arguments (hint, reactor, and `update_status` callable), and must return a (endpoint, hostname) tuple. The handler will not be given hints for which it was not registered, but if it is unable to parse the hint, it should raise `ipb.InvalidHintError`. Also note that the handler will be given the whole hint, including the type prefix that was used to locate the handler. `hint_to_endpoint()` is allowed to return a Deferred that fires with the (endpoint, hostname) tuple, instead of returning an immediate value. The endpoint returned should implement `twisted.internet.interfaces.IStreamClientEndpoint`, and the endpoint's final connection object must implement `ITLSTransport` and offer the `startTLS` method. Normal TCP sockets (`TCP4ClientEndpoint` objects) do exactly this. The `hostname` value is used to construct an HTTP `Host:` header during negotiation. This is currently underused, but if the connection hint has anything hostname-shaped, put it here. Note that these are not strictly plugins, in that the code doesn't automatically scan the filesystem for new handlers (e.g. with twisted.plugin or setuptools entrypoint plugins). You must explicitly install them into each Tub to have any effect. Applications are free to use plugin-management frameworks to discover objects that implement `IConnectionHintHandler` and install them into each Tub, however most handlers probably need some local configuration (e.g. which SOCKS port to use), and all need a hint_type for the registration, so this may not be as productive as it first appears. Status delivery: the third argument to ``hint_to_endpoint()`` will be a one-argument callable named ``update_status()``. While the handler is trying to produce an endpoint, it may call ``update_status(status)`` with a (native) string argument each time the connection process has achieved some new state (e.g. ``launching tor``, ``connecting to i2p``). This will be used by the ``ConnectionInfo`` object to provide connection status to the application. Note that once the handler returns an endpoint (or the handler's Deferred finally fires), the status will be replaced by ``connecting``, and the handler should stop calling the status function. If the handler raises an error (or yields a Deferred that errbacks), and the exception object has a (native) string attribute named ``foolscap_connection_handler_error_suffix``, this string will be appended to the usual connection status (a value in the ``ConnectionInfo.connectorStatuses`` dict), which is normally just the stringified exception value. By setting this to something like ``(while connecting to Tor)``, this can be used to distinguish between a failure to connect to the Tor daemon, versus Tor failing to connect to the target onion service. foolscap-0.13.1/doc/copyable.rst0000644000076500000240000003231112766553111017127 0ustar warnerstaff00000000000000Using Pass-By-Copy in Foolscap ============================== Certain objects (including subclasses of ``foolscap.Copyable`` and things for which an ``ICopyable`` adapter has been registered) are serialized using copy-by-value semantics. Each such object is serialized as a (copytype, state) pair of values. On the receiving end, the "copytype" is looked up in a table to find a suitable deserializer. The "state" information is passed to this deserializer to create a new instance that corresponds to the original. Note that the sending and receiving ends are under no obligation to use the same class on each side: it is fairly common for the remote form of an object to have different methods than the original instance. Copy-by-value (as opposed to copy-by-reference) means that the remote representation of an object leads an independent existence, unconnected to the original. Sending the same object multiple times will result in separate independent copies. Sending the result of a pass-by-copy operation back to the original sender will, at best, result in the sender holding two separate objects containing similar state (and at worst will not work at all: not all RemoteCopies are themselves Copyable). More complex copy semantics can be accomplished by writing custom Slicer code. For example, to get an object that is copied by value the first time it traverses the wire, and then copied by reference all later times, you will need to write a Slicer/Unslicer pair to implement this functionality. Likewise the oldpb ``Cacheable`` class would need to be implemented with a custom Slicer/Unslicer pair. Copyable -------- The easiest way to send your own classes over the wire is to use ``Copyable`` . On the sending side, this requires two things: your class must inherit from ``foolscap.Copyable`` , and it must define an attribute named ``typeToCopy`` with a unique string. This copytype string is shared between both sides, so it is a good idea to use a stable and globally unique value: perhaps a URL rooted in a namespace that you control, or a UUID, or perhaps the fully-qualified package+module+class name of the class being serialized. Any string will do, as long as it matches the one used on the receiving side. The object being sent is asked to provide a state dictionary by calling its ``ICopyable.getStateToCopy`` method. The default implementation of ``getStateToCopy`` will simply return ``self.__dict__`` . You can override ``getStateToCopy`` to control what pieces of the source object get copied to the target. In particular, you may want to override ``getStateToCopy`` if there is any portion of the object's state that should **not** be sent over the wire: references to objects that can not or should not be serialized, or things that are private to the application. It is common practice to create an empty dictionary in this method and then copy items into it. On the receiving side, you must register the copytype and provide a function to deserialize the state dictionary back into an instance. For each ``Copyable`` subclass you will create a corresponding ``RemoteCopy`` subclass. There are three requirements which must be fulfilled by this subclass: #. ``copytype`` : Each ``RemoteCopy`` needs a ``copytype`` attribute which contains the same string as the corresponding ``Copyable`` 's ``typeToCopy`` attribute. (metaclass magic is used to auto-register the ``RemoteCopy`` class in the global copytype-to-RemoteCopy table when the class is defined. You can also use ``registerRemoteCopy`` to manually register a class). #. ``__init__`` : The ``RemoteCopy`` subclass must have an __init__ method that takes no arguments. When the receiving side is creating the incoming object, it starts by creating a new instance of the correct ``RemoteCopy`` subclass, and at this point it has no arguments to work with. Later, once the instance is created, it will call ``setCopyableState`` to populate it. #. ``setCopyableState`` : Your ``RemoteCopy`` subclass must define a method named ``setCopyableState`` . This method will be called with the state dictionary that came out of ``getStateToCopy`` on the sending side, and is expected to set any necessary internal state. Note that ``RemoteCopy`` is a new-style class: if you want your copies to be old-style classes, inherit from ``RemoteCopyOldStyle`` and manually register the copytype-to-subclass mapping with ``registerRemoteCopy`` . (doc/listings/copyable-send.py) .. code-block:: python #! /usr/bin/python from twisted.internet import reactor from foolscap.api import Copyable, Referenceable, Tub # the sending side defines the Copyable class UserRecord(Copyable): # this class uses the default Copyable behavior typeToCopy = "unique-string-UserRecord" def __init__(self, name, age, shoe_size): self.name = name self.age = age self.shoe_size = shoe_size # this is a secret def getStateToCopy(self): d = {} d['name'] = self.name d['age'] = self.age # don't tell anyone our shoe size return d class Database(Referenceable): def __init__(self): self.users = {} def addUser(self, name, age, shoe_size): self.users[name] = UserRecord(name, age, shoe_size) def remote_getuser(self, name): return self.users[name] db = Database() db.addUser("alice", 34, 8) db.addUser("bob", 25, 9) tub = Tub() tub.listenOn("tcp:12345") tub.setLocation("localhost:12345") url = tub.registerReference(db, "database") print "the database is at:", url tub.startService() reactor.run() (doc/listings/copyable-receive.py) .. code-block:: python #! /usr/bin/python import sys from twisted.internet import reactor from foolscap.api import RemoteCopy, Tub # the receiving side defines the RemoteCopy class RemoteUserRecord(RemoteCopy): copytype = "unique-string-UserRecord" # this matches the sender def __init__(self): # note: our __init__ must take no arguments pass def setCopyableState(self, d): self.name = d['name'] self.age = d['age'] self.shoe_size = "they wouldn't tell us" def display(self): print "Name:", self.name print "Age:", self.age print "Shoe Size:", self.shoe_size def getRecord(rref, name): d = rref.callRemote("getuser", name=name) def _gotRecord(r): # r is an instance of RemoteUserRecord r.display() reactor.stop() d.addCallback(_gotRecord) from foolscap.api import Tub tub = Tub() tub.startService() d = tub.getReference(sys.argv[1]) d.addCallback(getRecord, "alice") reactor.run() Registering Copiers to serialize third-party classes ---------------------------------------------------- If you wish to serialize instances of third-party classes that are out of your control (or you simply want to avoid subclassing), you can register a Copier to provide serialization mechanisms for those instances. There are plenty of cases where it is difficult to arrange for all of the data you send over the wire to be in the form of ``Copyable`` subclasses. For example, you might have a codebase that produces a deeply-nested data structure that contains instances of pre-existing classes. Those classes are written by other people, and do not happen to inherit from ``Copyable`` . Without Copiers, you would have to traverse the whole structure, locate all instances of these non-``Copyable`` classes, and wrap them in some new ``Copyable`` subclass. Registering a Copier for the third-party class is much easier. The ``foolscap.copyable.registerCopier`` function is used to provide a "copier" for any given class. This copier is a function that accepts an instance of the given class, and returns a (copytype, state) tuple. For example [#]_ , the xmlrpclib module provides a ``DateTime`` class, and you might have a data structure that includes some instances of them: .. code-block:: python import xmlrpclib from foolscap import registerCopier def copy_DateTime(xd): return ("_xmlrpclib_DateTime", {"value": xd.value}) registerCopier(xmlrpclib.DateTime, copy_DateTime) This insures that any ``xmlrpclib.DateTime`` that is encountered while serializing arguments or return values will be serialized with a copytype of "_xmlrpclib_DateTime" and a state dictionary containing the single "value" key. Even ``DateTime`` instances that appear arbitrarily deep inside nested data structures will be serialized this way. For example, one a method argument might be dictionary, and one of its keys was a list, and that list could containe a ``DateTime`` instance. To deserialize this object, the receiving side needs to register a corresponding deserializer. ``foolscap.copyable.registerRemoteCopyFactory`` is the receiving-side parallel to ``registerCopier`` . It associates a copytype with a function that will receive a state dictionary and is expected to return a fully-formed instance. For example: .. code-block:: python import xmlrpclib from foolscap import registerRemoteCopyFactory def make_DateTime(state): return xmlrpclib.DateTime(state["value"]) registerRemoteCopyFactory("_xmlrpclib_DateTime", make_DateTime) Note that the "_xmlrpclib_DateTime" copytype **must** be the same for both the copier and the RemoteCopyFactory, otherwise the receiving side will be unable to locate the correct deserializer. It is perfectly reasonable to include both of these function/registration pairs in the same module, and import it in the code on both sides of the wire. The examples describe the sending and receiving sides separately to emphasize the fact that the recipient may be running completely different code than the sender. Registering ICopyable adapters ------------------------------ A slightly more generalized way to teach Foolscap about third-party classes is to register an ``ICopyable`` adapter for them, using the usual (i.e. zope.interface) adapter-registration mechanism. The object that provides ``ICopyable`` needs to implement two methods: ``getTypeToCopy`` (which returns the copytype), and ``getStateToCopy`` , which returns the state dictionary. Any object which can be adapted to ``ICopyable`` can be serialized this way. On the receiving side, the copytype is looked up in the ``CopyableRegistry`` to find a corresponding UnslicerFactory. The ``registerRemoteCopyUnslicerFactory`` function accepts two arguments: the copytype, and the unslicer factory to use. This unslicer factory is simply a function that takes no arguments and returns a new Unslicer. Each time an inbound message with the matching copytype is received, ths unslicer factory is invoked to create an Unslicer that will be responsible for the single instance described in the message. This Unslicer must implement an interface described in the Unslicer specifications, in "doc/specifications/pb". Registering ISlicer adapters ---------------------------- The most generalized way to serialize classes is to register a whole ``ISlicer`` adapter for them. The ``ISlicer`` gets complete control over serialization: it can stall the production of tokens by implementing a ``slice`` method that yields Deferreds instead of basic objects. It can also interact with other objects while the target is being serialized. As an extreme example, if you had a service that wanted to migrate an open HTTP connection from one process to another, the ``ISlicer`` could communication with a front-end load-balancing box to redirect the connection to the new host. In this case, the slicer could theoretically tell the load-balancer to pause the connection and assign it a rendezvous number, then serialize this rendezvous number as a form of "claim check" to the target process. The ``IUnslicer`` on the receiving end could open a new listening port, then use the claim check to tell the load-balancer to direct the connection to this new port. Likewise two services running on the same host could conspire to pass open file descriptors over a Foolscap connection (via an auxilliary unix-domain socket) through suitable magic in the ``ISlicer`` and ``IUnslicer`` on each end. The Slicers and Unslicers are described in more detail in the specifications: "doc/specifications/pb". Note that a ``Copyable`` with a copytype of "foo" is serialized as the following token stream: ``OPEN, "copyable", "foo", [state dictionary..], CLOSE``. Any ``ISlicer`` adapter which wishes to match what ``Copyable`` does needs to include the extra "copyable" opentype string first. Also note that using a custom Slicer introduces an opportunity to violate serialization coherency. ``Copyable`` and Copiers transform the original object into a state dictionary in one swell foop, not allowing any other code to get control (and possibly mutate the object's state). If your custom Slicer allows other code to get control during serialization, then the object's state might be changed, and thus the serialized state dictionary could wind up looking pretty weird. .. rubric:: Footnotes .. [#] many thanks to Ricky Iacovou for the xmlrpclib.DateTime example foolscap-0.13.1/doc/failures.rst0000644000076500000240000005016612766553111017153 0ustar warnerstaff00000000000000Foolscap Failure Reporting ========================== Signalling Remote Exceptions ---------------------------- The ``remote_`` -prefixed methods which Foolscap invokes, just like their local counterparts, can either return a value or raise an exception. Foolscap callers can use the normal Twisted conventions for handling asyncronous failures: ``callRemote`` returns a Deferred object, which will eventually either fire its callback function (if the remote method returned a normal value), or its errback function (if the remote method raised an exception). There are several reasons that the Deferred returned by ``callRemote`` might fire its errback: - local outbound schema violation: the outbound method arguments did not match the ``RemoteInterface`` that is in force. This is an optional form of typechecking for remote calls, and is activated when the remote object describes itself as conforming to a named ``RemoteInterface`` which is also declared in a local class. The local constraints are checked before the message is transmitted over the wire. A constraint violation is indicated by raising ``foolscap.schema.Violation`` , which is delivered through the Deferred's errback. - network partition: if the underlying TCP connection is lost before the response has been received, the Deferred will errback with a ``foolscap.ipb.DeadReferenceError`` exception. Several things can cause this: the remote process shutting down (intentionally or otherwise), a network partition or timeout, or the local process shutting down (``Tub.stopService`` will terminate all outstanding remote messages before shutdown). - remote inbound schema violation: as the serialized method arguments were unpacked by the remote process, one of them violated that processes inbound ``RemoteInterface`` . This check serves to protect each process from incorrect types which might either confuse the subsequent code or consume a lot of memory. These constraints are enforced as the tokens are read off the wire, and are signalled with the same ``Violation`` exception as above (but this may be wrapped in a ``RemoteException`` : see below). - remote method exception: if the ``remote_`` method raises an exception, or returns a Deferred which subsequently fires its errback, the remote side will send the caller that an exception occurred, and may attempt to provide some information about this exception. The caller will see an errback that may or may not attempt to replicate the remote exception. This may be wrapped in a ``RemoteException`` . See below for more details. - remote outbound schema violation: as the remote method's return value is serialized and put on the wire, the values are compared against the return-value constraint (if a ``RemoteInterface`` is in effect). If it does not match the constraint, a Violation will be raised (but may be wrapped in a ``RemoteException`` ). - local inbound schema violation: when the serialized return value arrives on the original caller's side of the wire, the return-value constraint of any effective ``RemoteInterface`` will be applied. This protects the caller's response code from unexpected values. Any mismatches will be signalled with a Violation exception. Distinguishing Remote Exceptions -------------------------------- When a remote call fails, what should you do about it? There are several factors to consider. Raising exceptions may be part of your remote API: easy-to-use exceptions are a big part of Python's success, and Foolscap provides the tools to use them in a remote-calling environment as well. Exceptions which are not meant to be part of the API frequently indicate bugs, sometimes as precondition assertions (of which schema Violations are a subset). It might be useful to react to the specific type of remote exception, and/or it might be important to log as much information as possible so a programmer can find out what went wrong, and in either case it might be appropriate to react by falling back to some alternative code path. Good debuggability frequently requires at least one side of the connection to get lots of information about errors that indicate possible bugs. Note that the ``Tub.setOption("logLocalFailures", True)`` and ``Tub.setOption("logRemoteFailures", True)`` options are relevant: when these options are enabled, exceptions that are sent over the wire (in one direction or the other) are recorded in the Foolscap log stream. If you use exceptions as part of your regular remote-object API, you may want to consider disabling both options. Otherwise the logs may be cluttered with perfectly harmless exceptions. Should your code pay attention to the details of a remote exception (other than the fact that an exception happened at all)? There are roughly two schools of thought: - Distrust Outsiders: assume, like any sensible program which connects to the internet, that the entire world is out to get you. Use external services to the extent you can, but don't allow them to confuse you or trick you into some code path that will expose a vulnerability. Treat all remote exceptions as identical. - "E" mode: treat external code with the same level of trust or distrust that you would apply to local code. In the "E" programming language (which inspires much of Foolscap's feature set), each object is a separate trust domain, and the only distinction made between "local" and "remote" objects is that the former may be called synchronously, while the latter may become partitioned. Treat remote exceptions just like local ones, interpreting their type as best you can. From Foolscap's point of view, what we care about is how to handle exceptions raised by the remote code. When operating in the first mode, Foolscap will merge all remote exceptions into a single exception type named ``foolscap.api.RemoteException`` , which cannot be confused with regular Python exceptions like ``KeyError`` and ``AttributeError`` . In the second mode, Foolscap will try to convert each remote exception into a corresponding local object, so that error-handling code can catch e.g. ``KeyError`` and use it as part of the remote API. To tell Foolscap which mode you want to use, call ``tub.setOption("expose-remote-exception-types", BOOL)`` , where BOOL is either True (for the "E mode") or False (for the "Distrust Outsiders" mode). The default is True. In "Distrust Outsiders" mode, a remote exception will cause the caller's errback handler to be called with a regular ``Failure`` object which contains a ``foolscap.api.RemoteException`` , effectively hiding all information about the nature of the problem except that it was caused by some other system. Caller code can test for this with ``f.check`` and ``f.trap`` as usual. If the caller's code decides to investigate further, it can use ``f.value.failure`` to obtain the ``CopiedFailure`` (see below) that arrived from the remote system. Note that schema Violations which are caught on the local system are reported normally, whereas Violations which are caught on the remote system are reported as RemoteExceptions. In "E mode", a remote exception will cause the errback handler to be called with a ``CopiedFailure`` object. This ``CopiedFailure`` will behave as much as possible like the corresponding Failure from the remote side, given the limitations of the serialization process (see below for details). In particular, if the remote side raises e.g. a standard Python ``IndexError`` , the local side can use ``f.trap(IndexError)`` to catch it. However, this same f.trap call would also catch locally-generated IndexErrors, which could be confusing. Examples: Distrust Outsiders ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Since Deferreds can be chained, it is quite common to see remote calls sandwiched in the middle of two (possibly asynchronous) local calls. The following snippet performs a local processing step, then asks a remote server for information, then adds that information into a local database. All three steps are asynchronous. .. code-block:: python # Example 1 def get_and_store_record(name): d = local_db.getIDNumber(name) d.addCallback(lambda idnum: rref.callRemote("get_record", idnum)) d.addCallback(lambda record: local_db.storeRecord(name)) return d To motivate an examination of error handling, we'll extend this example to use two separate servers for the record: if one of them doesn't have it, we ask the other. The first server might raise ``KeyError`` to tell us it can't find the record, or it might experience some other internal error, or we might lose the connection to that server before it can get us an answer: all three cases should prompt us to talk to the second server. .. code-block:: python # Example 2 from foolscap.api import Tub, RemoteException t = Tub() t.setOption("expose-remote-exception-types", False) # Distrust Outsiders ... def get_and_store_record(name): d = local_db.getIDNumber(name) def get_record(idnum): d2 = server1.callRemote("get_record", idnum) # could raise KeyError def maybe_try_server2(f): f.trap(RemoteException) return server2.callRemote("get_record", idnum) # or KeyError d2.addErrback(maybe_try_server2) return d2 d.addCallback(get_record) d.addCallback(lambda record: local_db.storeRecord(name)) return d In this example, only a failure that occurs on server1 will cause the code to attempt to use server2. A locally-triggered error will be trapped by the first line of ``maybe_try_server2`` and will not proceed to the second ``callRemote`` . This allows a more complex control flow like the following: .. code-block:: python # Example 3 def get_and_store_record(name): d = local_db.getIDNumber(name) # could raise IndexError def get_record(idnum): d2 = server1.callRemote("get_record", idnum) # or KeyError def maybe_try_server2(f): f.trap(RemoteException) return server2.callRemote("get_record", idnum) # or KeyError d2.addErrback(maybe_try_server2) return d2 d.addCallback(get_record) d.addCallback(lambda record: local_db.storeRecord(name)) def ignore_unknown_names(f): f.trap(IndexError) print "Couldn't get ID for name, ignoring" return None d.addErrback(ignore_unknown_names) def failed(f): print "didn't get data!" if f.check(RemoteException): if f.value.failure.check(KeyError): print "both servers claim to not have the record" else: print "both servers had error" else: print "local error" print "error details:", f d.addErrback(failed) return d The final ``failed`` method will catch any unexpected error: this is the place where you want to log enough information to diagnose a code bug. For example, if the database fetch had returned a string, but the RemoteInterface had declared ``get_record`` as taking an integer, then the ``callRemote`` would signal a (local) Violation exception, causing control to drop directly to the ``failed()`` error handler. On the other hand, if the first server decided to throw a Violation on its inbound argument, the ``callRemote`` would signal a RemoteException (wrapping a Violation), and control would flow to the ``maybe_try_server2`` fallback. It is usually best to put the errback as close as possible to the call which might fail, since this provides the highest "signal to noise ratio" (i.e. it reduces the number of possibilities that the error-handler code must handle). But it is frequently more convenient to place the errback later in the Deferred chain, so it can be useful to distinguish between the local ``IndexError`` and a remote exception of the same type. This is the same decision that needs to be made with synchronous code: whether to use lots of ``try:/except:`` blocks wrapped around individual method calls, or to use one big block around a whole sequence of calls. Smaller blocks will catch an exception sooner, but larger blocks are less effort to write, and can be more appropriate, especially if you do not expect exceptions to happen very often. Note that if this example had used "E mode" and the first remote server decided (perhaps maliciously) to raise ``IndexError`` , then the client could be tricked into following the same ignore-unknown-names code path that was meant to be reserved for a local database miss. To examine the type of failure more closely, the error-handling code should access the ``RemoteException`` 's ``.value.failure`` attribute. By making the following change to ``maybe_try_server2`` , the behavior is changed to only query the second server in the specific case of a remote ``KeyError`` . Other remote exceptions (and all local exceptions) will skip the second query and signal an error to ``failed()`` . You might want to do this if you believe that a remote failure like ``AttributeError`` is worthy of error-logging rather than fallback behavior. .. code-block:: python # Example 4 def maybe_try_server2(f): f.trap(RemoteException) if f.value.failure.check(KeyError): return server2.callRemote("get_record", idnum) # or KeyError return f Note that you should probably not use ``f.value.failure.trap`` , since if the exception type does not match, that will raise the inner exception (i.e. the ``KeyError`` ) instead of the ``RemoteException`` , potentially confusing subsequent error-handling code. Examples: E Mode ~~~~~~~~~~~~~~~~ Systems which use a lot of remote exceptions as part of their inter-process API can reduce the size of the remote-error-handling code by switching modes, at the expense of risking confusion between local and remote occurrences of the same exception type. In the following example, we use "E Mode" and look for ``KeyError`` to indicate a remote ``get_record`` miss. .. code-block:: python # Example 5 from foolscap.api import Tub t = Tub() t.setOption("expose-remote-exception-types", True) # E Mode ... def get_and_store_record(name): d = local_db.getIDNumber(name) def get_record(idnum): d2 = server1.callRemote("get_record", idnum) # or KeyError def maybe_try_server2(f): f.trap(KeyError) return server2.callRemote("get_record", idnum) # or KeyError d2.addErrback(maybe_try_server2) return d2 d.addCallback(get_record) d.addCallback(lambda record: local_db.storeRecord(name)) def ignore_unknown_names(f): f.trap(IndexError) print "Couldn't get ID for name, ignoring" return None d.addErrback(ignore_unknown_names) def failed(f): print "didn't get data!" if f.check(KeyError): # don't bother showing details print "both servers claim to not have the record" else: # show details by printing "f", the Failure instance print "other error", f d.addErrback(failed) return d In this example, ``KeyError`` is part of the remote ``get_record`` method's API: it either returns the data, or it raises KeyError, and anything else indicates a bug. The caller explicitly catches KeyError and responds by either falling back to the second server (the first time) or announcing a servers-have-no-record error (if the fallback failed too). But if something else goes wrong, the client indicates a different error, along with the exception that triggered it, so that a programmer can investigate. The remote error-handling code is slightly simpler, relative to the identical behavior expressed in Example 4, since ``maybe_try_server2`` only needs to use ``f.trap(KeyError)`` , instead of needing to unwrap a ``RemoteException`` first. But when this error-handling code is at the end of a larger block (such as the ``f.trap(IndexError)`` in ``ignore_unknown_names()`` , or the ``f.check(KeyError)`` in ``failed()`` ), it is vulnerable to confusion: if ``local_db.getIDNumber`` raised ``KeyError`` (instead of the expected ``IndexError`` ), or if the remote server raised ``IndexError`` (instead of ``KeyError`` ), then the error-handling logic would follow the wrong path. Default Mode ~~~~~~~~~~~~ Exception modes were introduced in Foolscap-0.4.0 . Releases before that only offered "E mode". The default in 0.4.0 is "E mode" (expose-remote-exception-types=True), to retain compatibility with the exception-handling code in existing applications. A future release of Foolscap may change the default mode to expose-remote-exception-types=False, since it seems likely that apps written in this style are less likely to be confused by remote exceptions of unexpected types. CopiedFailures -------------- Twisted uses the ``twisted.python.failure.Failure`` class to encapsulate Python exceptions in an instance which can be passed around, tested, and examined in an asynchronous fashion. It does this by copying much of the information out of the original exception context (including a stack trace and the exception instance itself) into the ``Failure`` instance. When an exception is raised during a Deferred callback function, it is converted into a Failure instance and passed to the next errback handler in the chain. When ``RemoteReference.callRemote`` needs to transport information about a remote exception over the wire, it uses the same convention. However, Failure objects cannot be cleanly serialized and sent over the wire, because they contain references to local state which cannot be precisely replicated on a different system (stack frames and exception classes). So, when an exception happens on the remote side of a ``callRemote`` invocation, and the exception-handling mode passes the remote exception back to the calling code somehow, that code will receive a ``CopiedFailure`` instance instead. In "E mode", the ``callRemote`` 's errback function will receive a ``CopiedFailure`` in response to a remote exception, and will receive a regular ``Failure`` in response to locally-generated exceptions. In "Distrust Outsiders" mode, the errback will always receive a regular ``Failure`` , but if ``f.check(foolscap.api.RemoteException)`` is True, then the ``CopiedFailure`` can be obtained with ``f.value.failure`` and examined further. ``CopiedFailure`` is designed to behave very much like a regular ``Failure`` object. The ``check`` and ``trap`` methods work on ``CopiedFailure`` s just like they do on ``Failure`` s. However, all of the Failure's attributes must be converted into strings for serialization. As a result, the original ``.value`` attribute (which contains the exception instance, which might contain additional information about the problem) is replaced by a stringified representation, which tends to lose information. The frames of the original stack trace are also replaced with a string, so they can be printed but not examined. The exception class is also passed as a string (using Twisted's ``reflect.qual`` fully-qualified-name utility), but ``check`` and ``trap`` both compare by string name instead of object equality, so most applications won't notice the difference. The default behavior of CopiedFailure is to include a string copy of the stack trace, generated with ``printTraceback()`` , which will include lines of source code when available. To reduce the amount of information sent over the wire, stack trace strings larger than about 2000 bytes are truncated in a fashion that tries to preserve the top and bottom of the stack. unsafeTracebacks ~~~~~~~~~~~~~~~~ Applications which consider their lines of source code or their exceptions' list of (filename, line number) tuples to be sensitive information can set the "unsafeTracebacks" flag in their Tub to False; the server will then remove stack information from the CopiedFailure objects it sends to other systems. .. code-block:: python t = Tub() t.unsafeTracebacks = False When unsafeTracebacks is False, the ``CopiedFailure`` will only contain the stringified exception type, value, and parent class names. foolscap-0.13.1/doc/flappserver.rst0000644000076500000240000003714012766553111017667 0ustar warnerstaff00000000000000Flappserver: The Foolscap Application Server ============================================ Foolscap provides an "app server", to conveniently deploy small applications that were written by others. It fulfills the same role as "twistd" does for Twisted code: it allows sysadmins to configure and launch services without obligating them to write Python code for each one. Example ------- This example creates a file-uploading service on one machine, and uses the corresponding client on a different machine to transfer a file. There are many different kinds of services that can be managed this way: file-uploading is just one of them. .. code-block:: console ## run this on the server machine S% flappserver create ~/fs Listening on 127.0.0.1:12345 Foolscap Application Server created in ~/fs S% mkdir ~/incoming S% flappserver add ~/fs upload-file ~/incoming Service created, FURL is pb://kykr3p2hsippfgxqq2icrbrncee2f6ef@127.0.0.1:12345/47nvyzu6dj6apyrdl7alpe2xasmi52jt S% flappserver start ~/fs Server Started S% ## run this on the client machine C% echo "pb://kykr3p2hsippfgxqq2icrbrncee2f6ef@127.0.0.1:12345/47nvyzu6dj6apyrdl7alpe2xasmi52jt" >~/.upload.furl C% flappclient --furlfile ~/.upload.furl upload-file foo.jpg ## that uploads the local file "foo.jpg" to the server C% ## run this on the server machine S% ls ~/incoming foo.jpg S% Concepts -------- "flappserver" is both the name of the Foolscap Application Server and the name of the command used to create, configure, and launch it. Each server creates a single foolscap Tub, which listens on one or more TCP ports, and is configured with a "location hint string" that explains (to eventual clients) how to contact the server. The server is given a working directory to store persistent data, including the Tub's private key. Each flappserver hosts an arbitrary number of "services". Each service gets a distinct FURL (all sharing the same TubID and location). Each service gets a private subdirectory which contains its configuration arguments and any persistent state it wants to maintain. When adding a service to a flappserver, you must specify the "service type". This indicates which kind of service you want to create. Any remaining arguments on the "flappserver add" command line will be passed to the service and can be used to configure its behavior. For each service that is added, a new FURL is generated and returned to the user (so they can copy it to the client system that wants to contact this service). The FURL can also be retrieved later through the "flappserver list" command. Nothing happens until the flappserver is started, with "flappserver start". This is simply a front-end for twistd, so it takes twistd arguments like --nodaemon, --syslog, etc (use "twistd --help" for a complete list). The server will run in the background as a standard unix daemon. "flappserver stop" will shut down the daemon. Services -------- The app server has a list of known service types. You can add multiple services of the same type to a single app server. This is analogous to object-oriented programming: the service types are **classes** , and the app server holds zero or more **instances** of each type (each of which is probably configured slightly differently). Service types are defined by plugins, each of which provides the code to implement a named service. The basic services that ship with Foolscap are: - **upload-file** : allow files to be written into a single directory by the corresponding "flappclient upload-file" command. Files are streamed to a neighboring temporary file before being atomically moved into place. The client gets to choose the target filename. Optionally allow the creation and use of subdirectories. - **run-command** : allow a preconfigured shell command to be executed by the corresponding "flappclient run-command" invocation. Client receives stdout/stderr/rc. Command runs in a preconfigured working directory. Optionally allow the client to provide stdin to the command. In a future version: optionally provide locking around the command (allow only one instance to run at a time), optionally merge multiple pending invocations, optionally allow the client to provide arguments to the command. Commands -------- ``flappserver create BASEDIR [options]`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Create a new server, using BASEDIR as a working directory. BASEDIR should not already exist, and nothing else should touch its contents. BASEDIR will be created with mode 0700, to prevent other users from reading it and learning the private key. "create" options: - ``--port`` : strports description of the TCP port to listen on - ``--location`` : location hints to use in generated FURLs. If not provided, the server will attempt to enumerate all network interfaces and create a location hint string using each viable IP address it finds. If you have configured an external NAT or port forwarding for this server, you will need to set --location with the externally-visible listening port. - ``--umask`` : set the (octal) file-creation mask that the server will use at runtime. When your services are invoked, any files they create will have accesss-permissions (the file "mode") controlled by this value. ``flappserver create`` will copy your current umask and use it in the server unless you override it with this option. ``--umask=022`` is a good way to let those created files be world-readable, and ``--umask=077`` is used to make them non-world-redable. ``flappserver add BASEDIR [options] SERVICE-TYPE SERVICE-ARGS`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Add a new service to the existing server that lives in BASEDIR. The new service will be of type SERVICE-TYPE (such as "upload-file" or "run-command"), and will be configured with SERVICE-ARGS. A new unguessable "swissnum" will be generated for the service, from which a FURL will be computed. Clients must use this FURL to contact the service. The FURL will be printed to stdout, where it can be copied and transferred to client machines. It can also be viewed later using the "list" command. The service instance will be created lazily, when a client actually connects to the FURL. There will be only one instance per service, which will last until the flappserver is terminated. (services are of course free to create new per-request objects, which can last as long as necessary) The "add" command takes certain options. Separately, each SERVICE-TYPE will accept one or more SERVICE-ARGS, whose format depends upon the specific type of service being created. The "add" command options must appear before the SERVICE-TYPE parameter, while the SERVICE-ARGS always appear after the SERVICE-TYPE parameter. "add" options: - ``--comment`` : short string explaining what this service is used for, appears in the output of ``flappserver list`` ``flappserver list BASEDIR`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ List information about each service that has been configured in the given flappserver. Each service is listed with the unguessable "swissnum", followed by the service-type and service-args, then any --comment that was given to the add command, finishing with the access FURL: .. code-block:: console % flappserver list ~/fs 47nvyzu6dj6apyrdl7alpe2xasmi52jt: upload-file ~/incoming --allow-subdirectories # --comment text appears here pb://kykr3p2hsippfgxqq2icrbrncee2f6ef@127.0.0.1:12345/47nvyzu6dj6apyrdl7alpe2xasmi52jt jgdqovf3tfd5xog34bxmkqwd3dxgycak: upload-file ~/repo/packages pb://kykr3p2hsippfgxqq2icrbrncee2f6ef@127.0.0.1:12345/jgdqovf3tfd5xog34bxmkqwd3dxgycak 22ngipsyp2smmgguemf5hu45prz4jeui: run-command ~/repo make update-repository pb://kykr3p2hsippfgxqq2icrbrncee2f6ef@127.0.0.1:12345/22ngipsyp2smmgguemf5hu45prz4jeui % The "list" command takes no options. ``flappserver start BASEDIR [twistd options]`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Launch (and usually daemonize) the server that lives in BASEDIR. This command will return quickly, leaving the server running in the background. Logs will be written to BASEDIR/twistd.log unless overridden. The "start" command accepts the same options as twistd, so use ``twistd --help`` to see the options that will be recognized. ``flappserver start BASEDIR`` is equivalent to ``cd BASEDIR && twistd -y *.tac [options]``. ``flappserver stop BASEDIR`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Terminate the server that is running in BASEDIR. This is equivalent to ``"cd BASEDIR && kill `cat twistd.pid`"``. The "stop" command takes no options. ``flappserver restart BASEDIR [twistd options]`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Terminate and restart the server that is running in BASEDIR. This is equivalent to ``"flappserver stop BASEDIR && flappserver start BASEDIR [options]"``. The "restart" command takes the same twistd arguments as **start** . Services -------- ``upload-file [options] TARGETDIR`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This service accepts files from ``flappclient upload-file`` , placing them in TARGETDIR (which must already exist and be writable by the flappserver). The filenames are chosen by the client. Existing files will be overwritten. This service will never write client files above TARGETDIR, even if the client attempts to use ".." or other pathname metacharacters (assuming that a local user has not placed upwards-leading symlinks in TARGETDIR). It will only write to subdirectories of TARGETDIR if the service was configured with ``--allow-subdirectories`` , in which case the client controls which subdirectory is used (and created if necessary). The files will be created with the flappserver's configured ``--umask`` , typically captured when the server is first created. If the server winds up with a restrictive umask like 077, then the files created in TARGETDIR will not be readable by other users. TODO: ``--allow-subdirectories`` is not yet implemented. Example: .. code-block:: console % flappserver create --listen 12345 --location example.com:12345 ~/fl Foolscap Application Server created in /usr/home/warner/fl TubID u5bca3u2wklkyyv7wzjetmfltyqeb6kv, listening on port tcp:12345 Now launch the daemon with 'flappserver start /usr/home/warner/fl' % flappserver add ~/fl upload-file ~/incoming Service added in /usr/home/warner/fl/services/vx3s2tb62ywct4pdgdicdpbxgz4ly7po FURL is pb://u5bca3u2wklkyyv7wzjetmfltyqeb6kv@example.com:12345/vx3s2tb62ywct4pdgdicdpbxgz4ly7po % flappserver start ~/fl Launching Server... Server Running % ``run-command [options] TARGETDIR COMMAND..`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This service invokes a preconfigured command in response to requests from ``flappclient run-command`` . The command is always run with TARGETDIR as its current working directory. COMMAND will be run with the flappserver's configured ``--umask`` , typically captured when the server is first created. If the server winds up with a restrictive umask like 077, then when COMMAND is run with that umask any files it creates will not be readable by other users. "run-command" options: - ``--accept-stdin`` : if set, any data written to the client's stdin will be streamed to the stdin of COMMAND. When the client's stdin is closed, the COMMAND's stdin will also be closed. If omitted, the client will be instructed to not read from its stdin, and COMMAND will not receive any stdin (the pipe will be left open, however). - ``--no-stdin`` : [default] opposite of --accept-stdin. - ``--send-stdout`` : [default] if set, any data written by COMMAND to its stdout will be streamed to the client, which will deliver the data to its own stdout pipe. - ``--no-stdout`` : if set, any data written by COMMAND to its stdout will be discarded, and not sent to the client. - ``--send-stderr`` : [default] if set, any data written by COMMAND to its stderr will be streamed to the client, which will deliver the data to its own stderr pipe. - ``--no-stderr`` : if set, any data written by COMMAND to its stderr will be discarded, and not sent to the client. - ``--log-stdin`` : if set, all incoming stdin data will be written to the twistd.log - ``--no-log-stdin`` : [default] do not log incoming stdin - ``--log-stdout`` : if set, all outgoing stdout data will be written to the twistd.log - ``--no-log-stdout`` : [default] do not log outgoing stdout - ``--log-stderr`` : [default] if set, all outgoing stderr data will be written to the twistd.log - ``--no-log-stderr`` : do not log outgoing stderr The numeric exit status of COMMAND will be delivered to the client, which will exit with the same status. If COMMAND terminates with a signal, a suitable non-zero exit status will be delivered (127). Future options will allow the client to modify COMMAND (in tightly controlled ways), and to wrap a semaphore around the invocation of COMMAND so that overlapping requests do not cause overlapping invocations. Another likely option is to coalesce multiple pending requests into a single invocation. Clients ------- To talk to the services described above, Foolscap comes with a simple multipurpose client tool named ``flappclient`` . This tool always takes a ``--furl=`` or ``--furlfile=`` argument to specify the FURL of the target server. For ``--furlfile=`` , the FURL should be stored in the given file. The client will ignore blank lines and comment lines (those which begin with "#"). It will use the first FURL it sees in the file, ignoring everything beyond that point. It is a good practice to put a comment in your furlfiles to remind you what the FURL points to and where you got it from: .. code-block:: console % cat ~/upload.furl # this FURL points to a file-uploader on ftp.example.com:~/incoming pb://kykr3p2hsippfgxqq2icrbrncee2f6ef@127.0.0.1:12345/47nvyzu6dj6apyrdl7alpe2xasmi52jt % % flappclient --furlfile ~/upload.furl upload-file foo.txt bar.txt foo.txt: uploaded bar.txt: uploaded % The --furlfile form is useful to keep the secret FURL out of a transcript of the command being run, such as in a buildbot logfile. Naming your furlfiles after their purpose is a good practice: the filename then behaves like a "petname": a local identifier that hides the secure connection information. ``flappclient [--furl|--furlfile] upload-file SOURCEFILES..`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This contacts a file-uploader service as created with ``flappserver add BASEDIR upload-file TARGETDIR`` and sends it one or more local files. The basename of each SOURCEFILE will be used to provide the remote filename. TODO (not yet implemented): If there is only one SOURCEFILE argument, then the ``--target-filename=`` option can be used to override the remote filename. If the server side has enabled subdirectories, then ``--target-subdirectory=`` can be used to place the file in a subdirectory of the server's targetdir. ``flappclient [--furl|--furlfile] run-command`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This contacts a command-executing service as created with ``flappserver add BASEDIR run-command TARGETDIR COMMAND`` and asks it to invoke the preconfigured command. If the server was configured with ``--accept-stdin`` , the client will read from stdin until it is closed, continuously sending data to the server, then closing the server's stdin pipe (this is useful for commands like 'grep' which read from stdin). If not, the client will ignore its stdin. By default, the client will write to its stdout and stderr as data arrives from the server (however the server can be configured to not send stdout or stderr). Once the server's process exits, the client will exit with the same exit code. foolscap-0.13.1/doc/jobs.txt0000644000076500000240000005057511562013212016273 0ustar warnerstaff00000000000000-*- outline -*- Reasonably independent newpb sub-tasks that need doing. Most important come first. * decide on a version negotiation scheme Should be able to telnet into a PB server and find out that it is a PB server. Pointing a PB client at an HTTP server (or an HTTP client at a PB server) should result in an error, not a timeout. Implement in banana.Banana.connectionMade(). desiderata: negotiation should take place with regular banana sequences: don't invent a new protocol that is only used at the start of the connection Banana should be useable one-way, for storage or high-latency RPC (the mnet folks want to create a method call, serialize it to a string, then encrypt and forward it on to other nodes, sometimes storing it in relays along the way if a node is offline for a few days). It should be easy for the layer above Banana to feed it the results of what its negotiation would have been (if it had actually used an interactive connection to its peer). Feeding the same results to both sides should have them proceed as if they'd agreed to those results. negotiation should be flexible enough to be extended but still allow old code to talk with new code. Magically predict every conceivable extension and provide for it from the very first release :). There are many levels to banana, all of which could be useful targets of negotiation: which basic tokens are in use? Is there a BOOLEAN token? a NONE token? Can it accept a LONGINT token or is the target limited to 32-bit integers? are there any variations in the basic Banana protocol being used? Could the smaller-scope OPEN-counter decision be deferred until after the first release and handled later with a compatibility negotiation flag? What "base" OPEN sequences are known? 'unicode'? 'boolean'? 'dict'? This is an overlap between expressing the capabilities of the host language, the Banana implementation, and the needs of the application. How about 'instance', probably only used for StorageBanana? What "top-level" OPEN sequences are known? PB stuff (like 'call', and 'your-reference')? Are there any variations or versions that need to be known? We may add new functionality in the future, it might be useful for one end to know whether this functionality is available or not. (the PB 'call' sequence could some day take numeric argument names to convey positional parameters, a 'reference' sequence could take a string to indicate globally-visible PB URLs, it could become possible to pass target.remote_foo directly to a peer and have a callable RemoteMethod object pop out the other side). What "application-level" sequences are available? (Which RemoteInterface classes are known and valid in 'call' sequences? Which RemoteCopy names are valid for targets of the 'copy' sequence?). This is not necessarily within the realm of Banana negotiation, but applications may need to negotiate this sort of thing, and any disagreements will be manifested when Banana starts raising Violations, so it may be useful to include it in the Banana-level negotiation. On the other hand, negotiation is only useful if one side is prepared to accomodate a peer which cannot do some of the things it would prefer to use, or if it wants to know about the incapabilities so it can report a useful failure rather than have an obscure protocol-level error message pop up an hour later. So negotiation isn't the only goal: simple capability awareness is a useful lesser goal. It kind of makes sense for the first object of a stream to be a negotiation blob. We could make a new 'version' opentype, and declare that the contents will be something simple and forever-after-parseable (like a dict, with heavy constraints on the keys and values, all strings emitted in full). DONE, at least the framework is in place. Uses HTTP-style header-block exchange instead of banana sequences, with client-sends-first and server-decides. This correctly handles PB-vs-HTTP, but requires a timeout to detect oldpb clients vs newpb servers. No actual feature negotiation is performed yet, because we still only have the one version of the code. * connection initiation ** define PB URLs [newcred is the most important part of this, the URL stuff can wait] A URL defines an endpoint: a pb.Referenceable, with methods. Somewhere along the way it defines a transport (tcp+host+port, or unix+path) and an object reference (pathname). It might also define a RemoteInterface, or that might be put off until we actually invoke a method. URL = f("pb:", host, port, pathname) d = pb.callRemoteURL(URL, ifacename, methodname, args) probably give an actual RemoteInterface instead of just its name a pb.RemoteReference claims to provide access to zero-or-more RemoteInterfaces. You may choose which one you want to use when invoking callRemote. TODO: decide upon a syntax for URLs that refer to non-TCP transports pb+foo://stuff, pby://stuff (for yURL-style self-authenticating names) TODO: write the URL parser, implementing pb.getRemoteURL and pb.callRemoteURL DONE: use a Tub/PBService instead TODO: decide upon a calling convention for callRemote when specifying which RemoteInterface is being used. DONE, PB-URL is the way to go. ** more URLs relative URLs (those without a host part) refer to objects on the same Broker. Absolute URLs (those with a host part) refer to objects on other Brokers. SKIP, interesting but not really useful ** build/port pb.login: newcred for newpb Leave cred work for Glyph. has some enhanced PB cred stuff (challenge/response, pb.Copyable credentials, etc). URL = pb.parseURL("pb://lothar.com:8789/users/warner/services/petmail", IAuthorization) URL = doFullLogin(URL, "warner", "x8yzzy") URL.callRemote(methodname, args) NOTDONE * constrain ReferenceUnslicer properly The schema can use a ReferenceConstraint to indicate that the object must be a RemoteReference, and can also require that the remote object be capable of handling a particular Interface. This needs to be implemented. slicer.ReferenceUnslicer must somehow actually ask the constraint about the incoming tokens. An outstanding question is "what counts". The general idea is that RemoteReferences come over the wire as a connection-scoped ID number and an optional list of Interface names (strings and version numbers). In this case it is the far end which asserts that its object can implement any given Interface, and the receiving end just checks to see if the schema-imposed required Interface is in the list. This becomes more interesting when applied to local objects, or if a constraint is created which asserts that its object is *something* (maybe a RemoteReference, maybe a RemoteCopy) which implements a given Interface. In this case, the incoming object could be an actual instance, but the class name must be looked up in the unjellyableRegistry (and the class located, and the __implements__ list consulted) before any of the object's tokens are accepted. * decide upon what the "Shared" constraint should mean The idea of this one was to avoid some vulnerabilities by rejecting arbitrary object graphs. Fundamentally Banana can represent most anything (just like pickle), including objects that refer to each other in exciting loops and whorls. There are two problems with this: it is hard to enforce a schema that allows cycles in the object graph (indeed it is tricky to even describe one), and the shared references could be used to temporarily violate a schema. I think these might be fixable (the sample case is where one tuple is referenced in two different places, each with a different constraint, but the tuple is incomplete until some higher-level node in the graph has become referenceable, so [maybe] the schema can't be enforced until somewhat after the object has actually finished arriving). However, Banana is aimed at two different use-cases. One is kind of a replacement for pickle, where the goal is to allow arbitrary object graphs to be serialized but have more control over the process (in particular we still have an unjellyableRegistry to prevent arbitrary constructors from being executed during deserialization). In this mode, a larger set of Unslicers are available (for modules, bound methods, etc), and schemas may still be useful but are not enforced by default. PB will use the other mode, where the set of conveyable objects is much smaller, and security is the primary goal (including putting limits on resource consumption). Schemas are enforced by default, and all constraints default to sensible size limits (strings to 1k, lists to [currently] 30 items). Because complex object graphs are not commonly transported across process boundaries, the default is to not allow any Copyable object to be referenced multiple times in the same serialization stream. The default is to reject both cycles and shared references in the object graph, allowing only strict trees, making life easier (and safer) for the remote methods which are being given this object tree. The "Shared" constraint is intended as a way to turn off this default strictness and allow the object to be referenced multiple times. The outstanding question is what this should really mean: must it be marked as such on all places where it could be referenced, what is the scope of the multiple-reference region (per- method-call, per-connection?), and finally what should be done when the limit is violated. Currently Unslicers see an Error object which they can respond to any way they please: the default containers abandon the rest of their contents and hand an Error to their parent, the MethodCallUnslicer returns an exception to the caller, etc. With shared references, the first recipient sees a valid object, while the second and later recipient sees an error. * figure out Deferred errors for immutable containers Somewhat related to the previous one. The now-classic example of an immutable container which cannot be created right away is the object created by this sequence: t = ([],) t[0].append((t,)) This serializes into (with implicit reference numbers on the left): [0] OPEN(tuple) [1] OPEN(list) [2] OPEN(tuple) [3] OPEN(reference #0) CLOSE CLOSE CLOSE In newbanana, the second TupleUnslicer cannot return a fully-formed tuple to its parent (the ListUnslicer), because that tuple cannot be created until the contents are all referenceable, and that cannot happen until the first TupleUnslicer has completed. So the second TupleUnslicer returns a Deferred instead of a tuple, and the ListUnslicer adds a callback which updates the list's item when the tuple is complete. The problem here is that of error handling. In general, if an exception is raised (perhaps a protocol error, perhaps a schema violation) while an Unslicer is active, that Unslicer is abandoned (all its remaining tokens are discarded) and the parent gets an Error object. (the parent may give up too.. the basic Unslicers all behave this way, so any exception will cause everything up to the RootUnslicer to go boom, and the RootUnslicer has the option of dropping the connection altogether). When the error is noticed, the Unslicer stack is queried to figure out what path was taken from the root of the object graph to the site that had an error. This is really useful when trying to figure out which exact object cause a SchemaViolation: rather than being told a call trace or a description of the *object* which had a problem, you get a description of the path to that object (the same series of dereferences you'd use to print the object: obj.children[12].peer.foo.bar). When references are allowed, these exceptions could occur after the original object has been received, when that Deferred fires. There are two problems: one is that the error path is now misleading, the other is that it might not have been possible to enforce a schema because the object was incomplete. The most important thing is to make sure that an exception that occurs while the Deferred is being fired is caught properly and flunks the object just as if the problem were caught synchronously. This may involve discarding an otherwise complete object graph and blaming the problem on a node much closer to the root than the one which really caused the failure. * adaptive VOCAB compression We want to let banana figure out a good set of strings to compress on its own. In Banana.sendToken, keep a list of the last N strings that had to be sent in full (i.e. they weren't in the table). If the string being sent appears more than M times in that table, before we send the token, emit an ADDVOCAB sequence, add a vocab entry for it, then send a numeric VOCAB token instead of the string. Make sure the vocab mapping is not used until the ADDVOCAB sequence has been queued. Sending it inline should take care of this, but if for some reason we need to push it on the top-level object queue, we need to make sure the vocab table is not updated until it gets serialized. Queuing a VocabUpdate object, which updates the table when it gets serialized, would take care of this. The advantage of doing it inline is that later strings in the same object graph would benefit from the mapping. The disadvantage is that the receiving Unslicers must be prepared to deal with ADDVOCAB sequences at any time (so really they have to be stripped out). This disadvantage goes away if ADDVOCAB is a token instead of a sequence. Reasonable starting values for N and M might be 30 and 3. * write oldbanana compatibility code? An oldbanana peer can be detected because the server side sends its dialect list from connectionMade, and oldbanana lists are sent with OLDLIST tokens (the explicit-length kind). * add .describe methods to all Slicers This involves setting an attribute between each yield call, to indicate what part is about to be serialized. * serialize remotely-callable methods? It might be useful be able to do something like: class Watcher(pb.Referenceable): def remote_foo(self, args): blah w = Watcher() ref.callRemote("subscribe", w.remote_foo) That would involve looking up the method and its parent object, reversing the remote_*->* transformation, then sending a sequence which contained both the object's RemoteReference and the appropriate method name. It might also be useful to generalize this: passing a lambda expression to the remote end could stash the callable in a local table and send a Callable Reference to the other side. I can smell a good general-purpose object classification framework here, but I haven't quite been able to nail it down exactly. * testing ** finish testing of LONGINT/LONGNEG test_banana.InboundByteStream.testConstrainedInt needs implementation ** thoroughly test failure-handling at all points of in/out serialization places where BananaError or Violation might be raised sending side: Slicer creation (schema pre-validation? no): no no pre-validation is done before sending the object, Broker.callFinished, RemoteReference.doCall slicer creation is done in newSlicerFor .slice (called in pushSlicer) ? .slice.next raising Violation .slice.next returning Deferrable when streaming isn't allowed .sendToken (non-primitive token, can't happen) .newSlicerFor (no ISlicer adapter) top.childAborted receiving side: long header (>64 bytes) checkToken (top.openerCheckToken) checkToken (top.checkToken) typebyte == LIST (oldbanana) bad VOCAB key too-long vocab key bad FLOAT encoding top.receiveClose top.finish top.reportViolation oldtop.finish (in from handleViolation) top.doOpen top.start plus all of these when discardCount != 0 OPENOPEN send-side uses: f = top.reportViolation(f) receive-side should use it too (instead of f.raiseException) ** test failure-handing during callRemote argument serialization ** implement/test some streaming Slicers ** test producer Banana * profiling/optimization Several areas where I suspect performance issues but am unwilling to fix them before having proof that there is a problem: ** Banana.produce This is the main loop which creates outbound tokens. It is called once at connectionMade() (after version negotiation) and thereafter is fired as the result of a Deferred whose callback is triggered by a new item being pushed on the output queue. It runs until the output queue is empty, or the production process is paused (by a consumer who is full), or streaming is enabled and one of the Slicers wants to pause. Each pass through the loop either pushes a single token into the transport, resulting in a number of short writes. We can do better than this by telling the transport to buffer the individual writes and calling a flush() method when we leave the loop. I think Itamar's new cprotocol work provides this sort of hook, but it would be nice if there were a generalized Transport interface so that Protocols could promise their transports that they will use flush() when they've stopped writing for a little while. Also, I want to be able to move produce() into C code. This means defining a CSlicer in addition to the cprotocol stuff before. The goal is to be able to slice a large tree of basic objects (lists, tuples, dicts, strings) without surfacing into Python code at all, only coming "up for air" when we hit an object type that we don't recognize as having a CSlicer available. ** Banana.handleData The receive-tokenization process wants to be moved into C code. It's definitely on the critical path, but it's ugly because it has to keep calling into python code to handle each extracted token. Maybe there is a way to have fast C code peek through the incoming buffers for token boundaries, then give a list of offsets and lengths to the python code. The b128 conversion should also happen in C. The data shouldn't be pulled out of the input buffer until we've decided to accept it (i.e. the memory-consumption guarantees that the schemas provide do not take any transport-level buffering into account, and doing cprotocol tokenization would represent memory that an attacker can make us spend without triggering a schema violation). Itamar's CLineReceiver is a good example: you tokenize a big buffer as much as you can, pass the tokens upstairs to Python code, then hand the leftover tail to the next read() call. The tokenizer always works on the concatenation of two buffers: the tail of the previous read() and the complete contents of the current one. ** Unslicer.doOpen delegation Unslicers form a stack, and each Unslicer gets to exert control over the way that its descendents are deserialized. Most don't bother, they just delegate the control methods up to the RootUnslicer. For example, doOpen() takes an opentype and may return a new Unslicer to handle the new OPEN sequence. Most of the time, each Unslicer delegates doOpen() to their parent, all the way up the stack to the RootUnslicer who actually performs the UnslicerRegistry lookup. This provides an optimization point. In general, the Unslicer knows ahead of time whether it cares to be involved in these methods or not (i.e. whether it wants to pay attention to its children/descendants or not). So instead of delegating all the time, we could just have a separate Opener stack. Unslicers that care would be pushed on the Opener stack at the same time they are pushed on the regular unslicer stack, likewise removed. The doOpen() method would only be invoked on the top-most Opener, removing a lot of method calls. (I think the math is something like turning avg(treedepth)*avg(nodes) into avg(nodes)). There are some other methods that are delegated in this way. open() is related to doOpen(). setObject()/getObject() keep track of references to shared objects and are typically only intercepted by a second-level object which defines a "serialization scope" (like a single remote method call), as well as connection-wide references (like pb.Referenceables) tracked by the PBRootUnslicer. These would also be targets for optimization. The fundamental reason for this optimization is that most Unslicers don't care about these methods. There are far more uses of doOpen() (one per object node) then there are changes to the desired behavior of doOpen(). ** CUnslicer Like CSlicer, the unslicing process wants to be able to be implemented (for built-in objects) entirely in C. This means a CUnslicer "object" (a struct full of function pointers), a table accessible from C that maps opentypes to both CUnslicers and regular python-based Unslicers, and a CProtocol tokenization code fed by a CTransport. It should be possible for the python->C transition to occur in the reactor when it calls ctransport.doRead python->and then not come back up to Python until Banana.receivedObject(), at least for built-in types like dicts and strings. foolscap-0.13.1/doc/listings/0000755000076500000240000000000013204747603016432 5ustar warnerstaff00000000000000foolscap-0.13.1/doc/listings/copyable-receive.py0000644000076500000240000000173111562013212022207 0ustar warnerstaff00000000000000#! /usr/bin/python import sys from twisted.internet import reactor from foolscap.api import RemoteCopy, Tub # the receiving side defines the RemoteCopy class RemoteUserRecord(RemoteCopy): copytype = "unique-string-UserRecord" # this matches the sender def __init__(self): # note: our __init__ must take no arguments pass def setCopyableState(self, d): self.name = d['name'] self.age = d['age'] self.shoe_size = "they wouldn't tell us" def display(self): print "Name:", self.name print "Age:", self.age print "Shoe Size:", self.shoe_size def getRecord(rref, name): d = rref.callRemote("getuser", name=name) def _gotRecord(r): # r is an instance of RemoteUserRecord r.display() reactor.stop() d.addCallback(_gotRecord) from foolscap.api import Tub tub = Tub() tub.startService() d = tub.getReference(sys.argv[1]) d.addCallback(getRecord, "alice") reactor.run() foolscap-0.13.1/doc/listings/copyable-send.py0000644000076500000240000000210711562013212021514 0ustar warnerstaff00000000000000#! /usr/bin/python from twisted.internet import reactor from foolscap.api import Copyable, Referenceable, Tub # the sending side defines the Copyable class UserRecord(Copyable): # this class uses the default Copyable behavior typeToCopy = "unique-string-UserRecord" def __init__(self, name, age, shoe_size): self.name = name self.age = age self.shoe_size = shoe_size # this is a secret def getStateToCopy(self): d = {} d['name'] = self.name d['age'] = self.age # don't tell anyone our shoe size return d class Database(Referenceable): def __init__(self): self.users = {} def addUser(self, name, age, shoe_size): self.users[name] = UserRecord(name, age, shoe_size) def remote_getuser(self, name): return self.users[name] db = Database() db.addUser("alice", 34, 8) db.addUser("bob", 25, 9) tub = Tub() tub.listenOn("tcp:12345") tub.setLocation("localhost:12345") url = tub.registerReference(db, "database") print "the database is at:", url tub.startService() reactor.run() foolscap-0.13.1/doc/listings/pb2client.py0000644000076500000240000000135011562013212020650 0ustar warnerstaff00000000000000#! /usr/bin/python import sys from twisted.internet import reactor from foolscap.api import Tub def gotError1(why): print "unable to get the RemoteReference:", why reactor.stop() def gotError2(why): print "unable to invoke the remote method:", why reactor.stop() def gotReference(remote): print "got a RemoteReference" print "asking it to add 1+2" d = remote.callRemote("add", a=1, b=2) d.addCallbacks(gotAnswer, gotError2) def gotAnswer(answer): print "the answer is", answer reactor.stop() if len(sys.argv) < 2: print "Usage: pb2client.py URL" sys.exit(1) url = sys.argv[1] tub = Tub() tub.startService() d = tub.getReference(url) d.addCallbacks(gotReference, gotError1) reactor.run() foolscap-0.13.1/doc/listings/pb2server.py0000644000076500000240000000075511562013212020710 0ustar warnerstaff00000000000000#! /usr/bin/python from twisted.internet import reactor from foolscap.api import Referenceable, Tub class MathServer(Referenceable): def remote_add(self, a, b): return a+b def remote_subtract(self, a, b): return a-b myserver = MathServer() tub = Tub(certFile="pb2server.pem") tub.listenOn("tcp:12345") tub.setLocation("localhost:12345") url = tub.registerReference(myserver, "math-service") print "the object is available at:", url tub.startService() reactor.run() foolscap-0.13.1/doc/listings/pb3calculator.py0000644000076500000240000000252411562013212021530 0ustar warnerstaff00000000000000#! /usr/bin/python from twisted.application import service from twisted.internet import reactor from foolscap.api import Referenceable, Tub class Calculator(Referenceable): def __init__(self): self.stack = [] self.observers = [] def remote_addObserver(self, observer): self.observers.append(observer) def log(self, msg): for o in self.observers: o.callRemote("event", msg=msg) def remote_removeObserver(self, observer): self.observers.remove(observer) def remote_push(self, num): self.log("push(%d)" % num) self.stack.append(num) def remote_add(self): self.log("add") arg1, arg2 = self.stack.pop(), self.stack.pop() self.stack.append(arg1 + arg2) def remote_subtract(self): self.log("subtract") arg1, arg2 = self.stack.pop(), self.stack.pop() self.stack.append(arg2 - arg1) def remote_pop(self): self.log("pop") return self.stack.pop() tub = Tub() tub.listenOn("tcp:12345") tub.setLocation("localhost:12345") url = tub.registerReference(Calculator(), "calculator") print "the object is available at:", url application = service.Application("pb2calculator") tub.setServiceParent(application) if __name__ == '__main__': raise RuntimeError("please run this as 'twistd -noy pb3calculator.py'") foolscap-0.13.1/doc/listings/pb3user.py0000644000076500000240000000170211562013212020352 0ustar warnerstaff00000000000000#! /usr/bin/python import sys from twisted.internet import reactor from foolscap.api import Referenceable, Tub class Observer(Referenceable): def remote_event(self, msg): print "event:", msg def printResult(number): print "the result is", number def gotError(err): print "got an error:", err def gotRemote(remote): o = Observer() d = remote.callRemote("addObserver", observer=o) d.addCallback(lambda res: remote.callRemote("push", num=2)) d.addCallback(lambda res: remote.callRemote("push", num=3)) d.addCallback(lambda res: remote.callRemote("add")) d.addCallback(lambda res: remote.callRemote("pop")) d.addCallback(printResult) d.addCallback(lambda res: remote.callRemote("removeObserver", observer=o)) d.addErrback(gotError) d.addCallback(lambda res: reactor.stop()) return d url = sys.argv[1] tub = Tub() tub.startService() d = tub.getReference(url) d.addCallback(gotRemote) reactor.run() foolscap-0.13.1/doc/logging.rst0000644000076500000240000012767312766553111016777 0ustar warnerstaff00000000000000Foolscap Logging ================ Foolscap comes with an advanced event-logging package. This package is used internally to record connection establishment, remote message delivery, and errors. It can also be used by applications built on top of Foolscap for their own needs. This logging package includes a viewer application that processes locally saved log data or data retrieved over a foolscap connection, and displays a selected subset of the events. It also includes code to create a web page inside your application that presents the same kind of log view. Philosophy ---------- My background is in embedded systems, specifically routers, in which bugs and unexpected operations happen from time to time, causing problems. In this environment, storage space is at a premium (most routers do not have hard drives, and only a limited amount of RAM and non-volatile flash memory), and devices are often deployed at remote sites with no operator at the console. Embedded devices are expected to function properly without human intervention, and crashes or other malfunctions are rare compared to interactive applications. In this environment, when an error occurs, it is a good idea to record as much information as possible, because asking the operator to turn on extra event logging and then try to re-create the failure is only going to make the customer more angry ("my network has already broken once today, you want me to intentionally break it again?"). That one crash is the only chance you have to learn about the cause. In addition, as new features are being developed (or completed ones are being debugged), it is important to have visibility into certain internal state. Extra logging messages are added to illuminate this state, sometimes resulting in hundreds of messages per second. These messages are useful only while the problem is being investigated. Since most log formats involve flat text files, lots of additional log messages tend to obscure important things like unhandled exceptions and assertion failures, so once the messages have outlived their usefulness they are just getting in the way. Each message costs a certain amount of human attention, so we are motiviated to minimize that cost by removing the unhelpful messages. Logging also gets removed because it consumes CPU time, disk IO, disk space, or memory space. Many operations that can be done in linear time can expand to super-linear time if additional work is required to log the actions taking place, or the current state of the program. As a result, many formerly-useful log messages are commented out once they have served their purpose. Having been disabled, the cost to re-enable them if the bug should resurface is fairly high: at the very least it requires modifying the source code and restarting the program, and for some languages requires a complete recompile/rebuild. Even worse, to keep the source code readable, disabled log messages are frequently deleted altogether. After many months it may not be obvious where the log messages should be put back, and developers will need to re-acquaint themselves with the code base to find suitable places for those messages. To balance these costs, developers try to leave enough log messages in place that unexpected events will be captured with enough detail to start debugging, but not so many that it impacts performance or a human's ability to spot problems while scanning the logs. But it would be nice if certain log messages could be disabled or disregarded in a way that didn't abandon all of the work that went into developing and placing them. Memory-limited, strangeness-triggered log dumping ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Each potential log message could be described (to a human) as being useful in a particular set of circumstances. For example, if the program tried to read from a file and got a permission-denied error, it would be useful to know which file it was trying to read from, and how it came to decide upon that particular filename, and what user command was responsible for triggering this action. If a protocol parser which implements a state machine finds itself in an invalid state, it would be useful to know what series of input messages had arrived recently, to work backwards to the place where things started to go awry. Flip this around and you can phrase it as: the filename we compute will be interesting only if we get an error when we finally try to access it. Likewise, the series of input messages **would** be interesting to know if, at some point in the near future, an invalid protocol state is reached. The thing to note about these patterns is that an event at time T **causes** events before time T to become interesting. Interesting messages are worth keeping (and storing, and examining). Non-interesting messages are not worth as much, but there are different kinds of costs, and as the message becomes less interesting (or loses the potential to become interesting), we want to lower our costs. Displaying a message to a human is pretty expensive, since it tends to obscure other, more important messages. Storing messages is less expensive, depending upon how long we expect to store them (and how much storage space is available). Generating messages may be expensive or cheap, depending upon their frequency and complexity. Foolscap's logging library seeks to capture this ex-post-facto interestingness by categorizing messages into "severity levels", recording each level into a separate size-limited circular buffer, and provoking a dump of all buffers when an "Incident" occurs. An "Incident Qualifier" is used to classify certain higher-severity events as worthy of triggering the log dump. The idea is that, at any given time, we have a record (in memory) of a lot of low-frequency important things (like program startup, user actions, and unhandled exceptions), and only the most recent high-frequency events (verbose debugging data). But, if something strange happens, we dump everything we've got to disk, because it is likely that some of these noisy high-frequency not-normally-interesting events will be helpful to track down the cause of the unusual event. The viewing tools then rearrange all of these events into a linear order, and make it easy to filter out events by severity level or origin. Each severity level is (roughly) inversely proportional to a message rate. Assertion checks in code are in the "this should never happen" category, and their resulting low expected rate puts them in a high-severity level. Routine actions are expected to happen all the time, which puts them into a low-severity level. (you might think of severity as a separate axis than frequency. Severity would mean "how much damage will this cause". Frequency equals cost, controlling how long we keep the message around in the hopes of it becoming interesting. But things which cause a lot of damage should not be happening very frequently, and things which happen frequently must not cause a lot of damage. So these axes are sufficiently aligned for us to use just a single parameter for now.) Structured Logging ~~~~~~~~~~~~~~~~~~ The usual approach to event logging involves a single file with a sequence of lines of text, new events being continually appended at the end, perhaps with the files being rotated once they become too large or old. Typically the source code is peppered with lines like: .. code-block:: python log.msg(text) Log.Log(source, facility, severity, text) log.log_stacktrace() log.err(failure) Each such function call adds some more text to the logfile, encoding the various parameters into a new line. Using a text-based file format enables the use of certain unix tools like 'grep' and 'wc' to analyze the log entries, but frequently inhbits the use of more complex tools because they must first parse the human-readable lines back into the structured arguments that were originally passed to the log() call. Frequently, the free-form text portion of the log cannot be reliably distinguished from the stringified metadata (the quoting issue), making analysis tools more difficult to write. In addition, the desire to make both logfiles and the generating source code more greppable is occasionally at odds with clean code structure (putting everything on a single line) or refactoring goals (sending all logging for a given module through a common function). The Foolscap log system uses binary logfiles that accurately and reversibly serialize all the metadata associated with a given event. Tools are provided to turn this data into a human-readable greppable form, but better tools are provided to perform many of the same tasks that 'grep' is typically used for. For example, a log viewer can apply a python expression to each event as a filter, and the expression can do arbitrary comparison of event parameters (e.g. "show me all events related to failing uploads of files larger than 20MB"). To accomplish this, all unrecognized keyword arguments to the ``log.msg`` call are recorded as additional keys in the log event. To encourage structured usage, the message string be provided as a format specifier instead of a pre-interpolated string, using the keyword args as a formatting dictionary. Any time the string is displayed to a human, the keyword args are interpolated into the format string first. (in compiled languages, it would be useful and cheap to embed the source file and line number of the log() call inside the log event. Unfortunately, in Python, this would require expensive stack crawling, so developers are generally stuck with grepping for the log message in their source tree to backtrack from a log message to the code that generated it) Remote log aggregation ~~~~~~~~~~~~~~~~~~~~~~ Code is provided to allow a Foolscap-based application to easily publish a 'logport': an object which providers remote callers with access to that application's accumulated log data. Events are delivered over a secure Foolscap connection, to prevent eavesdroppers from seeing sensitive data inside the log messages themselves. This can be useful for a developer who wants to find out what just happened inside a given application, or who is about to do something to the application and wants to see how it responds from the inside. The ``flogtool tail`` tool is provided for this job. Each Tub always activates a logport, and a Tub option makes it possible to use a persistent FURL for remote access. (TODO: really?) The log-viewer application can either read log data from a local log directory, or it can connect to the logport on a remote host. A centralized "log gatherer" program can connect to multiple logports and aggregate all the logs collected from each, similar to the unix 'syslog' facility. This is most useful when the gatherer is configured to store more messages than the applications (perhaps it stores all of them), since it allows the costs to be shifted to a secondary machine with extra disk and fewer CPU-intensive responsibilities. To facilitate this, each Tub can either be given the FURL of a Log Gatherer, or the name of a file that might contain this FURL. This makes deployment easier: just copy the FURL of your central gatherer into this file in each of your application's working directories. A basic log gatherer is created by running ``flogtool create-gatherer`` and giving it a storage directory: this emits a gatherer FURL that can be used in the app configuration, and saves all incoming log events to disk. Causality Tracing ~~~~~~~~~~~~~~~~~ Log messages correspond to events. Events are triggered by other events. Sometimes the relationship between events is visible to the local programmer, sometimes it involves external hosts that can confuse the relationships. For local "application-level" causality, Foolscap's logging system makes it possible to define hierarchies of log events. Each call to ``log.msg`` returns an identifier (really just a number). If you pass this same identifier into a later ``log.msg`` call as the``parent=`` parameter, that second message is said to be a "child" of the first. This creates multiple trees of log events, in which the tree tops are the parentless messages. For example, a user command like "copy this file" could be a top-level event, while the various steps involved in copying the file (compute source filename, open source file, compute target filename, open target file, read data, write data, close) would be children of that top-level event. The viewer application has a way to hide or expand the nodes of these trees, to make it easy to look at just the messages that are related to a specific action. This lets you prioritize events along both severity (is this a common event?) and relevance (is this event related to the one of interest?) In the future, Foolscap's logging system will be enhanced to offer tools for analyzing causality relationships between multiple systems, taking inspiration from the E `Causeway `_ debugger. In this system, when one Tub sends a message to another, enough data is logged to enable a third party (with access to all the logs) to figure out the set of operations that were **caused** by the first message. Each message send is recorded, with an index that includes the TubID, current event number, and stack trace. Event A on tub 1 triggers event B on tub 2, along with certain operations and log messages. Event B triggers further operations, etc. The viewer application will offer a causality-oriented view in addition to the temporal one. Using Foolscap Logging ---------------------- The majority of your application's interaction with the Foolscap logging system will be in the form of calls to its ``log.msg`` function. Logging Messages From Application Code ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To emit log messages from application code, just use the ``foolscap.log.msg`` function: .. code-block:: python from foolscap.logging import log log.msg("hello world") You can add arguments that will be lazily evaluated and stringified by treating the log message as a normal format string: .. code-block:: python log.msg("queue depth %d exceeds limit %d", current_depth, limit) Or you can use keyword arguments instead. The format string can use positional parameters, or keyword arguments, but not both. .. code-block:: python log.msg(format="Danger %(name)s %(surname)s", name="Will", surname="Robinson") Passing arguments as separate parameters (instead of interpolating them before calling ``log.msg`` has the benefit of preserving more information: later, when you view the log messages, you can apply python filter expressions that use these parameters as search criteria. Regardless of how you format the main log message, you can always pass additional keyword arguments, and their values will be serialized into the log event. This will not be automatically stringified into a printed form of the message, but it will be available to other tools (either to filter upon or to display): .. code-block:: python log.msg("state changed", previous=states[now-1], current=stats[now]) Modifying Log Messages ^^^^^^^^^^^^^^^^^^^^^^ There are a number of arguments you can add to the ``log.msg`` call that foolscap will treat specially: .. code-block:: python parent = log.msg(facility="app.initialization", level=log.INFREQUENT, msg="hello world", stacktrace=False) log.msg(facility="app.storage", level=log.OPERATIONAL, msg="init storage", stacktrace=False, parent=parent) The ``level`` argument is how you specify a severity level, and takes a constant from the list defined in ``foolscap/log.py`` : - ``BAD`` : something which significantly breaks functionality. Unhandled exceptions and broken invariants fall into this category. - ``SCARY`` : something which is a problem, and shouldn't happen in normal operation, but which causes minimal functional impact, or from which the application can somehow recover. - ``WEIRD`` : not as much of a problem as SCARY, but still not right. - ``CURIOUS`` - ``INFREQUENT`` : messages which are emitted as a normal course of operation, but which happen infrequently, perhaps once every ten to one hundred seconds. User actions like triggering an upload or sending a message fall into this category. - ``UNUSUAL`` : messages which indicate events that are not normal, but not particularly fatal. Examples include excessive memory or CPU usage, minor errors which can be corrected by fallback code. - ``OPERATIONAL`` : messages which are emitted as a normal course of operation, like all the steps involved in uploading a file, potentially one to ten per second.. - ``NOISY`` : verbose debugging about small operations, potentially emitting tens or hundreds per second The ``stacktrace`` argument controls whether or not a stack trace is recorded along with the rest of the log message. The ``parent`` argument allows messages to be related to earlier messages. Logging Messages Through a Tub ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Each Tub offers a log method: this is just like the process-wide ``log.msg`` described above, but it adds an additional parameter named ``tubid`` . This is convenient during analysis, to identify which messages came from which applications. .. code-block:: python class Example: def __init__(self): self.tub = Tub() ... def query(self, args): self.tub.log("about to send query to server") self.server.callRemote("query", args).addCallback(self._query_done) Facilities ~~~~~~~~~~ Facility names are up to the application: the viewer app will show a list of checkboxes, one for each facility name discovered in the logged data. Facility names should be divided along functional boundaries, so that developers who do not care about, say, UI events can turn all of them off with a single click. Related facilities can be given names separated with dots, for example "ui.internationalization" and "ui.toolkit", and the viewer app may make it easy to enable or disable entire groups at once. Facilities can also be associated with more descriptive strings by calling ``log.explain_facility`` at least once: .. code-block:: python log.explain_facility("ui.web", "rendering pages for the web UI") "That Was Weird" Buttons ~~~~~~~~~~~~~~~~~~~~~~~~ Sometimes it is the user of your application who is in the best position to decide that something weird has taken place. Internal consistency checks are useful, but the user is the final judge of what meets their needs. So if they were expecting one thing to happen and something else happened instead, they should be able to declare that an Incident has taken place, perhaps by pushing a special "That Was Weird" button in your UI. To implement this sort of button for your user, just take the user's reason string and log it in an event at level WEIRD or higher. Since events at this level trigger Incidents by default, Foolscap's normal incident-handling behavior will take care of the rest for you. .. code-block:: python def that_was_weird_button_pushed(reason): log.msg(format="The user said that was weird: %(reason)s", reason=reason, level=log.WEIRD) Configuring Logging ------------------- Foolscap's logging system is always enabled, but the unconfigured initial state is lacking a number of useful features. By configuring the logging system at application startup, you can enable these features. Saving Log Events to Disk ~~~~~~~~~~~~~~~~~~~~~~~~~ The first missing piece is that it does not have a place to save log events in the event of something strange happening, so the short-term circular buffers are the only source of historical log events. To give the logging system some disk space to work with, just give it a logdir. The logging system will dump the circular buffers into this directory any time something strange happens, and both the in-memory buffers and the on-disk records are made available to viewing applications: .. code-block:: python from foolscap.logging import log log.setLogDir("~/saved-log-events") # == log.theLogger.setLogDir The foolscap logging code does not delete files from this directory. Applications which set up a logdir should arrange to delete old files once storage space becomes a problem. TODO: we could provide a maximum size for the logdir and have Foolscap automatically delete the oldest logfiles to stay under the size limit: this would make the disk-based logdir an extension of the memory-based circular buffers. Incidents ^^^^^^^^^ Foolscap's logging subsystem uses the term "Incident" to describe the "something strange" that causes the buffered log events to be dumped. The logger has an "Incident Qualifier" that controls what counts as an incident. The default qualifier simply fires on events at severity level ``log.WEIRD`` or higher. You can override the qualifier by subclassing ``foolscap.logging.incident.IncidentQualifier`` and calling ``log.setIncidentQualifier`` with an instance of your new class. For example, certain facilities might be more important than others, and you might want to declare an Incident for unusual but relatively low-severity events in those facilities: .. code-block:: python from foolscap.logging import log, incident class BetterQualifier(incident.IncidentQualifier): def check_event(self, ev): if ev.get('facility',"").startswith("lifesupport"): if ev['level'] > log.UNUSUAL: return True return incident.IncidentQualifier.check_event(self, ev) log.setIncidentQualifier(BetterQualifier()) The qualifier could also keep track of how many events of a given type had occurred, and trigger an incident if too many UNUSUAL events happen in rapid succession, or if too many recoverable errors are observed within a single operation. Once the Incident has been declared, the "Incident Reporter" is responsible for recording the recent events to the file on disk. The default reporter copies everything from the circular buffers into the logfiles, then waits an additional 5 seconds or 100 events (whichever comes first), recording any trailing events into the logfile too. The idea is to capture the application's error-recovery behavior: if the application experiences a problem, it should log something at the ``log.WEIRD`` level (or similar), then attempt to fix the problem. The post-trigger trailing event logging code should capture the otherwise-ordinary events performed by this recovery code. Overlapping incidents will be combined: if an incident reporter is already active when the qualifier sees a new triggering event, that event is just added to the existing reporter. The incident reporter can be overridden as well, by calling ``log.setIncidentReporterFactory`` with a **class** that will produce reporter instances. For example, if you wanted to increase the post-trigger event recording to 1000 events or 10 seconds, then you could do something like this: .. code-block:: python from foolscap.logging import log, incident class MoreRecoveryIncidentReporter(incident.IncidentReporter): TRAILING_DELAY = 10.0 TRAILING_EVENT_LIMIT = 1000 log.setIncidentReporterFactory(MoreRecoveryIncidentReporter) Recorded Incidents will be saved in the logdir with filenames like ``incident-2008-05-02--01-12-35Z-w2qn32q.flog.bz2`` , containing both a (UTC) timestamp and a random/unique suffix. These can be read with tools like ``flogtool dump`` and ``flogtool web-viewer`` . Setting up the logport ~~~~~~~~~~~~~~~~~~~~~~ The ``logport`` is a ``foolscap.Referenceable`` object which provides access to all available log events. Viewer applications can either retrieve old events (buffered in RAM or on disk), or subscribe to hear about new events that occur later. The logport implements the ``foolscap.logging.interfaces.RILogPublisher`` interface, which defines the methods that can be called on it. Each Tub automatically creates and registers a logport: the ``tub.getLogPort()`` and ``tub.getLogPortFURL()`` methods make it possible to grant access to others: .. code-block:: python t = Tub() ... # usual Tub setup: startService, listenOn, setLocation logport_furl = t.getLogPortFURL() # this is how you learn the logport furl print "please point your log viewer at: %s" % logport_furl logport = t.getLogPort() # a Referenceable you can pass over the wire rref.callRemote("please_use_my_logport", logport) The default behavior is register the logport object with an ephemeral name, and therefore its FURL will change from one run of the program to the next. This can be an operational nuisance, since the external log viewing program you're running (``flogtool tail LOGPORT`` ) would need a new FURL each time the target program is restarted. By giving the logport a place to store its FURL between program runs, the logport gets a persistent name. The ``logport-furlfile`` option is used to identify this file. If the file exists, the desired FURL will be read out of it. If it does not, the newly-generated FURL will be written into it. If you use ``logport-furlfile`` , it must be set before you call ``getLogPortFURL`` (and also before you pass the result of ``getLogPort`` over the wire), otherwise an ephemeral name will have already been registered and the persistent one will be ignored. The call to ``setOption`` can take place before ``setLocation`` , and the logport-furlfile will be created as soon as both the filename and the location hints are known. However, note that the logport will not be available until after ``setLocation`` is called: ``getLogPortFURL`` and ``getLogPort`` will raise exceptions. .. code-block:: python tub.setOption("logport-furlfile", "~/logport.furl") print "please point your log viewer at: %s" % tub.getLogPortFURL() This ``logport.furl`` file can be read directly by other tools if you want to point them at an operating directory rather than the actual logport FURL. For example, the ``flogtool tail`` command (described below) can accept either an actual FURL, or the directory in which a file named ``logport.furl`` can be located, making it easier to examine the logs of a local application. Note that the ``logport-furlfile`` is chmod'ed ``go-r`` , since it is a secret: the idea is that only people with access to the application's working directory (and presumeably to the application itself) should get access to the logs. Configuring a Log Gatherer ~~~~~~~~~~~~~~~~~~~~~~~~~~ The third feature that requires special setup is the log gatherer. You can either tell the Tub a specific gatherer to use, or give it a filename where the FURL of a log gatherer is stored. The ``tub.setOption("log-gatherer-furl", gatherer_FURL)`` call can be used to have the Tub automatically connect to the log gatherer and offer its logport. The Tub uses a Reconnector to make sure the gatherer connection is reestablished each time it gets dropped. .. code-block:: python t = Tub() t.setOption("log-gatherer-furl", gatherer_FURL) Alternatively, you can use the ``tub.setOption("log-gatherer-furlfile", "~/gatherer.furl")`` call to tell the Tub about a file where a gatherer FURL might be found. If that file exists, the Tub will read a FURL from it, otherwise the Tub will not use a gatherer. The file can contain multiple log-gatherer FURLs, one per line. This is probably the easiest deployment mode: .. code-block:: python t = Tub() t.setOption("log-gatherer-furlfile", "~/gatherer.furl") In both cases, the gatherer FURL is expected to point to a remote object which implements the ``foolscap.logging.RILogGatherer`` interface (such as the service created by ``flogtool create-gatherer`` ). The Tub will connect to the gatherer and offer it the logport. The ``log-gatherer-furl`` and ``log-gatherer-furlfile`` options can be set at any time, however the connection to the gatherer will not be initiated until ``setLocation`` is called. Interacting With Other Logging Systems ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ There are two other logging systems that the Foolscap logging code knows how to handle: ``twisted.python.log`` and the stdlib ``logging`` system. First, a brief discussion of the single-instance nature of Foolscap's logging is in order. Each process that uses Foolscap gets a single instance of the Foolscap logging code (named ``theLogger`` and defined at module level in ``foolscap.logging.log`` ). This maintains a single logdir. Each time a process is started it gets a new "incarnation record", which consists of a randomly generated (unique) number, and (if a logdir is available) (TODO) a continuously incrementing sequence number. All log events are tagged with this incarnation record: it is used to distinguish between event#23 in one process versus the same event number from a different process. Each Tub has a distinct TubID, and all log events that go through the Tub (via ``tub.log`` ) are tagged with this TubID. Each Tub maintains its own logport (specifically there is a single ``LogPublisher`` object, but like all Referenceables it can be registered in multiple Tubs and gets a distinct FURL for each one). twisted.python.log ^^^^^^^^^^^^^^^^^^ Twisted's logging mechanism is used by importing ``twisted.python.log`` and invoking its ``log.msg()`` and ``log.err`` methods. This mechanism is used extensively by Twisted itself; the most important messages are those concerning "Unhandled Error in Deferred" and other exceptions in processing received data and timed calls. The normal destination for Twisted log messages depends upon how the application is run: the ``twistd`` daemonization tool sends the log messages to a file named ``twistd.log`` , the ``trial`` unit-test tool puts them in ``_trial_temp/test.log`` , and standalone scripts discard these logs by default (unless you use something like ``log.startLogging(sys.stderr)`` ). To capture these log messages, you need a "bridge", which will add a Twisted log observer and copy each Twisted log message into Foolscap. There can be at most one such bridge per python process. Either you will use a generic bridge (which tags each message with the incarnation record), or you will use a Tub as a bridge (which additionally tags each message with the TubID). Each time you set the twisted log bridge, any previous bridge is discarded. When you have only one Tub in an application, use the Tub bridge. Likewise if you have multiple Tubs but there is one that is long-lived, use that Tub for the bridge. If you have mutiple Tubs with no real primary one, use the generic bridge. Using a Tub bridge adds slightly more information to the log events, and may make it a bit easier to correlate Twisted log messages with actions of your application code, especially when you're combining events from several applications together for analysis. To set up the generic bridge, use the following code: .. code-block:: python from foolcap.logging import log log.bridgeTwistedLogs() To set up a Tub bridge, use this instead: .. code-block:: python t = Tub() t.setOption("bridge-twisted-logs", True) Note that for Tub bridges, the Twisted log messages will only be delivered while the Tub is running (specifically from the time its startService() method is until its stopService() method is called). TODO: review this behavior, we want earlier messages to be bridged too. To bridge log events in the other direction (i.e. taking foolscap log messages and copying them into twisted), use the ``log.bridgeLogsToTwisted()`` call, or the ``FLOGTOTWISTED`` environment variable. This is useful to get foolscap.logging.log.msg() events copied into ``twistd.log`` . The default filter only bridges non-noisy events (i.e. those at level OPERATIONAL or higher), and does not bridge foolscal internal events. You might use this if you don't buy into the foolscap logging philosophy and really want log events to be continually written out to disk. You might also use it if you want a long-term record of operationally-significant events, or a record that will survive application crashes which don't get handled by the existing Incident-recording mechanism. .. code-block:: python from foolscap.logging import log log.bridgeLogsToTwisted() stdlib 'logging' module ^^^^^^^^^^^^^^^^^^^^^^^ stdlib ``logging`` messages must be bridged in the same way. TODO: define and implement the bridge setup Preferred Logging API ^^^^^^^^^^^^^^^^^^^^^ To take advantage of the parent/child causality mechanism, you must use Foolscap's native API. (to be precise, you can pass in ``parent=`` to either Twisted's ``log.msg`` or stdlib's ``logging.log`` , but to get a handle to use as a value to ``parent=`` you must use ``foolscap.log.msg`` , because neither stdlib's nor Twisted's log calls provide a return value) Controlling Buffer Sizes ~~~~~~~~~~~~~~~~~~~~~~~~ There is a separate circular buffer (with some maximum size) for each combination of level and facility. After each message is added, the size of the buffer is checked and enough old messages are discarded to bring the size back down to the limit. Each facility uses a separate set of buffers, so that e.g. the NOISY messages from the "ui" facility do not evict the NOISY messages from the "upload" facility. The sizes of these buffers can be controlled with the ``log.set_buffer_size`` function, which is called with the severity level, the facility name, and the desired buffer size (maximum number of messages). If ``set_buffer_size`` is called without a facility name, then it will set the default size that will be used when a log.msg call references an as-yet-unknown facility). .. code-block:: python log.set_buffer_size(log.NOISY, 10000) log.set_buffer_size(level=log.NOISY, facility="upload", size=10000) log.allocate_facility_buffers("web") print log.get_buffer_size(log.NOISY, facility="upload") Some Messages Are Not Worth Generating ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If the message to be logged is below some threshold, it will not even be generated. This makes it easy to leave the log line in the source code, but not consume CPU time or memory space by actually using it. Such messages must be enabled before use (either through the logport (TODO) or by restarting the application with different log settings(TODO)), but at least developers will not have to re-learn the source code to figure out where it might be useful to add some messages. This threshold can be configured for all facilities at the same time, or on a facility-by-facility basis. .. code-block:: python log.set_generation_threshold(log.NOISY) log.set_generation_threshold(level=log.OPERATIONAL, facility="web") print log.get_generation_threshold() print log.get_generation_threshold(facility="web") Viewing Log Messages -------------------- There are a variety of ways for humans (and their tools) to read and analyze log messages. The ``flogtool`` program, provided with Foolscap, provides access to many of them. - ``flogtool dump`` : look at the saved log events (in a logdir) and display their contents to stdout. Options are provided to specify the log source, the facilities and severity levels to display, and grep-like filters on the messages to emit. - ``flogtool tail`` : connect to a logport and display new log events to stdout. The ``--catchup`` option will also display old events. - ``flogtool gtk-viewer`` : a Gtk-based graphical tool to examine log messages. - ``flogtool web-viewer`` : runs a local web server, through which log events can be examined. This tool uses a log-viewing API defined in ``foolscap/logging/interfaces.py`` . (TODO) Application code can use the same API to get access to log messages from inside a python program. Log Views ~~~~~~~~~ (NOTE: this section is incomplete and has not been implemented) Many of these tools share the concept of "Log Views". This is a particular set of filters which can be applied to the overall log event stream. For example, one view might show all events that are UNUSUAL or worse. Another view might show NOISY messages for the "ui" facility but nothing else. Each view is described by a set of thresholds: each facility gets a severity threshold, and all messages at or above the threshold will be included in the view. While in principle there is a threshold for each facility, this may be expressed as a single generic threshold combined with overrides for a few specific facilities. Log Observers ~~~~~~~~~~~~~ A "Log Observer" can be attached to a foolscap-using program (either internally or by subscribing through the flogport). Once attached, this observer will receive a stream of log messages, which the observer is then free to format, store, or ignore as it sees fit. Each log message is a dictionary, as defined in doc/specifications/logfiles . .. code-block:: python def observe(event): print strftime(fmt, event.timestamp) print event["level"] # a number print event.get("facility" # a string like "ui" print event["message"] # a unicode object with the actual event text log.theLogger.addObserver(observe) Running a Log Gatherer ~~~~~~~~~~~~~~~~~~~~~~ A "Log Gatherer" is a python server to which the process under examination sends some or all of its log messages. These messages are saved to a file as they arrive, so they can be examined later. The resulting logfiles can be compressed, and they can be automatically rotated (saved, rename, reopened) on a periodic interval. In addition, sending a SIGHUP to the gatherer will cause it to rotate the logfiles. To create one, choose a new directory for it to live in, and run "``flogtool create-gatherer`` ". You can then start it with "twistd", and stop it by using the ``twistd.pid`` file: .. code-block:: console % flogtool create-gatherer lg Gatherer created in directory lg Now run '(cd lg && twistd -y gatherer.tac)' to launch the daemon % cd lg % ls gatherer.tac % twistd -y gatherer.tac % ls from-2008-07-28--13-30-34Z--to-present.flog log_gatherer.furl twistd.pid gatherer.pem portnum gatherer.tac twistd.log % cat log_gatherer.furl pb://g7yntwfu24w2hhb54oniqowfgizpk73d@192.168.69.172:54611,127.0.0.1:54611/z4ntcdg4jpdg3pnabhmyu3qvi3a7mdp3 % kill `cat twistd.pid` % The ``log_gatherer.furl`` string is the one that should be provided to all applications whose logs should be gathered here. By using ``tub.setOption("log-gatherer-furlfile", "log_gatherer.furl")`` in the application, you can just copy this .furl file into the application's working directory. Running an Incident Gatherer ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ An "Incident Gatherer" is like a Log Gatherer, but it only gathers weirdness-triggered Incidents. It records these incidents into files on the local disk, and provides access to them through a web server. The Incident Gatherer can also be configured to classify the incidents into various categories (perhaps expressions of a specific bug), to facilitate analysis by separating known problems from new ones. To create one, choose a new directory for it to live in, and run "``flogtool create-incident-gatherer`` ", just like the log gatherer: .. code-block:: console % flogtool create-incident-gatherer ig Gatherer created in directory ig Now run '(cd ig && twistd -y gatherer.tac)' to launch the daemon % cd ig % ls gatherer.tac % twistd -y gatherer.tac % Incident Storage ^^^^^^^^^^^^^^^^ Inside the gatherer's base directory (which we refer to as BASEDIR here), the ``incidents/`` directory will contain a subdirectory for each tub that connects to the gatherer. Each subdir will contain the incident files, named ``incident-TIMESTAMP-UNIQUE.flog.bz2`` . A simple unix command like ``find BASEDIR/incidents -name 'incident-*.flog.bz2'`` will locate all incident files. Each incident file can be examined with a tool like ``flogtool dump`` . The format is described in the doc/specifications/logfiles docs. Classification ^^^^^^^^^^^^^^ The Incident Gatherer uses a collection of user-supplied classification functions to analyze each Incident and place it into one or more categories. To add a classification function, create a file with a name like "``classify_*.py`` " (such as ``classify_foolscap.py`` or ``classify_db.py`` ), and define a function in it named "``classify_incident()`` ". Place this file in the gatherer's directory. All such files will be loaded and evaluated when the gatherer starts. The ``classify_incident()`` function will accept a single triggering event (a regular log Event dictionary, see logfiles.xhtml for details, which can be examined as follows: .. code-block:: python def classify_incident(trigger): m = trigger.get('message', '') if "Tub.connectorFinished:" in m: return 'foolscap-tubconnector' The function should return a list (or set) of categories, or a single category string, or None. Each incident can wind up in multiple categories. If no function finds a category for the incident, it will be added to the "unknown" category. All incidents are added to the "all" category. The ``classified/`` directory will contain a file for each defined classification. This file will contain one line for each incident that falls into that category, containing the BASEDIR-relative pathname of the incident file (i.e. each line will look like ``incidents/TUBID/incident-TIMESTAMP-UNIQUE.flog.bz2`` ). The ``classified/all`` file will contain the same filenames as the ``find`` command described earlier. If the ``classified/`` directory does not exist when the gatherer is started, all stored Incidents will be re-classified. After modifying or adding classification functions, you should delete the ``classified/`` directory and restart the gatherer. Incident Gatherer Web Server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The Incident Gatherer can run a small webserver, to publish information about the incidents it collects. The plan is to have it publish an RSS feed of incidents by category, and to serve incidents as HTML just like the ``foolscap web-viewer`` command. This code is not yet written. Incident Reports by Email ^^^^^^^^^^^^^^^^^^^^^^^^^ The Incident Gatherer can also be configured to send email with a description of the incident for various categories. The incident report will be included as an attachment for further analysis. This code is not yet written. Incomplete And Misleading Notes On stdlib 'Logging' Module --------- (NOTE: this section is incomplete and has not been implemented. In addition it may be entirely false and misleading.) The Python stdlib ``logging`` module offers portions of the desired functionality. The Foolscap logging framework is built as an extension to the native Python facilities. The ``logging`` module provides a tree of facilities, one ``Logger`` instance per facility (in which the child path names are joined with periods to form the Logger's name). Each ``Logger`` gets a set of ``Handlers`` which receive all messages sent to that ``Logger`` or below; the ``Handlers`` attached to the root ``Logger`` see all messages. Each message arrives as a ``LogRecord`` instance, and handlers are responsible for formatting them into text or a record on disk or whatever is necessary. Each log message has a severity (from DEBUG at 10 up to CRITICAL at 50), and both ``Loggers`` and ``Handlers`` have thresholds to discard low-severity messages. ``logging`` Plan of attack: foolscap installs a root Logger handler, with a threshold set very low (0), so it gets everything. The root Logger is set to a low threshold (since it defaults to WARNING=30), to make sure that all events are passed through to its handlers. Foolscap's handler splits the events it receives out by facility (Logger name) and severity level, and appends them to a space-limited buffer (probably a dequeue). That covers all native users of logging.py . Foolscap users deal with foolscap.log.msg(), which massages the arguments before passing them through to logging.log(). In particular, each log message processed by the foolscap handler gets a serial number assigned to it. This number is used as a marker, which can be passed to later msg() calls. The foolscap.log.msg code manages these serial numbers and uses them to construct the call to logging.log(), then the foolscap handler pulls the serial number out of the event and records it. foolscap-0.13.1/doc/schema.rst0000644000076500000240000005700212766553111016575 0ustar warnerstaff00000000000000Foolscap Schemas ================ *NOTE! This is all preliminary and is more an exercise in semiconscious protocol design than anything else. Do not believe this document. This sentence is lying. So there.* Existing ``Constraint`` classes -------------------------------- +-----------------------------------------------------------------------------+--------------+-------------------------------------------------------------------------------+ | class name | shortcut | \ | +=============================================================================+==============+===============================================================================+ | ``Any()`` | \ | accept anything | +-----------------------------------------------------------------------------+--------------+-------------------------------------------------------------------------------+ | ``StringConstraint(maxLength=1000)`` | ``str`` | string of up to maxLength characters (maxLength=None means | | | | unlimited), or a VOCAB sequence of any length | +-----------------------------------------------------------------------------+--------------+-------------------------------------------------------------------------------+ | ``IntegerConstraint(maxBytes=-1)`` | ``int`` | integer. maxBytes=-1 means s_int32_t, =N means LONGINT which can be | | | | expressed in N or fewer bytes (i.e. abs(num) < 2**(8*maxBytes)), | | | | =None means unlimited. NOTE: shortcut 'long' is like shortcut 'int' but | | | | maxBytes=1024. | +-----------------------------------------------------------------------------+--------------+-------------------------------------------------------------------------------+ | ``NumberConstraint(maxBytes=1024)`` | ``float`` | integer or float. Integers are limited by maxBytes as in | | | | ``IntegerConstraint`` , floats are fixed size. | +-----------------------------------------------------------------------------+--------------+-------------------------------------------------------------------------------+ | ``BooleanConstraint(value=None)`` | ``bool`` | True or False. If value=True, only accepts True. If value=False, | | | | only accepts False. NOTE: value= is a very silly parameter. | +-----------------------------------------------------------------------------+--------------+-------------------------------------------------------------------------------+ | ``InterfaceConstraint(iface)`` | Interface | TODO. UNSAFE. Accepts an instance which claims to implement the given | | | | Interface. The shortcut is simply any Interface subclass. | +-----------------------------------------------------------------------------+--------------+-------------------------------------------------------------------------------+ | ``ClassConstraint`` | Class | TODO. UNSAFE. Accepts an instance which claims to be of the given | | | | class name. The shortcut is simply any (old-style) class object. | +-----------------------------------------------------------------------------+--------------+-------------------------------------------------------------------------------+ | ``PolyConstraint(*alternatives)`` | (alt1, alt2) | Accepts any object which obeys at least one of the alternative | | | | constraints provided. Implements a logical OR function of the given | | | | constraints. Also known as ``ChoiceOf`` . | +-----------------------------------------------------------------------------+--------------+-------------------------------------------------------------------------------+ | ``TupleConstraint(*elemConstraints)`` | \ | Accepts a tuple of fixed length with elements that obey the given | | | | constraints. Also known as ``TupleOf`` . | +-----------------------------------------------------------------------------+--------------+-------------------------------------------------------------------------------+ | ``ListConstraint(elemConstraint, maxLength=30)`` | \ | Accepts a list of up to maxLength items, each of which obeys the | | | | element constraint provided. Also known as ``ListOf`` . | +-----------------------------------------------------------------------------+--------------+-------------------------------------------------------------------------------+ | ``DictConstraint(keyConstraint, valueConstraint, maxKeys=30)`` | \ | Accepts a dictionary of up to maxKeys items. Each key must obey | | | | keyConstraint and each value must obey valueConstraint. Also known | | | | as ``DictOf`` . | +-----------------------------------------------------------------------------+--------------+-------------------------------------------------------------------------------+ | ``AttributeDictConstraint(*attrTuples, **kwargs)`` | \ | Constrains dictionaries used to describe instance attributes, as used | | | | by RemoteCopy. Each attrTuple is a pair of (attrname, constraint), used | | | | to constraint individual named attributes. kwargs['attributes'] | | | | provides the same control. kwargs['ignoreUnknown'] is a boolean flag | | | | which indicates that unknown attributes in inbound state should simply | | | | be dropped. kwargs['acceptUnknown'] indicates that unknown attributes | | | | should be accepted into the instance state dictionary. | +-----------------------------------------------------------------------------+--------------+-------------------------------------------------------------------------------+ | ``RemoteMethodSchema(method=None, _response=None, __options=[], **kwargs)`` | \ | Constrains arguments and return value of a single remotely-invokable | | | | method. If method= is provided, the ``inspect`` module is used | | | | to extract constraints from the method itself (positional arguments are | | | | not allowed, default values of keyword arguments provide constraints for | | | | each argument, the results of running the method provide the return value | | | | constraint). If not, most kwargs items provide constraints for method | | | | arguments, _response provides a constraint for the return value. | | | | __options and additional kwargs keys provide neato whiz-bang future | | | | expansion possibilities. | +-----------------------------------------------------------------------------+--------------+-------------------------------------------------------------------------------+ | ``Shared(constraint, refLimit=None)`` | \ | TODO. Allows objects with refcounts no greater than refLimit (=None | | | | means unlimited). Wraps another constraint, which the object must obey. | | | | refLimit=1 rejects shared objects. | +-----------------------------------------------------------------------------+--------------+-------------------------------------------------------------------------------+ | ``Optional(constraint, default)`` | \ | TODO. Can be used to tag Copyable attributes or (maybe) method | | | | arguments. Wraps another constraint. If an object is provided, it must | | | | obey the constraint. If not provided, the default value will be given in | | | | its place. | +-----------------------------------------------------------------------------+--------------+-------------------------------------------------------------------------------+ | ``FailureConstraint()`` | \ | Constrains the contents of a CopiedFailure. | +-----------------------------------------------------------------------------+--------------+-------------------------------------------------------------------------------+ :: """ RemoteReference objects should all be tagged with interfaces that they implement, which point to representations of the method schemas. When a remote method is called, Foolscap should look up the appropriate method and serialize the argument list accordingly. We plan to eliminate positional arguments, so local RemoteReferences use their schema to convert callRemote calls with positional arguments to all-keyword arguments before serialization. Conversion to the appropriate version interface should be handled at the application level. Eventually, with careful use of twisted.python.context, we might be able to provide automated tools for helping application authors automatically convert interface calls and isolate version-conversion code, but that is probably pretty hard. """ class Attributes: def __init__(self,*a,**k): pass X = Attributes( ('hello', str), ('goodbye', int), # Allow the possibility of multiple or circular references. The default # is to make multiple copies to avoid making the serializer do extra # work. ('next', Shared(Narf)), ('asdf', ListOf(Narf, maxLength=30)), ('fdsa', (Narf, String(maxLength=5), int)), ('qqqq', DictOf(str, Narf, maxKeys=30)), ('larg', AttributeDict(('A', int), ('X', Number), ('Z', float))), Optional("foo", str), Optional("bar", str, default=None), ignoreUnknown=True, ) X = Attributes( attributes={ 'hello': str, # this form doesn't allow Optional() 'goodbye': int, }, Optional("foo", str), # but both can be used at once ignoreUnknown=True) class Narf(Remoteable): # step 1 __schema__ = X # step 2 (possibly - this loses information) class schema: hello = str goodbye = int class add: x = Number y = Number __return__ = Copy(Number) class getRemoteThingy: fooID = Arg(WhateverID, default=None) barID = Arg(WhateverID, default=None) __return__ = Reference(Narf) # step 3 - this is the only example that shows argument order, which we # _do_ need in order to translate positional arguments to callRemote, so # don't take the nested-classes example too seriously. schema = """ int add (int a, int b) """ # Since the above schema could also be used for Formless, or possibly for # World (for state) we can also do: class remote_schema: """blah blah """ # You could even subclass that from the other one... class remote_schema(schema): dontUse = 'hello', 'goodbye' def remote_add(self, x, y): return x + y def rejuvinate(self, deadPlayer): return Reference(deadPlayer.bringToLife()) # "Remoteable" is a new concept - objects which may be method-published # remotely _or_ copied remotely. The schema objects support both method # / interface definitions and state definitions, so which one gets used # can be defined by the sending side as to whether it sends a # Copy(theRemoteable) or Reference(theRemoteable) # (also, with methods that are explicitly published by a schema there is # no longer technically any security need for the remote_ prefix, which, # based on past experience can be kind of annoying if you want to # provide the same methods locally and remotely) # outstanding design choice - Referenceable and Copyable are subclasses # of Remoteable, but should they restrict the possibility of sending it # the other way or def getPlayerInfo(self, playerID): return CopyOf(self.players[playerID]) def getRemoteThingy(self, fooID, barID): return ReferenceTo(self.players[selfPlayerID]) class RemoteNarf(Remoted): __schema__ = X # or, example of a difference between local and remote class schema: class getRemoteThingy: __return__ = Reference(RemoteNarf) class movementUpdate: posX = int posY = int # No return value __return__ = None # Don't wait for the answer __wait__ = False # Feel free to send this over UDP __reliable__ = False # but send in order! __ordered__ = True # use priority queue / stream 3 __priority__ = 3 # allow full serialization of failures __failure__ = FullFailure # default: trivial failures, or str or int __failure__ = ErrorMessage # These options may imply different method names - e.g. '__wait__ = # False' might imply that you can't use callRemote, you have to # call 'sendRemote' instead... __reliable__ = False might be # 'callRemoteUnreliable' (longer method name to make it less # convenient to call by accident...) ## (and yes, donovan, we know that TypedInterface exists and we are going to ## use it. we're just screwing around with other syntaxes to see what about PB ## might be different.) Common banana sequences: A reference to a remote object. (On the sending side: Referenceable or ReferenceTo(aRemoteable) On the receiving side: RemoteReference) ('remote', reference-id, interface, version, interface, version, ...) A call to a remote method: ('fastcall', request-id, reference-id, method-name, 'arg-name', arg1, 'arg-name', arg2) A call to a remote method with extra spicy metadata: ('call', request-id, reference-id, interface, version, method-name, 'arg-name', arg1, 'arg-name', arg2) Special hack: request-id of 0 means 'do not answer this call, do not acknowledge', etc. Answer to a method call: ('answer', request-id, response) ('error', request-id, response) Decrement a reference incremented by 'remote' command: ('decref', reference-id) Broker currently has 9 proto_ methods: _version(vnum): accept a version number, compare to ours, reject if different _didNotUnderstand(command): log command, maybe drop connection _message(reqID, objID, message, answerRequired, netArgs, netKw): _cachemessage (like _message but finds objID with self.cachedLocallyAs instead of self.localObjectForID, used by RemoteCacheMethod and RemoteCacheObserver) look up objID, invoke it with .remoteMessageReceived(message, args), send "answer"(reqID, results) _answer(reqID, results): look up self.waitingForAnswers[reqID] and fire callback with results _error(reqID, failure): lookup waitingForAnswers, fire errback _decref(objID): dec refcount of self.localObjects[objID]. Sent in RemoteReference.__del__ _decache(objID): dec refcount of self.remotelyCachedObjects[objID] _uncache(objID): remove obj from self.locallyCachedObjects[objID] stuff ----- A RemoteReference/RemoteCopy (called a Remote for now) has a schema attached to it. remote.callRemote(methodname, *args) does schema.getMethodSchema(methodname) to obtain a MethodConstraint that describes the individual method. This MethodConstraint (or MethodSchema) has other attributes which are used by either end: what arguments are allowed and/or expected, calling conventions (synchronous, in-order, priority, etc), and how the return value should be constrained. To use the Remote like a RemoteCopy ... :: Remote: .methods .attributes .getMethodSchema(methodname) -> MethodConstraint .getAttributeSchema(attrname) -> a Constraint XPCOM idl specifies methods and attributes (readonly, readwrite). A Remote object which represented a distant XPCOM object would have a Schema that is created by parsing the IDL. Its callRemote would do the appropriate marshalling. Issue1: XPCOM lets methods have in/out/inout parameters.. these must be detected and a wrapper generated. Issue2: what about attribute set/get operations? Could add setRemote and getRemote for these. --- Some of the schema questions come down to how PBRootSlicer should deal with instances. The question is whether to treat the instance as a Referenceable (copy-by-reference: create and transmit a reference number, which will be turned into a RemoteReference on the other end), or as a Copyable (copy-by-value: collect some instance state and send it as an instance). This decision could be made by looking at what the instance inherits from: if isinstance(obj, pb.Referenceable): sendByReference(obj) elif isinstance(obj, pb.Copyable): sendByValue(obj) else: raise InsecureJellyError or by what it can be adapted to: r = IReferenceable(obj, None) if r: sendByReference(r) else: r = ICopyable(obj, None) if r: sendByValue(r) else: raise InsecureJellyError The decision could also be influenced by the sending schema currently in effect. Side A invokes a method on side B. A knows of a schema which states that the 'foo' argument of this method should be a CopyableSpam, so it requires the object be adaptable to ICopyableSpam (which is then copied by value) tries to comply when that argument is serialized. B will enforce its own schema. When B returns a result to A, the method-result schema on B's side can influence how the return value is handled. For bonus points, it may be possible to send the object as a combination of these two. That may get pretty hard to follow, though. adapters and Referenceable/Copyable ----------------------------------- One possibility: rather than using a SlicerRegistry, use Adapters. The ISliceable interface has one method: getSlicer(). Slicer.py would register adapters for basic types (list, dict, etc) that would just return an appropriate ListSlicer, etc. Instances which would have been pb.Copyable subclasses in oldpb can still inherit from pb.Copyable, which now implements ISliceable and produces a Slicer (opentype='instance') that calls getStateToCopy() (although the subclass-__implements__ handling is now more important than before). pb.Referenceable implements ISlicer and produces a Slicer (opentype='reference'?) which (possibly) registers itself in the broker and then sends the reference number (along with a schema if necessary (and the other end wants them)). Classes are also welcome to implement ISlicer themselves and produce whatever clever (streaming?) Slicer objects they like. On the receiving side, we still need a registry to provide reasonable security. There are two registries. The first is the RootUnslicer.openRegistry, and maps OPEN types to Unslicer factories. It is used in doOpen(). The second registry should map opentype=instance class names to something which can handle the instance contents. Should this be a replacement Unslicer? foolscap-0.13.1/doc/serializing.rst0000644000076500000240000000702512766553111017655 0ustar warnerstaff00000000000000Using Foolscap for Serialization ================================ The same code that Foolscap uses to transport object graphs from one process to another (over a wire) can be used to transport those graphs from one process to another (over time). This code serializes an object (and the other objects it references) into a series of bytes: these bytes can then either be written to a network socket, or written to a file. The main difference between the two cases is the way that certain special objects are represented (and whether they can be represented at all). Inert data like strings, lists, and numbers are equally serializable. ``foolscap.Copyable`` objects are also serializable. But ``foolscap.Referenceable`` objects only make sense to serialize in the context of a connection through which ``callRemote`` messages can eventually be sent. So trying to serialize a ``Referenceable`` when the results are going to be written to disk should cause an error. The way that Foolscap enables the re-use of its serialization code is to allow it to be called with a different "Root Slicer". This root gets to decide how all objects are serialized. The Root Slicer for a live Foolscap connection knows that Referenceables can be serialized with a special marker that tells the other end of the connection how to construct a corresponding ``RemoteReference`` (one which will be able to send ``callRemote`` s over the connection). Serializing to Bytes -------------------- To use Foolscap to serialize an object graph to a big string, use ``foolscap.serialize`` . Note that this returns a Deferred.: .. code-block:: python import foolscap obj = ["look at the pretty graph", 3, True] obj.append(obj) # and look at the pretty cycle d = foolscap.serialize(obj) d.addCallback(lambda data: foolscap.unserialize(data)) def _check(obj2): assert obj2[1] == "3" assert obj2[3] is obj2 d.addCallback(_check) This form of serialization has the following restrictions: - it can serialize any inert Python type - it can serialize ``foolscap.Copyable`` instances, and any other instance that has an ISlicer or ICopyable adapter registered for it - it cannot serialize Referenceables - it cannot serialize non-Copyable instances These restrictions mean that ``foolscap.serialize`` cannot serialize everything that Python's stdlib ``pickle`` module can handle. However, it is safer (since ``foolscap.unserialize`` will never import or execute arbitrary code like ``pickle.load`` will do), and more extensible (since ISlicer/ICopyable adapters can be registered for third-party classes). Including Referenceables ------------------------ To include Referenceables in the serialized data, you must use a Tub to do the serialization, and the process returns a Deferred rather than running synchronously: .. code-block:: python r = Referenceable() obj = ["look at the pretty graph", 3, r] d = tub1.serialize(obj) def _done(data): return tub2.unserialize(data) d.addCallback(_done) def _check(obj2): assert obj2[1] == "3" assert isinstance(obj2[2], RemoteReference) d.addCallback(_check) For this to work, the first Tub must have a location set on it, so that you could do ``registerReference`` . The first Tub will serialize the Referenceable with a special marker that the second Tub will be able to use to establish a connection to the original object. This will only succeed if the original Tub is still running and still knows about the Referenceable: think of the embedded marker as a weakref. foolscap-0.13.1/doc/specifications/0000755000076500000240000000000013204747603017601 5ustar warnerstaff00000000000000foolscap-0.13.1/doc/specifications/banana.rst0000644000076500000240000017416112766553111021566 0ustar warnerstaff00000000000000The Banana Protocol =================== *NOTE! This is all preliminary and is more an exercise in semiconscious protocol design than anything else. Do not believe this document. This sentence is lying. So there.* Banana tokens ------------- At the lowest layer, the wire transport takes the form of Tokens. These all take the shape of header/type-byte/body. - Header: zero or more bytes, all of which have the high bit clear (they range in value from 0 to 127). They form a little-endian base-128 number, so 1 is represented as 0x01, 128 is represented as 0x00 0x01, 130 as 0x02 0x01, etc. 0 can be represented by any string of 0x00 bytes, including an empty string. The maximum legal header length is 64 bytes, so it has a maximum value of 2**(64*7)-1. Not all tokens have headers. - Type Byte: the high bit is set to distinguish it from the header bytes that precede it (it has a value from 128 to 255). The Type Byte determines how to interpret both the header and the body. All valid type bytes are listed below. - Body: zero or more arbitrary bytes, length is specified by the header. Not all tokens have bodies. Tokens are described below as [header-TOKEN-body], where either "header" or "body" may be empty. For example, [len-LIST-empty] indicates that the length is put into the header, "LIST" is the token being used, and the body is empty. The possible Token types are: - ``0x80: LIST (old): [len-LIST-empty]`` This token marks the beginning of a list with LEN elements. It acts as the "open parenthesis" , and the matching "close parenthesis" is implicit, based upon the length of the list. It will be followed by LEN things, which may be tokens like INTs or STRINGS, or which may be sublists. Banana keeps a list stack to handle nested sublists. This token (and the notion of length-prefixed lists in general) is from oldbanana. In newbanana it is only used during the initial dialect negotiation (so that oldbanana peers can be detected). Newbanana requires that LIST(old) tokens be followed exclusively by strings and have a rather limited allowable length (say, 640 dialects long). - ``0x81: INT: [value-INT-empty]`` This token defines a single positive integer. The protocol defines its range as [0, 2**31), so the largest legal value is 2**31-1. The recipient is responsible for choosing an appropriate local type to hold the number. For Python, if the value represented by the incoming base-128 digits grows larger than a regular Python IntType can accomodate, the receiving system will use a LongType or a BigNum as necessary. Anything larger than this range must be sent with a LONGINT token instead. (oldbanana compatibility note: a python implementation can accept anything in the range [0, 2**448), limited by the 64-byte maximum header size). The range was chosen to allow INT values to always fit in C's s_int32_t type, so an implementation that doesn't have a useful bignum type can simply reject LONGINT tokens. - ``0x82: STRING [len-STRING-chars]`` This token defines a string. To be precise, it defines a sequence of bytes. The length is a base-128-encoded integer. The type byte is followed by LEN bytes of data which make up the string. LEN is required to be shorter than 640k: this is intended to reduce the amount of memory that can be consumed on the receiving end before user code gets to decide whether to accept the data or not. - ``0x83: NEG: [value-NEG-empty]`` This token defines a negative integer. It is identical to the ``INT`` tag except that the results are negated before storage. The range is defined as [-2**31, 0), again to make an implementation using s_int32_t easier. Any numbers smaller (more negative) than this range must be sent with a LONGNEG token. Implementations should be tolerant when receiving a "negative zero" and turn it into a 0, even though they should not send such things. Note that NEG can represent a number (-2**31) whose absolute value (2**31) is one larger than the greatest number that INT can represent (2**31-1). - ``0x84: FLOAT [empty-FLOAT-value]`` This token defines a floating-point number. There is no header, and the type byte is followed by 8 bytes which are a 64-bit IEEE "double" , as defined by ``struct.pack("!d", num)`` . - ``0x85: OLDLONGINT: [value-OLDLONGINT-empty]`` ``0x86: OLDLONGNEG: [value-OLDLONGNEG-empty]`` These were used by oldbanana to represent large numbers. Their size was limited by the number of bytes in the header (max 64), so they can represent [0, 2**448). - ``0x87: VOCAB: [index-VOCAB-empty]`` This defines a tokenized string. Banana keeps a mapping of common strings, each one is assigned a small integer. These strings can be sent compressed as a two-byte (index, VOCAB) sequence. They are delivered to Jelly as plain strings with no indication that they were compressed for transit. The strings in this mapping are populated by the sender when it sends a special "vocab" OPEN sequence. The intention is that this mapping will be sent just once when the connection is first established, but a sufficiently ambituous sender could use this to implement adaptive forward compression. - ``0x88: OPEN: [[num]-OPEN-empty]`` ``0x89: CLOSE: [[num]-CLOSE-empty]`` These tokens are the newbanana parenthesis markers. They carry an optional number in their header: if present, the number counts the appearance of OPEN tokens in the stream, starting at 0 for the first OPEN used for a given connection and incrementing by 1 for each subsequent OPEN. The matching CLOSE token must contain an identical number. These numbers are solely for debugging and may be omitted. They may be removed from the protocol once development has been completed. In contrast to oldbanana (with the LIST token), newbanana does not use length-prefixed lists. Instead it relies upon the Banana layer to track OPEN/CLOSE tokens. OPEN markers are followed by the "Open Index" tuple: one or more tokens to indicate what kind of new sub-expression is being started. The first token must be a string (either STRING or VOCAB), the rest may be strings or other primitive tokens. The recipient decides when the Open Index has finished and the body has begun. - ``0x8A: ABORT: [[num]-ABORT-empty]`` This token indicates that something has gone wrong on the sender side, and that the resulting object must not be handed upwards in the unslicer stack. It may be impossible or inconvenient for the sender to stop sending the tokens associated with the unfortunate object, so the receiver must be prepared to silently drop all further tokens up to the matching STOP marker. The STOP token must always follow eventually: this is just a courtesy notice. The number, if present, will be the same one used by the OPEN token. - ``0x8B: LONGINT: [len-LONGINT-bytes]`` ``0x8C: LONGNEG: [len-LONGNEG-bytes]`` These are processed like STRING tokens, but the bytes form a base-256 encoded number, most-significant-byte first (note that this may require several passes and some intermediate storage). The size is (barely) limited by the length field, so the theoretical range is [0, 2**(2**(64*7)-1)-1), but the receiver can impose whatever length limit they wish. LONGNEG is handled exactly like LONGINT but the number is negated first. - ``0x8D: ERROR [len-ERROR-chars]`` This token defines a string of ASCII characters which hold an error message. When a severe protocol violation occurs, the offended side will emit an ERROR token and then close the transport. The side which receives the ERROR token should put the message in a developer-readable logfile and close the transport as well. The ERROR token is formatted exactly like the STRING token, except that it is defined to be encoded in ASCII (the STRING token does not claim to be encoded in any particular character set, nor does it necessarily represent human-readable characters). The ERROR token is limited to 1000 characters. - ``0x8E: PING [[num]-PING-empty]`` ``0x8F: PONG [[num]-PONG-empty]`` These tokens have no semantic value, but are used to implement connection timeouts and keepalives. When one side receives a PING message, it should immediately queue a PONG message on the return stream. The optional number can be used to associate a PONG with the PING that prompted it: if present, it must be duplicated in the response. Other than generating a PONG, these tokens are ignored by both ends. They are not delivered to higher levels. They may appear in the middle of an OPEN sequence without affecting it. The intended use is that each side is configured with two timers: the idle timer and the disconnect timer. The idle timer specifies how long the inbound connection is allowed to remain quiet before poking it. If no data has been received for this long, a PING is sent to provoke some kind of traffic. The disconnect timer specifies how long the inbound connection is allowed to remain quiet before concluding that the other end is dead and thus terminating the connection. These messages can also be used to estimate the connection's round-trip time (including the depth of the transmit/receive queues at either end). Just send a PING with a unique number, and measure the time until the corresponding PONG is seen. TODO: Add TRUE, FALSE, and NONE tokens. (maybe? These are currently handled as OPEN sequences) Serialization ------------- When serializing an object, it is useful to view it as a directed graph. The root object is the one you start with, any objects it refers to are children of that root. Those children may point back to other objects that have already been serialized, or which will be serialized later. Banana, like pickle and other serialization schemes, does a depth-first traversal of this graph. Serialization is begun on each node before going down into the child nodes. Banana tracks previously-handled nodes and replaces them with numbered ``reference`` tokens to break loops in the graph. Banana Slicers ~~~~~~~~~~~~~~ A *Banana Slicer* is responsible for serializing a single user object: it "slices" that object into a series of smaller pieces, either fundamental Banana tokens or other Sliceable objects. On the receiving end, there is a corresponding *Banana Unslicer* which accepts the incoming tokens and re-creates the user object. There are different kinds of Slicers and Unslicers for lists, tuples, dictionaries, etc. Classes can provide their own Slicers if they want more control over the serialization process. In general, there is a Slicer object for each act of serialization of a given object (although this is not strictly necessary). This allows the Slicer to contain state about the serialization process, which enables producer/consumer -style pauses, and slicer-controlled streaming serialization. The entire context is stored in a small tuple (which includes the Slicer), so it can be set aside for a while. In the future, this will allow interleaved serialization of multiple objects (doing context switching on the wire), to do things like priority queues and avoid head-of-line blocking. The most common pattern is to have the Slicer be the ``ISlicer`` Adapter for the object, in which it gets a new Slicer case each it is serialized. Classes which do not need to store a lot of state can have a single Slicer per serialized object, presumably through some adapter tricks. It is also valid to have the serialized object be its own Slicer. The Slicer has other duties (described below), but the main one is to implement the ``slice`` method, which should return a sequence or an iterable which yields the Open Index Tokens, followed by the body tokens. (Note that the Slicer should not include the OPEN or CLOSE tokens: those are supplied by the SendBanana wrapping code). Any item which is a fundamental type (int, string, float) will be sent as a banana token, anything else will be handled by recursion (with a new Slicer). Most subclasses of ``BaseSlicer`` implement a companion method named ``sliceBody`` , which supplies just the body tokens. (This makes the code a bit easier to follow). ``sliceBody`` is usually just a "return [token, token]" , or a series of ``yield`` statements, one per token. However, classes which wish to have more control over the process can implement ``sliceBody`` or even ``slice`` differently. .. code-block:: python class ThingySlicer(slicer.BaseSlicer): opentype = ('thingy',) trackReferences = True def sliceBody(self, streamable, banana): return [self.obj.attr1, self.obj.attr2] If "attr1" and "attr2" are integers, the preceding Slicer would create a token sequence like: OPEN STRING(thingy) 13 16 CLOSE. If "attr2" were actually another Thingy instance, it might produce OPEN STRING(thingy) 13 OPEN STRING(thingy) 19 18 CLOSE CLOSE. Doing this with a generator gives the same basic results but avoids the temporary buffer, which can be important when sending large amounts of data. The following Slicer could be combined with a concatenating Unslicer to implement the old FilePager class without the extra round-trip inefficiencies. .. code-block:: python class DemandSlicer(slicer.BaseSlicer): opentype = ('demandy',) trackReferences = True def sliceBody(self, streamable, banana): f = open("data", "r") for chunk in f.read(2048): yield chunk The SendBanana code controls the pacing: if the transport is full, it has the option of pausing the generator until the receiving end has caught up. It also has the option of pulling tokens out of the Slicer anyway, and buffering them in memory. This may be necessary to achieve serialization coherency, discussed below. If the "streamable" flag is set, then the *slicer* gets to control the pacing too: it is allowed to yield a Deferred where it would normally provide a regular token. This tells Banana that serialization needs to wait for a while (perhaps we are streaming data from another source which has run dry, or we are trying to implement some kind of rate limiting). Banana will wait until the Deferred fires before attempting to retrieve another token. If the "streamable" flag is *not* set, then a parent Slicer has decided that it is unwilling to allow streaming (perhaps it needs to serialize a coherent state, and a pause for streaming would allow that state to change before it was completely serialized). The Slicer is not allowed to return a Deferred when streaming is disabled. .. code-block:: python class URLGetterSlicer(slicer.BaseSlicer): opentype = ('urldata',) trackReferences = True def gotPage(self, page): self.page = page def sliceBody(self, streamable, banana): yield self.url d = web.client.getPage(self.url) d.addCallback(self.gotPage) yield d # here we hover in limbo until it fires yield self.page (the code is a bit kludgy because generators have no way to pass data back out of the "yield" statement) (at the time this was first written). The Slicer can also raise a "Violation" exception, in which case the slicer will be aborted: no further tokens will be pulled from it. This causes an ABORT token to be sent over the wire, followed immediately by a CLOSE token. The dead Slicer's parent is notified with a ``childAborted`` method, then the Banana continues to extract tokens from the parent as if the child had finished normally. (TODO: we need a convenient way for the parent to indicate that it wishes to give up too, such as raising a Violation from within ``childAborted`` ). Serialization Coherency ~~~~~~~~~~~~~~~~~~~~~~~ Streaming serialization means the object is serialized a little bit at a time, never consuming too much memory at once. The tradeoff is that, by doing other useful work inbetween, our object may change state while it is being serialized. In oldbanana this process was uninterruptible, so coherency was not an issue. In newbanana it is optional. Some objects may have more trouble with this than others, so Banana provides Slicers with a means to influence the process. Banana makes certain promises about what takes place between successive "yield" statements, when the Slicer gives up control to Banana. The most conservative approach is to: - disable the RootSlicer's "streamable" flag to tell all Slicers that they should not return Deferreds: this avoids loss of control due to child Slicers giving it away - set the SendBanana policy to buffer data in memory rather than do a .pauseProducing: this removes pauses due to the output channel filling up - return a list from ``slice`` (or ``sliceBody`` ) instead of using a generator: this fixes the object contents at a single point in time. (you can also create a list at the beginning of that routine and then yield pieces of it, which has exactly the same effect) Slicers aren't supposed to do anything which changes the state observed by other Slicers: if this is really the case than it is safe to use a generator. A parent Slicer which yields a non-primitive object will give up control to the child Slicer needed to handle that object, but that child should do its business and finish quickly, so there should be no way for the parent object's state to change in the meantime. If the SendBanana is allowed to give up control (.pauseProducing), then arbitrary code will get to run in between "yield" calls, possibly changing the state being accessed by those yields. Likewise child Slicers might give up control, threatening the coherency of one of their parents. Slicers can invoke ``banana.inhibitStreaming()`` (TODO: need a better name) to inhibit streaming, which will cause all child serialization to occur immediately, buffering as much data in memory as necessary to complete the operation without give up control. Coherency issues are a new area for Banana, so expect new tools and techniques to be developed which allow the programmer to make sensible tradeoffs. The Slicer Stack ~~~~~~~~~~~~~~~~ (docs note: our directions are inconsistent: the RootSlicer is the parent, but lives at the bottom of the stack. I think of delegation as going "upwards" to your parent (like upcalls), so I describe it that way, but that "up" is at odds with the stack's "bottom") The serialization context is stored in a "SendBanana" object, which is one of the two halves of the Banana object (a subclass of Protocol). This holds a stack of Banana Slicers, one per object currently being serialized (i.e. one per node in the path from the root object to the object currently being serialized). For example, suppose a class instance is being serialized, and this class chose to use a dictionary to hold its instance state. That dictionary holds a list of numbers in one of its values. While the list of numbers is being serialized, the Slicer Stack would hold: the RootSlicer, an InstanceSlicer, a DictSlicer, and finally a ListSlicer. The stack is used to determine two things: - How to handle a child object: which Slicer should be used, or if a Violation should be raised - How to track object references, to break cycles in the object graph When a new object needs to be sent, it is first submitted to the top-most Slicer (to its ``slicerForObject`` method), which is responsible for either returning a suitable Slicer or raising a Violation exception (if the object is rejected by a security policy). Most Slicers will just delegate this method up to the RootSlicer, but Slicers which wish to pass judgement upon enclosed objects (or modify the Slicer selected) can do something else. Unserializable objects will raise an exception here. Once the new Slicer is obtained, the OPEN token is emitted, which provides the "openID" number (just an implicit count of how many OPEN tokens have been sent over the wire). This is where we break cycles in the object graph: before serializing the object, we record a reference to it (the openID), and any time we encounter the object again, we send the reference number instead of a new copy. This reference number is tracked in the SlicerStack, by handing the number/object pair to the top-most Slicer's ``registerReference`` method. Most Slicers will delegate this up to the RootSlicer, but again they can perform additional registrations or consume the request entirely. This is used in PB to provide "scoped references" , where (for example) a list *should* be sent twice if it occurs in two separate method calls. In this case the CallSlicer (which sits above the PBRootSlicer) does its own registration. The ``slicerForObject`` process is responsible for catching the second time the object is sent. It looks in the same mapping created by ``registerReference`` and returns a ``ReferenceSlicer`` instead of the usual one. The ``RootSlicer`` , which sits at the bottom of the stack, is a special case. It is never pushed or popped, and implements most of the policy for the whole Banana process. The RootSlicer can also be interpreted as a "root object" , if you imagine that any given user object being serialized is somehow a child of the overall serialization context. In PB, for example, the root object would be related to the connection and needs to track things like which remotely-invokable objects are available. The default RootSlicer implements the following behavior: - Allow all objects to be serialized that can be - Use its ``.slicerTable`` to get a Slicer for an object. If that fails, adapt the object to ISlicer - Record object references in its ``.references`` dict The ``RootSlicer`` class only does "safe" serialization: basic types and whatever you've registered an ISlicer adapter for. The ``TrustingRootSlicer`` uses that .slicerTable mapping to serialize unsafe things (arbitrary instances, classes, etc), which is suitable for local storage instead of network communication (i.e. when you want to use banana as a pickle replacement). TODO: The idea is to let other serialization contexts do other things. For example, the final tokens could go to the parent slice for handling instead of straight to the Protocol, which would provide more control over turning the tokens into bytes and sending over a wire, saving to a file, etc. Finally, the stack can be queried to find out what path leads from the root object to the one currently being serialized. If something goes wrong in the serialization process (an exception is thrown), this path can make it much easier to find out *when* the trouble happened, as opposed to merely where. Knowing that the ".oops" method of your FooObject failed during serialization isn't very useful when you have 500 FooObjects inside your data structure and you need to know whether it was ``bar.thisfoo`` or ``bar.thatfoo`` which caused the problem. To this end, each Slicer has a ``.describe`` method which is supposed to return a short string that explains how to get to the child node currently being processed. When an error occurs, these strings are concatenated together and put into the failure object. Deserialization --------------- The other half of the Banana class is the ``ReceiveBanana`` , which accepts incoming tokens and turns them into objects. It is organized just like the ``SendBanana`` , with a stack of "Banana Unslicer" objects, each of which assembles tokens or child objects into a larger one. Each Unslicer receives the tokens emitted by the matching Slicer on the sending side. The whole stack is used to create new Unslicers, enforce restrictions upon what objects will be accepted, and manage object references. Each Unslicer accepts tokens that turn into an object of some sort. They pass this object up to their parent Unslicer. Eventually a finished object is given to the ``RootUnslicer`` , which decides what to do with it. When the Banana is being used for data storage (like pickle), the root will just deliver the object to the caller. When Banana is used in PB, the actual work is done by some intermediate objects like the ``CallUnslicer`` , which is responsible for a single method invocation. The ``ReceiveBanana`` itself is responsible for pulling well-formed tokens off the incoming data stream, tracking OPEN and CLOSE tokens, maintaining synchronization with the transmitted token stream, and discarding tokens when the receiving Unslicers have rejected one of the inbound objects. Unslicer methods may raise Violation exceptions: these are caught by the Unbanana and cause the object currently being unserialized to fail: its parent gets a UnbananaFailure instead of the dict or list or instance that it would normally have received. OPEN tokens are followed by a short list of tokens called the "opentype" to indicate what kind of object is being started. This is looked up in the UnbananaRegistry just like object types are looked up in the BananaRegistry (TODO: need sensible adapter-based registration scheme for unslicing). The new Unslicer is pushed onto the stack. "ABORT" tokens indicate that something went wrong on the sending side and that the current object is to be aborted. It causes the receiver to discard all tokens until the CLOSE token which closes the current node. This is implemented with a simple counter of how many levels of discarding we have left to do. "CLOSE" tokens finish the current node. The Unslicer will pass its completed object up to the "receiveChild" method of its parent. Open Index tokens: the Opentype ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ OPEN tokens are followed by an arbitrary list of other tokens which are used to determine which UnslicerFactory should be invoked to create the new Unslicer. Basic Python types are designated with a simple string, like (OPEN "list") or (OPEN "dict"), but instances are serialized with two strings (OPEN "instance" "classname"), and various exotic PB objects like method calls may involve a list of strings and numbers (OPEN "call" reqID objID methodname). The unbanana code works with the unslicer stack to apply constraints to these indexing tokens and finally obtain the new Unslicer when enough indexing tokens have been received. The reason for assembling this "opentype" list before creating the Unslicer (instead of using a generic InstanceUnslicer which switches behavior depending upon its first received token) is to support classes or PB methods which wish to push custom Unslicers to handle their deserialization process. For example, a class could push a StreamingFileUnslicer that accepts a series of string tokens and appends their contents to a file on disk. This Unslicer could reduce memory consumption (by only holding one chunk at a time) and update some kind of progress indicator as the data arrives. This particular feature was provided by the old StringPager utility, but custom Unslicers offer more flexibility and better efficiency (no additional round-trips). (note: none of this affects the serialization side: those Slicers emit both their indexing tokens and their state tokens. It is only the receiving side where the index tokens are handled by a different piece of code than the content tokens). In yet greater detail: - Each OPEN sequence is divided into an "Index phase" and a "Contents phase" . The first one (or two or three) tokens are the Index Tokens and the rest are the Body Tokens. The sequence ends with a CLOSE token. - Banana.inOpen is a boolean which indicates that we are in the Index Phase. It is set to True when the OPEN token is received and returns to False after the new Unslicer has been pushed. - Banana.opentype is a list of Index Tokens that are being accumulated. It is cleared each time .inOpen is set to True. The tuple form of opentype is passed to Slicer.doOpen, Constraint.checkOpentype, and used as a key in the RootSlicer.openRegistry dictionary. Each Unslicer type is indexed by an opentype tuple. If .inOpen is True, each new token type will be passed (through Banana.getLimit and top.openerCheckToken) to the opener's .openerCheckToken method, along with the current opentype tuple. The opener gets to decide if the token is acceptable (possibly raising a Violation exception). Note that the opener does not maintain state about what phase the decoding process is in, so it may want to condition its response upon the length of the opentype. After each index token is complete, it is appended to .opentype, then the list is passed (through Banana.handleOpen, top.doOpen, and top.open) to the opener's .open method. This can either return an Unslicer (which will finish the index phase: all further tokens will be sent to the new Unslicer), return None (to continue the index phase), raise a Violation (which causes an UnbananaFailure to be passed to the current top unslicer), or raise another exception (which causes the connection to be abandoned). Unslicer Lifecycle ~~~~~~~~~~~~~~~~~~ Each Unslicer has access to the following attributes: - ``.parent`` : This is set by the ReceiveBanana before ``.start`` is invoked, and provides a reference to the Unslicer responsible for the containing object. You can follow ``.parent`` all the way up the object graph to the single ``RootUnslicer`` object for this connection. It is appropriate to invoke ``openerCheckToken`` and ``open`` on your parent. - ``.protocol`` : This is set by the ReceiveBanana before ``.start`` is invoked, and provides access to the Banana object which maintains the connection on which this object is being received. It is appropriate to examine the ``.debugReceive`` attribute on the protocol. It is also appropriate to invoke ``.setObject`` on it to register references for shared containers (like lists). - ``openCount`` : This is set by the ReceiveBanana before ``.start`` is invoked, and contains the optional OPEN-count for this object, an implicit sequence number incremented for each OPEN token seen on the wire. During protocol development and testing the OPEN tokens may include an explicit OPEN-count value, but usually it is left out of the packet. If present, it is used by Banana.handleClose to assert that the CLOSE token is associated with the right OPEN token. Unslicers will not normally have a use for it. - ``.count`` : This is provided as the "count" argument to ``.start`` , and contains the "object counter" for this object. This is incremented for each new object which is created by the receive Banana code. This is similar to (but not always the same as) the OPEN-count. Containers should call ``self.protocol.setObject`` to register a Deferred during ``start`` , then call it again in ``receiveClose`` with the real (finished) object. It is sometimes also included in a debug message. - ``.broker`` : PB objects are given .broker, which is exactly equal to the .protocol attribute. The synonym exists because it makes several PB routines easier to read. Each Unslicer handles a single "OPEN sequence" , which starts with an OPEN token and ends with a CLOSE token. Creation ^^^^^^^^ Acceptance of the OPEN token simply sets a flag to indicate that we are in the Index Phase. (The OPEN token might not be accepted: it is submitted to checkToken for approval first, as described below). During the Index Phase, all tokens are appended to the current ``opentype`` list and handed as a tuple to the top-most Unslicer's ``doOpen`` method. This method can do one of the following things: - Return a new Unslicer object. It does this when there are enough index tokens to specify a new Unslicer. The new child is pushed on top of the Unslicer stack (Banana.receiveStack) and initialized by calling the ``start`` method described below. This ends the Index Phase. - Return None. This indicates that more index tokens are required. The Banana protocol object simply remains in the Index Phase and continues to accumulate index tokens. - Raise a Violation. If the open type is unrecognized, then a Violation is a good way to indicate it. When a new Unslicer object is pushed on the top of the stack, it has its ``.start`` method called, in which it has an opportunity to create whatever internal state is necessary to record the incoming content tokens. Each created object will have a separate Unslicer instance. The start method can run normally, or raise a Violation exception. ``.start`` is distinct from the Unslicer's constructor function to minimize the parameter-passing requirements for doOpen() and friends. It is also conceivable that keeping arguments out of ``__init__`` would make it easier to use adapters in this context, although it is not clear why that might be useful on the Unslicing side. TODO: consider merging ``.start`` into the constructor. This Unslicer is responsible for all incoming tokens until either 1: it pushes a new one on the stack, or 2: it receives a CLOSE token. checkToken ^^^^^^^^^^ Each token starts with a length sequence, up to 64 bytes which are turned into an integer. This is followed by a single type byte, distinguished from the length bytes by having the high bit set (the type byte is always 0x80 or greater). When the typebyte is received, the topmost Unslicer is asked about its suitability by calling the ``.checkToken`` method. (note that CLOSE and ABORT tokens are always legal, and are not submitted to checkToken). Both the typebyte and the header's numeric value are passed to this methoed, which is expected to do one of the following: - Return None to indicate that the token and the header value are acceptable. - Raise a ``Violation`` exception to reject the token or the header value. This will cause the remainder of the current OPEN sequence to be discarded (all tokens through the matching CLOSE token). Unslicers should raise this if their constraints will not accept the incoming object: for example a constraint which is expecting a series of integers can accept INT/NEG/LONGINT/LONGNEG tokens and reject OPEN/STRING/VOCAB/FLOAT tokens. They should also raise this if the header indicates, e.g., a STRING which is longer than the constraint is willing to accept, or a LONGINT/LONGNEG which is too large. The topmost Unslicer (the same one which raised Violation) will receive (through its ``.receiveChild`` method) an UnbananaFailure object which encapsulates the reason for the rejection If the token sequence is in the "index phase" (i.e. it is just after an OPEN token and a new Unslicer has not yet been pushed), then instead of ``.checkToken`` the top unslicer is sent ``.openerCheckToken`` . This method behaves just like checkToken, but in addition to the type byte it is also given the opentype list (which is built out of all the index tokens received during this index phase). receiveChild ^^^^^^^^^^^^ If the type byte is accepted, and the size limit is obeyed, then the rest of the token is read and a finished (primitive) object is created: a string or number (TODO: maybe add boolean and None). This object is handed to the topmost Unslicer's ``.receiveChild`` method, where again it is has a few options: - Run normally: if the object is acceptable, it should append or record it somehow. - Raise Violation, just like checkToken. - invoke ``self.abort`` , which does ``protocol.abandonUnslicer`` If the child is handed an UnbananaFailure object, and it wishes to pass it upwards to its parent, then ``self.abort`` is the appropriate thing to do. Raising a Violation will accomplish the same thing, but with a new UnbananaFailure that describes the exception raised here instead of the one raised by a child object. It is bad to both call ``abort`` and raise an exception. Finishing ^^^^^^^^^ When the CLOSE token arrives, the Unslicer will have its ``.receiveClose`` method called. This is expected to do: - Return an object: this object is the finished result of the deserialization process. It will be passed to ``.receiveChild`` of the parent Unslicer. - Return a Deferred: this indicates that the object cannot be created yet (tuples that contain references to an enclosing tuple, for example). The Deferred will be fired (with the object) when it completes. - Raise Violation After receiveClose has finished, the child is told to clean up by calling its ``.finish`` method. This can complete normally or raise a Violation. Then, the old top-most Unslicer is popped from the stack and discarded. Its parent is now the new top-most Unslicer, and the newly-unserialized object is given to it with the ``.receiveChild`` method. Note that this method is used to deliver both primitive objects (from raw tokens) *and* composite objects (from other Unslicers). Error Handling ~~~~~~~~~~~~~~ Schemas are enforced by Constraint objects which are given an opportunity to pass judgement on each incoming token. When they do not like something they are given, they respond by raising a ``Violation`` exception. The Violation exception is sometimes created with an argument that describes the reason for the rejection, but frequently it is just a bare exception. Most Violations are raised by the ``checkOpentype`` and ``checkObject`` methods of the various classes in ``schema.py`` . Violations which occur in an Unslicer can be confined to a single sub-tree of the object graph. The object being deserialized (and all of its children) is abandoned, and all remaining tokens for that object are discarded. However, the parent object (to which the abandoned object would have been given) gets to decide what happens next: it can either fail itself, or absorb the failure (much like an exception handler can choose to re-raise the exception or eat it). When a Violation occurs, it is wrapped in an ``UnbananaFailure`` object (just like Deferreds wrap exceptions in Failure objects). The UnbananaFailure behaves like a regular ``twisted.python.failure.Failure`` object, except that it has an attribute named ``.where`` which indicate the object-graph pathname where the problem occurred. The Unslicer which caused the Violation is given a chance to do cleanup or error-reporting by invoking its ``reportViolation`` method. It is given the UnbananaFailure so it can modify or copy it. The default implementation simply returns the is expected to return the UnbananaFailure it was given, but it is also allowed to return a different one. It must return an UnbananaFailure: it cannot ignore the Violation by returning None. This method should not raise any exceptions: doing so will cause the connection to be dropped. The UnbananaFailure returned by ``reportViolation`` is passed up the Unslicer stack in lieu of an actual object. Most Unslicers have code in their ``receiveChild`` methods to detect an UnbananaFailure and trigger an abort (``propagateUnbananaFailures`` ), which causes all further tokens of the sub-tree to be discarded. The connection is not dropped. Unslicers which partition their children's sub-graphs (like the PBRootUnslicer, for which each child is a separate operation) can simply ignore the UnbananaFailure, or respond to it by sending an error message to the other end. Other exceptions may occur during deserialization. These indicate coding errors or severe protocol violations and cause the connection to be dropped (they are not caught by the Banana code and thus propagate all the way up to the reactor, which drops the socket). The exception is logged on the local side with ``log.err`` , but the remote end will not be told any reason for the disconnection. The banana code uses the BananaError exception to indicate protocol violations, but others may be encountered. The Banana object can also choose to respond to Violations by terminating the connection. For example, the ``.hangupOnLengthViolation`` flag causes string-too-long violations to be raised directly instead of being handled, which will cause the connection to be dropped (as it occurs in the dataReceived method). Example ~~~~~~~ The serialized form of ``["foo",(1,2)]`` is the following token sequence: OPEN STRING(list) STRING(foo) OPEN STRING(tuple) INT(1) INT(2) CLOSE CLOSE. In practice, the STRING(list) would really be something like VOCAB(7), likewise the STRING(tuple) might be VOCAB(8). Here we walk through how this sequence is processed. The initial Unslicer stack consists of the single RootUnslicer ``rootun`` . :: OPEN rootun.checkToken(OPEN) : must not raise Violation enter index phase VOCAB(7) (equivalent to STRING(list)) rootun.openerCheckToken(VOCAB, ()) : must not raise Violation VOCAB token is looked up in .incomingVocabulary, turned into "list" rootun.doOpen(("list",)) : looks in UnslicerRegistry, returns ListUnslicer exit index phase the ListUnslicer is pushed on the stack listun.start() STRING(foo) listun.checkToken(STRING, 3) : must return None string is assembled listun.receiveChild("foo") : appends to list OPEN listun.checkToken(OPEN) : must not raise Violation enter index phase VOCAB(8) (equivalent to STRING(tuple)) listun.openerCheckToken(VOCAB, ()) : must not raise Violation VOCAB token is looked up, turned into "tuple" listun.doOpen(("tuple",)) : delegates through: BaseUnslicer.open self.opener (usually the RootUnslicer) self.opener.open(("tuple",)) returns TupleUnslicer exit index phase TupleUnslicer is pushed on the stack tupleun.start() INT(1) tupleun.checkToken(INT) : must not raise Violation integer is assembled tupleun.receiveChild(1) : appends to list INT(2) tupleun.checkToken(INT) : must not raise Violation integer is assembled tupleun.receiveChild(2) : appends to list CLOSE tupleun.receiveClose() : creates and returns the tuple (1,2) (could also return a Deferred) TupleUnslicer is popped from the stack and discarded listun.receiveChild((1,2)) CLOSE listun.receiveClose() : creates and returns the list ["foo", (1,2)] ListUnslicer is popped from the stack and discarded rootun.receiveChild(["foo", (1,2)]) Other Issues ------------ Deferred Object Recreation: The Trouble With Tuples ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Types and classes are roughly classified into containers and non-containers. The containers are further divided into mutable and immutable. Some examples of immutable containers are tuples and bound methods. Lists and dicts are mutable containers. Ints and strings are non-containers. Non-containers are always leaf nodes in the object graph. During unserialization, objects are in one of three states: uncreated, referenceable (but not complete), and complete. Only mutable containers can be referenceable but not complete: immutable containers have no intermediate referenceable state. Mutable containers (like lists) are referenceable but not complete during traversal of their child nodes. This means those children can reference the list without trouble. Immutable containers (like tuples) present challenges when unserializing. The object cannot be created until all its components are referenceable. While it is guaranteed that these component objects will be complete before the graph traversal exits the current node, the child nodes are allowed to reference the current node during that traversal. The classic example is the graph created by the following Python fragment: .. code-block:: python a = ([],) a[0].append((a,)) To handle these cases, the TupleUnslicer installs a Deferred into the object table when it begins unserializing (in the .start method). When the tuple is finally complete, the object table is updated and the Deferred is fired with the new tuple. Containers (both mutable and immutable) are required to pay attention to the types of their incoming children and notice when they receive Deferreds instead of normal objects. These containers are not complete (in the sense described above) until those Deferreds have been replaced with referenceable objects. When the container receives the Deferred, it should attach a callback to it which will perform the replacement. In addition, immutable containers should check after each update to see if all the Deferreds have been cleared, and if so, complete their own object (and fire their own Deferreds so any containers *they* are a child of may be updated and/or completed). TODO: it would be really handy to have the RootUnslicer do Deferred Accounting: each time a Deferred is installed instead of a real object, add its the graph-path to a list. When the Deferred fires and the object becomes available, remove it. If deserialization completes and there are still Deferreds hanging around, flag an error that points to the culprits instead of returning a broken object. Security Model ~~~~~~~~~~~~~~ Having the whole Slicer stack get a chance to pass judgement on the outbound object is very flexible. There are optimizations possibly because of the fact that most Slicers don't care, perhaps a separate stack for the ones that want to participate, or a chained delegation function. The important thing is to make sure that exception cases don't leave a "taster" stranded on the stack when the object that put it there has gone away. On the receiving side, the top Unslicer gets to make a decision about the token before its body has arrived (limiting memory exposure to no more than 65 bytes). In addition, each Unslicer receives component tokens one at a time. This lets you catch the dangerous data before it gets turned into an object. However, tokens are a pretty low-level place to do security checks. It might be more useful to have some kind of "instance taster stack" , with tasters that are asked specifically about (class,state) pairs and whether they should be turned into objects or not. Because the Unslicers receive their data one token at a time, things like InstanceUnslicer can perform security checks one attribute at a time. "traits" -style attribute constraints (see the Chaco project or the PyCon-2003 presentation for details) can be implemented by having a per-class dictionary of tests that attribute values must pass before they will be accepted. The instance will only be created if all attributes fit the constraints. The idea is to catch violations before any code is run on the receiving side. Typical checks would be things like ".foo must be a number" , ".bar must not be an instance" , ".baz must implement the IBazzer interface" . TODO: the rest of this section is somewhat out of date. Using the stack instead of a single Taster object means that the rules can be changed depending upon the context of the object being processed. A class that is valid as the first argument to a method call may not be valid as the second argument, or inside a list provided as the first argument. The PBMethodArgumentsUnslicer could change the way its .taste method behaves as its state machine progresses through the argument list. There are several different ways to implement this Taster stack: - Each object in the Unslicer stack gets to raise an exception if they don't like what they see: unanimous consent is required to let the token or object pass - The top-most unslicer is asked, and it has the option of asking the next slice down. It might not, allowing local "I'm sure this is safe" classes to override higher-level paranoia. - Unslicer objects may add and remove Taster objects on a separate stack. This is undoubtedly faster but must be done carefully to make sure Tasters and Unslicers stay in sync. Of course, all this holds true for the sending side as well. A Slicer could enforce a policy that no objects of type Foo will be sent while it is on the stack. It is anticipated that something like the current Jellyable/Unjellyable classes will be created to offer control over the Slicer/Unslicers used to handle instance of that class. One eventual goal is to allow PB to implement E-like argument constraints. Streaming Slices ~~~~~~~~~~~~~~~~ The big change from the old Jelly scheme is that now serialization/unserialization is done in a more streaming format. Individual tokens are the basic unit of information. The basic tokens are just numbers and strings: anything more complicated (starting at lists) involves composites of other tokens. Producer/Consumer-oriented serialization means that large objects which can't fit into the socket buffers should not consume lots of memory, sitting around in a serialized state with nowhere to go. This must be balanced against the confusion caused by time-distributed serialization. PB method calls must retain their current in-order execution, and it must not be possible to interleave serialized state (big mess). One interesting possibility is to allow multiple parallel SlicerStacks, with a context-switch token to let the receiving end know when they should switch to a different UnslicerStack. This would allow cleanly interleaved streams at the token level. "Head-of-line blocking" is when a large request prevents a smaller (quicker) one from getting through: grocery stores attempt to relieve this frustration by grouping customers together by expected service time (the express lane). Parallel stacks would allow the sender to establish policies on immediacy versus minimizing context switches. CBanana, CBananaRun, RunBananaRun ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Another goal of the Jelly+Banana->JustBanana change is the hope of writing Slicers and Unslicers in C. The CBanana module should have C objects (structs with function pointers) that can be looked up in a registry table and run to turn python objects into tokens and vice versa. This ought to be faster than running python code to implement the slices, at the cost of less flexibility. It would be nice if the resulting tokens could be sent directly to the socket at the C level without surfacing into python; barring this it is probably a good idea to accumulate the tokens into a large buffer so the code can do a few large writes instead of a gazillion small ones. It ought to be possible to mix C and Python slices here: if the C code doesn't find the slice in the table, it can fall back to calling a python method that does a lookup in an extensible registry. Beyond Banana ------------- Random notes and wild speculations: take everything beyond here with *two* grains of salt Oldbanana usage ~~~~~~~~~~~~~~~ The oldbanana usage model has the layer above banana written in one of two ways. The simple form is to use the ``banana.encode`` and ``banana.decode`` functions to turn an object into a bytestream. This is used by twisted.spread.publish . The more flexible model is to subclass Banana. The largest example of this technique is, of course, twisted.spread.pb.Broker, but others which use it are twisted.trial.remote and twisted.scripts.conch (which appears to use it over unix-domain sockets). Banana itself is a Protocol. The Banana subclass would generally override the ``expressionReceived`` method, which receives s-expressions (lists of lists). These are processed to figure out what method should be called, etc (processing which only has to deal with strings, numbers, and lists). Then the serialized arguments are sent through Unjelly to produce actual objects. On output, the subclass usually calls ``self.sendEncoded`` with some set of objects. In the case of PB, the arguments to the remote method are turned into s-expressions with jelly, then combined with the method meta-data (object ID, method name, etc), then the whole request is sent to ``sendEncoded`` . Newbanana ~~~~~~~~~ Newbanana moves the Jelly functionality into a stack of Banana Slices, and the lowest-level token-to-bytestream conversion into the new Banana object. Instead of overriding ``expressionReceived`` , users could push a different root Unslicer. to get more control over the receive process. Currently, Slicers call Banana.sendOpen/sendToken/sendClose/sendAbort, which then creates bytes and does transport.write . To move this into C, the transport should get to call CUnbanana.receiveToken There should be CBananaUnslicers. Probably a parent.addMe(self) instead of banana.stack.append(self), maybe addMeC for the C unslicer. The Banana object is a Protocol, and has a dataReceived method. (maybe in some C form, data could move directly from a CTransport to a CProtocol). It parses tokens and hands them to its Unslicer stack. The root Unslicer is probably created at connectionEstablished time. Subclasses of Banana could use different RootUnslicer objects, or the users might be responsible for setting up the root unslicer. The Banana object is also created with a RootSlicer. Banana.writeToken serializes the token and does transport.write . (a C form could have CSlicer objects which hand tokens to a little CBanana which then hands bytes off to a CTransport). Doing the bytestream-to-Token conversion in C loses a lot of utility when the conversion is done token at a time. It made more sense when a whole mess of s-lists were converted at once. All Slicers currently have a Banana pointer.. maybe they should have a transport pointer instead? The Banana pointer is needed to get to top of the stack. want to be able to unserialize lists/tuples/dicts/strings/ints ("basic types" ) without surfacing into python. want to deliver the completed object to a python function. Streaming Methods ~~~~~~~~~~~~~~~~~ It would be neat if a PB method could indicate that it would like to receive its arguments in a streaming fashion. This would involve calling the method early (as soon as the objectID and method name were known), then somehow feeding objects to it as they arrive. The object could return a handler or consumer sub-object which would be fed as tokens arrive over the wire. This consumer should have a way to enforce a constraint on its input. This consumer object sounds a lot like an Unslicer, so maybe the method schema should indicate that the method will would like to be called right away so it can return an Unslicer to be pushed on the stack. That Unslicer could do whatever it wanted with the incoming tokens, and could enforce constraints with the usual checkToken/doOpen/receiveChild/receiveClose methods. On the sending side, it would be neat to let a callRemote() invocation provide a Producer or a generator that will supply data as the network buffer becomes available. This could involve pushing a Slicer. Slicers are generators. Common token sequences ---------------------- Any given Banana instance has a way to map objects to the Open Index tuples needed to represent them, and a similar map from such tuples to incoming object factories. These maps give rise to various "classes" of objects, depending upon how widespread any particular object type is. A List is a fairly common type of object, something you would expect to find implemented in pretty much any high-level language, so you would expect a Banana implementation in that language to be capable of accepting an (OPEN, 'list') sequence. However, a Failure object (found in ``twisted.python.failure`` , providing an asynchronous-friendly way of reporting python exceptions) is both Python- and Twisted- specific. Is it reasonable for one program to emit an (OPEN, 'failure') sequence and expect another speaker of the generic "Banana" protocol to understand it? This level of compatibility is (somewhat arbitrarily) named "dialect compatibility" . The set of acceptable sequences will depend upon many things: the language in which the program at each end of the wire is implemented, the nature of the higher-level software that is using Banana at that moment (PB is one such layer), and application-specific registrations that have been performed by the time the sequence is received (the set of ``pb.Copyable`` sequences that can be received without error will depend upon which ``RemoteCopyable`` class definitions and ``registerRemoteCopy`` calls have been made). Ideally, when two Banana instances first establish a connection, they will go through a negotiation phase where they come to an agreement on what will be sent across the wire. There are two goals to this negotiation: #. least-surprise: if one side cannot handle a construct which the other side might emit at some point in the future, it would be nice to know about it up front rather than encountering a Violation or connection-dropping BananaError later down the line. This could be described as the "strong-typing" argument. It is important to note that different arguments (both for and against strong typing) may exist when talking about remote interfaces rather than local ones. #. adapability: if one side cannot handle a newer construct, it may be possible for the other side to back down to some simpler variation without too much loss of data. Dialect negotiation is a very much still an active area of development. Base Python Types ~~~~~~~~~~~~~~~~~ The basic python types are considered "safe" : the code which is invoked by their receipt is well-understood and there is no way to cause unsafe behavior during unserialization. Resource consumption attacks are mitigated by Constraints imposed by the receiving schema. Note that the OPEN(dict) slicer is implemented with code that sorts the list of keys before serializing them. It does this to provide deterministic behavior and make testing easier. +----------------------------+-------------------------------------------------+ | IntType, LongType (small+) | INT(value) | +----------------------------+-------------------------------------------------+ | IntType, LongType (small-) | NEG(value) | +----------------------------+-------------------------------------------------+ | IntType, LongType (large+) | LONGINT(value) | +----------------------------+-------------------------------------------------+ | IntType, LongType (large-) | LONGNEG(value) | +----------------------------+-------------------------------------------------+ | FloatType | FLOAT(value) | +----------------------------+-------------------------------------------------+ | StringType | STRING(value) | +----------------------------+-------------------------------------------------+ | StringType (tokenized) | VOCAB(tokennum) | +----------------------------+-------------------------------------------------+ | UnicodeType | OPEN(unicode) STRING(str.encode('UTF-8')) CLOSE | +----------------------------+-------------------------------------------------+ | ListType | OPEN(list) elem.. CLOSE | +----------------------------+-------------------------------------------------+ | TupleType | OPEN(tuple) elem.. CLOSE | +----------------------------+-------------------------------------------------+ | DictType, DictionaryType | OPEN(dict) (key,value).. CLOSE | +----------------------------+-------------------------------------------------+ | NoneType | OPEN(none) CLOSE | +----------------------------+-------------------------------------------------+ | BooleanType | OPEN(boolean) INT(0/1) CLOSE | +----------------------------+-------------------------------------------------+ Extended (unsafe) Python Types ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To serialize arbitrary python object graphs (including instances) requires that we allow more types in. This begins to get dangerous: with complex graphs of inter-dependent objects, instances may need to be used (by referencing objects) before they are fully initialized. A schema can be used to make assertions about what object types live where, but in general the contents of those objects are difficult to constrain. For this reason, these types should only be used in places where you trust the creator of the serialized stream (the same places where you would be willing to use the standard Pickle module). Saving application state to disk and reading it back at startup time is one example. +--------------+------------------------------------------------------+ | InstanceType | OPEN(instance) STRING(reflect.qual(class)) | | | (attr,value).. CLOSE | +--------------+------------------------------------------------------+ | ModuleType | OPEN(module) STRING(__name__) CLOSE | +--------------+------------------------------------------------------+ | ClassType | OPEN(class) STRING(reflect.qual(class)) CLOSE | +--------------+------------------------------------------------------+ | MethodType | OPEN(method) STRING(__name__) im_self im_class CLOSE | +--------------+------------------------------------------------------+ | FunctionType | OPEN(function) STRING(module.__name__) CLOSE | +--------------+------------------------------------------------------+ PB Sequences ~~~~~~~~~~~~ See the "specifications/pb" document for details. Unhandled types ~~~~~~~~~~~~~~~ The following types are not handled by any slicer, and will raise a KeyError if one is referenced by an object being sliced. This technically imposes a limit upon the kinds of objects that can be serialized, even by a "unsafe" serializer, but in practice it is not really an issue, as many of these objects have no meaning outside the program invocation which created them. - - types that might be nice to have - ComplexType - SliceType - TypeType - XRangeType - - types that aren't really that useful - BufferType - BuiltinFunctionType - BuiltinMethodType - CodeType - DictProxyType - EllipsisType - NotImplementedType - UnboundMethodType - - types that are meaningless outside the creator - TracebackType - FileType - FrameType - GeneratorType - LambdaType Unhandled (but don't worry about it) types ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``ObjectType`` is the root class of all other types. All objects are known by some other type in addition to ``ObjectType`` , so the fact that it is not handled explicitly does not matter. ``StringTypes`` is simply a list of ``StringType`` and ``UnicodeType`` , so it does not need to be explicitly handled either. Internal types ~~~~~~~~~~~~~~ The following sequences are internal. The OPEN(vocab) sequence is used to update the forward compression token-to-string table used by the VOCAB token. It is followed by a series of number/string pairs. All numbers that appear in VOCAB tokens must be associated with a string by appearing in the most recent OPEN(vocab) sequence. +------------+----------------------------------+ | vocab dict | OPEN(vocab) (num,string).. CLOSE | +------------+----------------------------------+ foolscap-0.13.1/doc/specifications/logfiles.rst0000644000076500000240000002503212766553111022142 0ustar warnerstaff00000000000000Foolscap Logging Formats ======================== This document describes the Foolscap logging format. Foolscap logging uses event dictionaries in memory. Certain tools will serialize these events onto disk. These on-disk files may have additional metadata stored in adjunct index files. This document describes the formats of all these objects. Events ------ Each call to ``log.msg`` produces an **event** . These events are stored in memory as dictionaries, with the following keys: - ``num`` (integer): this defines a full ordering of events within a single invocation of a program. The counter that produces these is maintained in the singleton ``FoolscapLogger`` instance known as ``log.theLogger`` . - ``time`` (float): the time at which ``log.msg`` was called. - ``incarnation`` (pair of binary strings or Nones): the "incarnation record", used to distinguish between distinct invocations of the same program/Tub. Each time the program is started, it gets a distinct incarnation record. The IR contains a (unique, sequential) tuple of values. 'unique' is a random binary string. At present, 'sequential' is always None. In a future release, Tubs which are given persistent storage will use an incrementing integer as 'sequential', to allow total ordering of events from multiple incarnations. Without 'sequential', events from one invocation of the program cannot be reliably sorted with respect to events from other invocations (except by timestamp, which depends upon comparable clocks). - ``level`` (integer): the severity level of the event. These are typically obtained by using one of the pre-defined constants like ``log.NOISY`` (10), ``log.WEIRD`` (30), or ``log.BAD`` (40). The default value is ``log.OPERATIONAL`` (20). - ``facility`` (string, optional): a facility name, like ``foolscap.negotiation`` . Strings are unconstrained, but foolscap tools are designed to treat the facility as a big-endian period-separated hierarchical list, i.e. ``foolscap.negotiation`` and ``foolscap.promises`` would be related. One such tool would be the ``flogtool filter --strip-facility "foolscap"`` command. - ``message`` (string, optional): the logged message. The first positional argument to ``log.msg`` will be stored here. All messages will either have ``["message"]`` or ``["format"]`` . - ``format`` (string, optional): a printf-style format specification string for the message. When the message is turned into a string, the event dictionary will be used for the string format operation, so ``log.msg(format="%(count)d apples", count=4)`` is a more structured way to say ``log.msg("%d apples" % count)`` . By using ``format=`` and delaying string interpolation until later, log-analysis tools will have more information to work with. - ``isError`` (integer, optional): ``log.err`` will set this to 1. ``log.msg`` will not set this. This is a simple test to see which entry point was used to record the message. - ``failure`` (Failure instance, optional): if ``["failure"]`` is present, formatting tools will render a brief traceback. The first positional argument to ``log.err`` will be stored here. - ``stacktrace`` (list of strings, optional): if ``log.msg`` is called with ``stacktrace=True`` , then ``traceback.format_stack()`` will be used to generate a stack trace string, storing it in this key. In addition to these keys, all other keyword arguments to the ``log.msg`` and ``log.err`` calls are recorded in the event dictionary. Some keys are reserved: those that begin with an underscore, and those that are not legal python identifiers (i.e. they contain dots). Some of these reserved keys are used for internal purposes. Developers are encouraged to store log parameters with keyword arguments rather than with string interpolation into the ``message=`` argument, so that later analysis/filtering tools can take advantage of it. For example, if you use this: .. code-block:: python log.msg(format="Uploading %(size)d byte file", size=SIZE) instead of: .. code-block:: python log.msg("Uploading %d byte file" % SIZE) Then later, you can write a filter expression that can do: .. code-block:: python def _big_uploads(e): return bool(e["format"] == "Uploading %(size)d byte file" and e["size"] > 1000) subset = filter(_big_uploads, all_events) Other tools will be provided in the future to make this more concise. This also makes it easier to write filtering expressions that can be serialized and sent over the wire, so that ``flogtool tail`` can subscribe to a narrowly-defined subset of events, rather than to everything. Logfiles -------- Several foolscap logging tools will record a sequence of events to disk: ``flogtool tail --save-to FILENAME`` and the gatherer created by ``flogtool create-gatherer`` are two of them. These tools know about two file formats, compressed and uncompressed. If the filename ends in ``.bz2`` , then the file is opened with the ``bzip`` module, but otherwise treated exactly like the uncompressed form. No support is provided for gzip or other compression schemes. The uncompressed save-file format contains a sequence of pickled "received event wrapper dictionaries". Each wrapper dict is pickled separately, such that code which wants to iterate over the contents needs to call ``pickle.load(f)`` repeatedly (this enables streaming processing). The wrapper dictionary is used to record some information that is not stored in the event dictionary itself, sometimes because it is the same for long runs of events from a single source (like the tubid that generated the event). (TODO: some of this split is arbitrary and historical, and ought to be cleaned up). The wrapper dictionary contains the following keys: - ``from`` (base32 string): the TubID that recorded the event. - ``d`` (dictionary): the event dictionary defined above. - ``rx_time`` (float): the time at which the recipient (e.g. ``flogtool tail`` ) received the event. If the generator and the recipient have synchronized clocks, then a significant delta between ``e["rx_time"]`` and ``e["d"]["time"]`` indicates delays in the event publishing process, possibly the result of reactor or network load. Logfile Headers --------------- The first wrapper dict in the logfile may be special: it contains **headers** . This header dict is distinguished by the fact that it does not contain a ``["d"]`` member. Instead, it contains a ``["header"]`` member. The tools which iterate over events in logfiles know to ignore the wrapper dicts which lack a ``["d"]`` key. On the other hand, the first wrapper dict might be a regular event. Older versions of foolscap (0.2.5 and earlier) did not produce header dicts. Tools which process logfiles must tolerate the lack of a header dict. The header dict allows the logfile to be used for various purposes, somewhat open-ended to allow for future extensions. All header dicts contain a key named ``type`` that describe the purpose of the logfile. The currently assigned values for type are: - ``log-file-observer`` : this indicates that the logfile was created by a ``LogFileObserver`` instance, for example the one created when the ``FLOGFILE=out.flog`` environment variable is used. - ``tail`` : this indicates that the logfile was created by the ``--save-to`` option of ``flogtool tail`` . - ``gatherer`` : the logfile was created by the foolscap log-gatherer, for which the ``flogtool create-gatherer`` command is provided. - ``incident`` : the logfile was created by an application as part of the incident reporting process. log-file-observer ~~~~~~~~~~~~~~~~~ The header dict produced by a ``LogFileObserver`` contains the following additional keys: - ``threshold`` (int): the severity threshold that was used for this logfile: no events below the threshold will be saved. Also note that the wrapper dicts recorded by the ``LogFileObserver`` will use a "from" value of "local", instead of a particular TubID, since these events are not recorded through a path that uses any specific Tub. flogtool tail ~~~~~~~~~~~~~ The header dict produced by ``flogtool tail`` contains the following additional keys: - ``pid`` (int): if present, this value contains the process id of the process which was being followed by 'flogtool tail'. - ``versions`` (dict): this contains a dictionary of component versions, mapping a string component name like "foolscap" to a version string. log-gatherer ~~~~~~~~~~~~ The header dict produced by the flogtool log-gatherer contains the following additional keys: - ``start`` (float): the time at which this logfile was first opened. Incident Reports ~~~~~~~~~~~~~~~~ An **Incident Report** is a logfile that was recorded because of an important triggering event: a dump of the short-term history buffers that saves the activity of the application just prior to the trigger. It can also contain some number of subsequent events, to record recovery efforts or additional information that is logged after the triggering event. Incident Reports are distinguished by their header type: ``e["header"]["type"]=="incident"`` . Their header dicts contain the following additional keys: - ``trigger`` (event dict): a copy of the event which triggered the incident. This event will also be present somewhere in the rest of the logfile, at its normal position in the event stream. - ``pid`` (int): this value contains the process id of the process which experienced the incident. - ``versions`` (dict): this contains a dictionary of component versions, mapping a string component name like "foolscap" to a version string. Index Files ----------- No index files have been defined yet. The vague idea is that each logfile could contain a summary in an index file of the same name (but with an extra .index suffix). This index would be used by other tools to quickly identify what is inside the main file without actually reading the whole contents. In addition, it may be possible to put a table of offsets into the index file, to accelerate random-access reads of the main logfile (i.e. put the offset of every 100 events into the index, reducing the worst-case access time to two seeks and a read of no more than 100 events). Some sort of restartable compression could make such an offset table useful for compressed files as well. These index files would need to exist as distinct files (rather than as a header in the main logfile) because they are variable-size and cannot be generated until after the main logfile is closed. Placing them at the start of the main logfile would require rewriting or copying the whole file. Further complications are present when the main logfile is compressed. foolscap-0.13.1/doc/specifications/pb.rst0000644000076500000240000012374412766553111020750 0ustar warnerstaff00000000000000NewPB ===== This document describes the new PB protocol. This is a layer on top of Banana which provides remote object access (method invocation and instance transfer). Fundamentally, PB is about one side keeping a ``RemoteReference`` to the other side's "Referenceable" . The Referenceable has some methods that can be invoked remotely: functionality it is offering to remote callers. Those callers hold RemoteReferences which point to it. The RemoteReference object offers a way to invoke those methods (generally through the ``callRemote`` method). There are plenty of other details, starting with how the RemoteReference is obtained, and how arguments and return values are communicated. For the purposes of this document, we will designate the side that holds the actual ``Referenceable`` object as "local" , and the side that holds the proxy ``RemoteReference`` object as "remote" . This distinction is only meaningful with respect to a single RemoteReference/Referenceable pair. One program may hold Referenceable "A" and RemoteReference "B" , paired with another that holds RemoteReference "A" and Referenceable "B" . Once initialization is complete, PB is a symmetric protocol. It is helpful to think of PB as providing a wire or pipe that connects two programs. Objects are put into this pipe at one end, and something related to the object comes out the other end. These two objects are said to correspond to each other. Basic types (like lists and dictionaries) are handled by Banana, but more complex types (like instances) are treated specially, so that most of the time there is a "native" form (as present on the local side) that goes into the pipe, and a remote form that comes out. Initialization -------------- The PB session begins with some feature negotiation and (generally) the receipt of a VocabularyDict. Usually this takes place over an interactive transport, like a TCP connection, but newpb can also be used in a more batched message-oriented mode, as long as both the creator of the method call request and its eventual consumer are in agreement about their shared state (at least, this is the intention.. there are still pieces that need to be implemented to make this possible). The local side keeps a table which provides a bidirectional mapping between ``Referenceable`` objects and a connection-local "object-ID" number. This table begins with a single object called the "Root" , which is implicitly given ID number 0. Everything else is bootstrapped through this object. For the typical PB Broker, this root object performs cred authentication and returns other Referenceables as the cred Avatar. The remote side has a collection of ``RemoteReference`` objects, each of which knows the object-ID of the corresponding Referenceable, as well as the Broker which provides the connection to the other Broker. The remote side must do reference-tracking of these RemoteReferences, because as long as it remains alive, the local-side Broker must maintain a reference to the original Referenceable. Method Calls ------------ The remote side invokes a remote method by calling ``ref.callRemote()`` on its RemoteReference. This starts by validating the method name and arguments against a "Schema" (described below). It then creates a new Request object which will live until the method call has either completed successfully or failed due to an exception (including the connection being lost). ``callRemote`` returns a Deferred, which does not fire until the request is finished. It then sends a ``call`` banana sequence over the wire. This sequence indicates the request ID (used to match the request with the resulting ``answer`` or ``error`` response), the object ID of the Referenceable being targeted, a string to indicate the name of the method being invoked, and the arguments to be passed into the method. All arguments are passed by name (i.e. keyword arguments instead of positional parameters). Each argument is subject to the "argument transformation" described below. The local side receives the ``call`` sequence, uses the object-ID to look up the Referenceable, finds the desired method, then applies the method's schema to the incoming arguments. If they are acceptable, it invokes the method. A normal return value it sent back immediately in an ``answer`` sequence (subject to the same transformation as the inbound arguments). If the method returns a Deferred, the answer will be sent back when the Deferred fires. If the method raises an exception (or the Deferred does an errback), the resulting Failure is sent back in a ``error`` sequence. Both the ``answer`` and the ``error`` start with the request-ID so they can be used to complete the Request object waiting on the remote side. The original Deferred (the one produced by ``callRemote`` ) is finally callbacked with the results of the method (or errbacked with a Failure or RemoteFailure object). Example ~~~~~~~ This code runs on the "local" side: the one with the ``pb.Referenceable`` which will respond to a remote invocation. .. code-block:: python class Responder(pb.Referenceable): def remote_add(self, a, b): return a+b and the following code runs on the "remote" side (the one which holds a ``pb.RemoteReference`` ): .. code-block:: python def gotAnswer(results): print results d = rr.callRemote("add", a=1, b=2) d.addCallback(gotAnswer) Note that the arguments are passed as named parameters: oldpb used both positional parameters and named (keyword) arguments, but newpb prefers just the keyword arguments. TODO: newpb will probably convert positional parameters to keyword arguments (based upon the schema) before sending them to the remote side. Using RemoteInterfaces ~~~~~~~~~~~~~~~~~~~~~~ To nail down the types being sent across the wire, you can use a ``RemoteInterface`` to define the methods that are implemented by any particular ``pb.Referenceable`` : .. code-block:: python class RIAdding(pb.RemoteInterface): def add(a=int, b=int): return int class Responder(pb.Referenceable): implements(RIAdding) def remote_add(self, a, b): return a+b # and on the remote side: d = rr.callRemote(RIAdding['add'], a=1, b=2) d.addCallback(gotAnswer) In this example, the "RIAdding" remote interface defines a single method "add" , which accepts two integer parameters and returns an integer. This method (technically a classmethod) is used instead of the string form of the method name. What does this get us? - The calling side will pre-check its arguments against the constraints that it believes to be imposed by the remote side. It will raise a Violation rather than send parameters that it thinks will be rejected. - The receiving side will enforce the constraints, causing the method call to errback (with a Violation) if they are not met. This means the code in ``remote_add`` does not need to worry about what strange types it might be given, such as two strings, or two lists. - The receiving side will pre-check its return argument before sending it back. If the method returns a string, it will cause a Violation exception to be raised. The caller will get this Violation as an errback instead of whatever (illegal) value the remote method computed. - The sending side will enforce the return-value constraint (raising a Violation if it is not met). This means the calling side (in this case the ``gotAnswer`` callback function) does not need to worry about what strange type the remote method returns. You can use either technique: with RemoteInterfaces or without. To get the type-checking benefits, you must use them. If you do not, PB cannot protect you against memory consumption attacks. RemoteInterfaces ~~~~~~~~~~~~~~~~ RemoteInterfaces are passed by name. Each side of a PB connection has a table which maps names to RemoteInterfaces (subclasses of ``pb.RemoteInterface`` ). Metaclass magic is used to add an entry to this table each time you define a RemoteInterface subclass, using the ``__remote_name__`` attribute (or reflect.qual() if that is not set). Each ``Referenceable`` that goes over the wire is accompanied by the list of RemoteInterfaces which it claims to implement. On the receiving side, these RemoteInterface names are looked up in the table and mapped to actual (local) RemoteInterface classes. TODO: it might be interesting to serialize the RemoteInterface class and ship it over the wire, rather than assuming both sides have a copy (and that they agree). However, if one side does not have a copy, it is unlikely that it will be able to do anything very meaningful with the remote end. The syntax of RemoteInterface is still in flux. The basic idea is that each method of the RemoteInterface defines a remotely invokable method, something that will exist with a "remote_" prefix on any ``pb.Referenceable`` s which claim to implement it. Those methods are defined with a number of named parameters. The default value of each parameter is something which can be turned into a ``Constraint`` according to the rules of schema.makeConstraint . This means you can use things like ``(int, str, str)`` to mean a tuple of exactly those three types. Note that the methods of the RemoteInterface do *not* list "self" as a parameter. As the zope.interface documentation points out, "self" is an implementation detail, and does not belong in the interface specification. Another way to think about it is that, when you write the code which calls a method in this interface, you don't include "self" in the arguments you provide, therefore it should not appear in the public documentation of those methods. The method is required to return a value which can be handled by schema.makeConstraint: this constraint is then applied to the return value of the remote method. Other attributes of the method (perhaps added by decorators of some sort) will, some day, be able to specify specialized behavior of the method. The brainstorming sessions have come up with the following ideas: - .wait=False: don't wait for an answer - .reliable=False: feel free to send this over UDP - .ordered=True: but enforce order between successive remote calls - .priority=3: use priority queue / stream #3 - .failure=Full: allow/expect full Failure contents (stack frames) - .failure=ErrorMessage: only allow/expect truncated CopiedFailures We are also considering how to merge the RemoteInterface with other useful interface specifications, in particular zope.interface and formless.TypedInterface . Argument Transformation ----------------------- To understand this section, it may be useful to review the Banana documentation on serializing object graphs. Also note that method arguments and method return values are handled identically. Basic types (lists, tuples, dictionaries) are serialized and unserialized as you would expect: the resulting object would (if it existed in the sender's address space) compare as equal (but of course not "identical" , because the objects will exist at different memory locations). Shared References, Serialization Scope ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Shared references to the same object are handled correctly. Banana is responsible for noticing that a sharable object has been serialized before (or at least has begun serialization) and inserts reference markers so that the object graph can be reconstructed. This introduces the concept of serialization scope: the boundaries beyond which shared references are not maintained. For PB, serialization is scoped to the method call. If an object is referenced by two arguments to the same method call, that method will see two references to the same object. If those arguments are containers of some form, which (eventually) hold a reference to the same object, the object graph will be preserved. For example: .. code-block:: python class Caller: def start(self): obj = [1, 2, 3] self.remote.callRemote("both", obj, obj) self.remote.callRemote("deeper", ["a", obj], (4, 5, obj)) class Called(pb.Referenceable): def remote_both(self, arg1, arg2): assert arg1 is arg2 assert arg1 == [1,2,3] def remote_deeper(self, listarg, tuplearg): ref1 = listarg[1] ref2 = tuplearg[2] assert ref1 is ref2 assert ref1 == [1,2,3] But if the remote-side object is referenced in two distinct remote method invocations, the local-side methods will see two separate objects. For example: .. code-block:: python class Caller: def start(self): self.obj = [1, 2, 3] d = self.remote.callRemote("one", self.obj) d.addCallback(self.next) def next(self, res): self.remote.callRemote("two", self.obj) class Called(pb.Referenceable): def remote_one(self, ref1): assert ref1 == [1,2,3] self.ref1 = ref1 def remote_two(self, ref2): assert ref2 == [1,2,3] assert ref1 is not ref2 # not the same object You can think of the method call itself being a node in the object graph, with the method arguments as its children. The method call node is picked up and the resulting sub-tree is serialized with no knowledge of anything outside the sub-tree [#]_ . The value returned by a method call is serialized by itself, without reference to the arguments that were given to the method. If a remote method is called with a list, and the method returns its argument unchanged, the caller will get back a deep copy of the list it passed in. Referenceables, RemoteReferences ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Referenceables are transformed into RemoteReferences when they are sent over the wire. As one side traverses the object graph of the method arguments (or the return value), each ``Referenceable`` object it encounters it serialized with a ``my-reference`` sequence, that includes the object-ID number. When the other side is unserializing the token stream, it creates a ``RemoteReference`` object, or uses one that already exists. Likewise, if an argument (or return value) contains a ``RemoteReference`` , and it is being sent back to the Broker that holds the original ``Referenceable`` then it will be turned back into that Referenceable when it arrives. In this case, the caller of a remote method which returns its argument unchanged *will* see a a result that is identical to what it passed in: .. code-block:: python class Target(pb.Referenceable): pass class Caller: def start(self): self.obj = Target() d = self.remote.callRemote("echo", self.obj) d.addCallback(self.next) def next(self, res): assert res is self.obj class Called(pb.Referenceable): def remote_echo(self, arg): # arg is a RemoteReference to a Target() instance return arg These references have a serialization scope which extends across the entire connection. As long as two method calls share the same ``Broker`` instance (which generally means they share the same TCP socket), they will both serialize ``Referenceable`` s into identical ``RemoteReference`` s. This also means that both sides do reference-counting to insure that the Referenceable doesn't get garbage-collected while a remote system holds a RemoteReference that points to it. In the future, there may be other classes which behave this way. In particular, "Referenceable" and "Callable" may be distinct qualities. Copyable, RemoteCopy ~~~~~~~~~~~~~~~~~~~~ Some objects can be marked to indicate that they should be copied bodily each time they traverse the wire (pass-by-value instead of pass-by-reference). Classes which inherit from ``pb.Copyable`` are passed by value. Their ``getTypeToCopy`` and ``getStateToCopy`` methods are used to assemble the data that will be serialized. These methods default to plain old ``reflect.qual`` (which provides the fully-qualified name of the class) and the instance's attribute ``__dict__`` . You can override these to provide a different (or smaller) set of state attributes to the remote end. .. code-block:: python class Source(pb.Copyable): def getStateToCopy(self): state = self.__dict__.copy() del state['private'] state['children'] = [] return state Rather than subclass ``pb.Copyable`` , you can also implement the ``flavors.ICopyable`` interface: .. code-block:: python from twisted.python import reflect class Source2: implements(flavors.ICopyable) def getTypeToCopy(self): return reflect.qual(self.__class__) def getStateToCopy(self): return self.__dict__ or register an ICopyable adapter. Using the adapter allows you to define serialization behavior for third-party classes that are out of your control (ones which you cannot rewrite to inherit from ``pb.Copyable`` ). .. code-block:: python class Source3: pass class Source3Copier: implements(flavors.ICopyable) def getTypeToCopy(self): return 'foo.Source3' def getStateToCopy(self): orig = self.original d = { 'foo': orig.foo, 'bar': orig.bar } return d registerAdapter(Source3Copier, Source3, flavors.ICopyable) On the other end of the wire, the receiving side must register a ``RemoteCopy`` subclass under the same name as returned by the sender's ``getTypeToCopy`` value. This subclass is used as a factory to create instances that correspond to the original ``Copyable`` . The registration can either take place explicitly (with ``pb.registerRemoteCopy`` ), or automatically (by setting the ``copytype`` attribute in the class definition). The default ``RemoteCopy`` behavior simply sets the instance's ``__dict__`` to the incoming state, which may be plenty if you are willing to let outsiders arbitrarily manipulate your object state. If so, and you believe both peers are importing the same source file, it is enough to create and register the ``RemoteCopy`` at the same time you create the ``Copyable`` : .. code-block:: python class Source(pb.Copyable): def getStateToCopy(self): state = self.__dict__.copy() del state['private'] state['children'] = [] return state class Remote(pb.RemoteCopy): copytype = reflect.qual(Source) You can do something special with the incoming object state by overriding the ``setCopyableState`` method. This may allow you to do some sanity-checking on the state before trusting it. .. code-block:: python class Remote(pb.RemoteCopy): def setCopyableState(self, state): state['count'] = 0 self.__dict__ = state self.total = self.one + self.two # show explicit registration, instead of using 'copytype' class attribute pb.registerRemoteCopy(reflect.qual(Source), Remote) You can also set a Constraint on the inbound object state, which provides a way to enforce some type checking on the state components as they arrive. This protects against resource-consumption attacks where someone sends you a zillion-byte string as part of the object's state. .. code-block:: python class Remote(pb.RemoteCopy): stateSchema = schema.AttributeDictConstraint(('foo', int), ('bar', str)) In this example, the object will only accept two attributes: "foo" (which must be a number), and "bar" (which must be a string shorter than the default limit of 1000 characters). Various classes from the``schema`` module can be used to construct more complicated constraints. Slicers, ISlicer ~~~~~~~~~~~~~~~~ Each object gets "Sliced" into a stream of tokens as they go over the wire: Referenceable and Copyable are merely special cases. These classes have Slicers which implement specific behaviors when the serialization process is asked to send their instances to the remote side. You can implement your own Slicers to take complete control over the serialization process. The most useful reason to take advantage of this feature is to implement "streaming slicers" , which can minimize in-memory buffering by only producing Banana tokens on demand as space opens up in the transport. Banana Slicers are documented in detail in the Banana documentation. Once you create a Slicer class, you will want to "register" it, letting Banana know that this Slicer is useful for conveying certain types of objects across the wire. The registry maps a type to a Slicer class (which is really a slicer factory), and is implemented by registering the slicer as a regular "adapter" for the ``ISlicer`` interface. For example, lists are serialized by the ``ListSlicer`` class, so ``ListSlicer`` is registered as the slicer for the ``list`` type: .. code-block:: python class ListSlicer(BaseSlicer): opentype = ("list",) slices = list Slicer registration can be either explicit or implicit. In this example, an implicit registration is used: by setting the "slices" attribute to the ``list`` type, the BaseSlicer's metaclass automatically registers the mapping from ``list`` to ListSlicer. To explicitly register a slicer, just leave ``opentype`` set to None (to disable auto-registration), and then register the slicer manually. .. code-block:: python class TupleSlicer(BaseSlicer): opentype = ("tuple",) slices = None ... registerAdapter(TupleSlicer, tuple, pb.ISlicer) As with ICopyable, registering an ISlicer adapter allows you to define exactly how you wish to serialize third-party classes which you do not get to modify. Unslicers ~~~~~~~~~ On the other side of the wire, the incoming token stream is handed to an ``Unslicer`` , which is responsible for turning the set of tokens into a single finished object. They are also responsible for enforcing limits on the types and sizes of the tokens that make up the stream. Unslicers are also described in greater detail in the Banana docs. As with Slicers, Unslicers need to be registered to be useful. This registry maps "opentypes" to Unslicer classes (i.e. factories which can produce an unslicer instance each time the given opentype appears in the token stream). Therefore it maps tuples to subclasses of ``BaseUnslicer`` . Again, this registry can be either implicit or explicit. If the Unslicer has a non-None class attribute named ``opentype`` , then it is automatically registered. If it does not have this attribute (or if it is set to None), then no registration is performed, and the Unslicer must be manually registered: .. code-block:: python class MyUnslicer(BaseUnslicer): ... pb.registerUnslicer(('myopentype',), MyUnslicer) Also remember that this registry is global, and that you cannot register two Unslicers for the same opentype (you'll get an exception at class-definition time, which will probably result in an ImportError). Slicer/Unslicer Example ~~~~~~~~~~~~~~~~~~~~~~~ The simplest kind of slicer has a ``sliceBody`` method (a generator) which yields a series of tokens. To demonstrate how to build a useful Slicer, we'll write one that can send large strings across the wire in pieces. Banana can send arbitrarily long strings in a single token, but each token must be handed to the transport layer in an indivisble chunk, and anything that doesn't fit in the transmit buffers will be stored in RAM until some space frees up in the socket. Practically speaking, this means that anything larger than maybe 50kb will spend a lot of time in memory, increasing the RAM footprint for no good reason. Because of this, it is useful to be able to send large amounts of data in smaller pieces, and let the remote end reassemble them. The following Slicer is registered to handle all open files (perhaps not the best idea), and simply emits the contents in 10kb chunks. (readers familiar with oldpb will notice that this Slicer/Unslicer pair provide similar functionality to the old FilePager class. The biggest improvement is that newpb can accomplish this without the extra round-trip per chunk. The downside is that, unless you enable streaming in your Broker, no other methods can be invoked while the file is being transmitted. The upside of the downside is that this lets you retain in-order execution of remote methods, and that you don't have to worry changes to the contents of the file causing corrupt data to be sent over the wire. The oter upside of the downside is that, if you enable streaming, you can do whatever other processing you wish between data chunks.) .. code-block:: python class BigFileSlicer(BaseSlicer): opentype = ("bigfile",) slices = types.FileType CHUNKSIZE = 10000 def sliceBody(self, streamable, banana): while 1: chunk = self.obj.read(self.CHUNKSIZE) if not chunk: return yield chunk To receive this, you would use the following minimal Unslicer at the other end. Note that this Unslicer does not do as much as it could in the way of constraint enforcement: an attacker could easily make you consume as much memory as they wished by simply sending you a never-ending series of chunks. .. code-block:: python class BigFileUnslicer(LeafUnslicer): opentype = ("bigfile",) def __init__(self): self.chunks = [] def checkToken(self, typebyte, size): if typebyte != tokens.STRING: raise BananaError("BigFileUnslicer only accepts strings") def receiveChild(self, obj): self.chunks.append(obj) def receiveClose(self): return "".join(self.chunks) The ``opentype`` attribute causes this Unslicer to be implicitly registered to handle any incoming sequences with an "index tuple" of ``("bigfile",)`` , so each time BigFileSlicer is used, a BigFileUnslicer will be created to handle the results. A more complete example would want to write the file chunks to disk at they arrived, or process them incrementally. It might also want to have some way to limit the overall size of the file, perhaps by having the first chunk be an integer with the promised file size. In this case, the example might look like this somewhat contrived (and somewhat insecure) Unslicer: .. code-block:: python class SomewhatLargeFileUnslicer(LeafUnslicer): opentype = ("bigfile",) def __init__(self): self.fileSize = None self.size = 0 self.output = open("/tmp/bigfile.txt", "w") def checkToken(self, typebyte, size): if self.fileSize is None: if typebyte != tokens.INT: raise BananaError("fileSize must be an INT") else: if typebyte != tokens.STRING: raise BananaError("BigFileUnslicer only accepts strings") if self.size + size > self.fileSize: raise BananaError("size limit exceeded") def receiveChild(self, obj): if self.fileSize is None: self.fileSize = obj # decide if self.fileSize is too big, raise error to refuse it else: self.output.write(obj) self.size += len(obj) def receiveClose(self): self.output.close() return open("/tmp/bigfile.txt", "r") This constrained BigFileUnslicer uses the fact that each STRING token comes with a size, which can be used to enforce the promised filesize that was provided in the first token. The data is streamed to a disk file as it arrives, so no more than CHUNKSIZE of memory is required at any given time. Streaming Slicers ~~~~~~~~~~~~~~~~~ TODO: add example The following slicer will, when the broker allows streaming, will yield the CPU to other reactor events that want processing time. (This technique becomes somewhat inefficient if there is nothing else contending for CPU time, and if this matters you might want to use something which sends N chunks before yielding, or yields only when some other known service announces that it wants CPU time, etc). .. code-block:: python class BigFileSlicer(BaseSlicer): opentype = ("bigfile",) slices = types.FileType CHUNKSIZE = 10000 def sliceBody(self, streamable, banana): while 1: chunk = self.obj.read(self.CHUNKSIZE) if not chunk: return yield chunk if streamable: d = defer.Deferred() reactor.callLater(0, d.callback, None) yield d The next example will deliver data as it becomes available from a hypothetical slow process. .. code-block:: python class OutputSlicer(BaseSlicer): opentype = ("output",) def sliceBody(self, streamable, banana): assert streamable # requires it while 1: if self.process.finished(): return chunk = self.process.read(self.CHUNKSIZE) if not chunk: d = self.process.waitUntilDataIsReady() yield d else: yield chunk Streamability is required in this example because otherwise the Slicer is required to provide chunks non-stop until the object has been completely serialized. If the process cannot deliver data, it's not like the Slicer can block waiting until it becomes ready. Prohibiting streamability is done to ensure coherency of serialized state, and the only way to guarantee this is to not let any non-Banana methods get CPU time until the object has been fully processed. Streaming Unslicers ~~~~~~~~~~~~~~~~~~~ On the receiving side, the Unslicer can be made streamable too. This is considerably easier than on the sending side, because there are fewer concerns about state coherency. A streaming Unslicer is merely one that delivers some data directly from the ``receiveChild`` method, rather than accumulating it until the ``receiveClose`` method. The SomewhatLargeFileUnslicer example from above is actually a streaming Unslicer. Nothing special needs to be done. On the other hand, it can be tricky to know where exactly to deliver the data being streamed. The streamed object is probably part of a larger structure (like a method call), where the higher-level attribute can be used to determine which object or method should be called with the incoming data as it arrives. The current Banana model is that each completed object (as returned by the child's ``receiveClose`` method) is handed to the parent's ``receiveChild`` method. The parent can do whatever it wants with the results. To make streaming Unslicers more useful, the parent should be able to set up a target for the data at the time the child Unslicer is created. More work is needed in this area to figure out how this functionality should be exposed. Arbitrary Instances are NOT serialized ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Arbitrary instances (that is, anything which does not have an ``ISlicer`` adapter) are *not* serialized. If an argument to a remote method contains one, you will get a Violation exception when you attempt to serialize it (i.e., the Deferred that you get from ``callRemote`` will errback with a Failure that contains a Violation exception). If the return value contains one, the Violation will be logged on the local side, and the remote caller will see an error just as if your method had raised a Violation itself. There are two reasons for this. The first is a security precaution: you must explicitly mark the classes that are willing to reveal their contents to the world. This reduces the chance of leaking sensitive information. The second is because it is not actually meaningful to send the contents of an arbitrary object. The recipient only gets the class name and a dictionary with the object's state. Which class should it use to create the corresponding object? It could attempt to import one based upon the classname (the approach pickle uses), but that would give a remote attacker unrestricted access to classes which could do absolutely anything: very dangerous. Both ends must be willing to transport the object. The sending side expresses this by marking the class (subclassing Copyable, or registering an ISlicer adapter). The receiving side must register the class as well, by doing registerUnslicer or using the ``opentype`` attribute in a suitable Unslicer subclass definition. PB Sequences ------------ There are several Banana sequences which are used to support the RPC mechanisms of Perspective Broker. These are in addition to the usual ones listed in the Banana docs. Top-Level Sequences ~~~~~~~~~~~~~~~~~~~ These sequences only appear at the top-level (never inside another object). +-------------------------------+-------------------------------------------------------+ | method call (``callRemote`` ) | OPEN(call) INT(request-id) INT/STR(your-reference-id) | | | STRING(interfacename) STRING(methodname) | | | (STRING(argname),argvalue).. | | | CLOSE | +-------------------------------+-------------------------------------------------------+ | method response (success) | OPEN(answer) INT(request-id) value CLOSE | +-------------------------------+-------------------------------------------------------+ | method response (exception) | OPEN(error) INT(request-id) value CLOSE | +-------------------------------+-------------------------------------------------------+ | RemoteReference.__del__ | OPEN(decref) INT(your-reference-id) CLOSE | +-------------------------------+-------------------------------------------------------+ Internal Sequences ~~~~~~~~~~~~~~~~~~ The following sequences are used to serialize PB-specific objects. They never appear at the top-level, but only as the argument value or return value (or somewhere inside them). +--------------------+--------------------------------------------+ | pb.Referenceable | OPEN(my-reference) INT(clid) | | | [OPEN(list) InterfaceList.. CLOSE] | | | CLOSE | +--------------------+--------------------------------------------+ | pb.RemoteReference | OPEN(your-reference) INT/STR(clid) | | | CLOSE | +--------------------+--------------------------------------------+ | pb.Copyable | OPEN(copyable) STRING(reflect.qual(class)) | | | (attr,value).. CLOSE | +--------------------+--------------------------------------------+ The first time a ``pb.Referenceable`` is sent, the second object is an InterfaceList, which is a list of interfacename strings, and therefore constrainable by a schema of ListOf(str) with some appropriate maximum-length restrictions. This InterfaceList describes all the Interfaces that the corresponding ``pb.Referenceable`` implements. The receiver uses this list to look up local Interfaces (and therefore Schemas) to attach to the ``pb.RemoteReference`` . This is how method schemas are checked on the sender side. This implies that Interfaces must be registered, just as classes are for ``pb.Copyable`` . TODO: what happens if an unknown Interface is received? Classes which wish to be passed by value should either inherit from ``pb.Copyable`` or have an ``ICopyable`` adapter registered for them. On the receiving side, the ``registerRemoteCopy`` function must be used to register a factory, which can be a ``pb.RemoteCopy`` subclass or something else which implements ``IRemoteCopy`` . ``Failure`` objects are sent as a ``pb.Copyable`` with a class name of "twisted.python.failure.Failure" . Implementation notes -------------------- Outgoing Referenceables ~~~~~~~~~~~~~~~~~~~~~~~ The side which holds the ``Referenceable`` uses a ReferenceableSlicer to serialize it. Each ``Referenceable`` is tracked with a "process-Unique ID" (abbreviated "puid" ). As the name implies, this number refers to a specific object within a given process: it is scoped to the process (and is never sent to another process), but it spans multiple PB connections (any given object will have the same ``puid`` regardless of which connection is referring to it). The ``puid`` is an integer, normally obtained with ``id(obj)`` , but you can override the object's ``processUniqueID`` method to use something else (this might be useful for objects that are really proxies for something else). Any two objects with the same ``puid`` are serialized identically. All Referenceables sent over the wire (as arguments or return values for remote methods) are given a "connection-local ID" (``clid`` ) which is scoped to one end of the connection. The Referenceable is serialized with this number, using a banana sequence of ``(OPEN "my-reference" clid)`` . The remote peer (the side that holds the ``RemoteReference`` ) knows the ``Referenceable`` by the ``clid`` sent to represent it. These are small integers. From a security point of view, any object sent across the wire (and thus given a ``clid`` ) is forever accessible to the remote end (or at least until the connection is dropped). The sending side uses the ``Broker.clids`` dict to map``puid`` to ``clid`` . It uses the ``Broker.localObjects`` dict to map ``clid`` to ``Referenceable`` . The reference from ``.localObjects`` also has the side-effect of making sure the Referenceable doesn't go out of scope while the remote end holds a reference. ``Broker.currentLocalID`` is used as a counter to create ``clid`` values. RemoteReference ~~~~~~~~~~~~~~~ In response to the incoming ``my-reference`` sequence, the receiving side creates a ``RemoteReference`` that remembers its Broker and the ``clid`` value. The RemoteReference is stashed in the ``Broker.remoteReferences`` weakref dictionary (which maps from ``clid`` to ``RemoteReference`` ), to make sure that a single ``Referenceable`` is always turned into the same ``RemoteReference`` . Note that this is not infallible: if the recipient forgets about the ``RemoteReference`` , PB will too. But if they really do forget about it, then they won't be able to tell that the replacement is not the same as the original [#]_ . It will have a different ``clid`` . [#]_ This ``RemoteReference`` is where the ``.callRemote`` method lives. When used to invoke remote methods, the ``clid`` is used as the second token of a ``call`` sequence. In this context, the ``clid`` is a "your-reference" : it refers to the recipient's ``.localObjects`` table. The ``Referenceable`` -holder's ``my-reference-id`` is sent back to them as the ``your-reference-id`` argument of the ``call`` sequence. The ``RemoteReference`` isn't always used to invoke remote methods: it could appear in an argument or a return value instead: the goal is to have the ``Referenceable`` -holder see their same ``Referenceable`` come back to them. In this case, the ``clid`` is used in a ``(OPEN "your-reference" clib)`` sequence. The ``Referenceable`` -holder looks up the ``clid`` in their ``.localObjects`` table and puts the result in the method argument or return value. URL References ~~~~~~~~~~~~~~ In addition to the implicitly-created numerically-indexed ``Referenceable`` instances (kept in the Broker's ``.localObjects`` dict), there are explicitly-registered string-indexed ``Referenceable`` s kept in the PBServerFactory's ``localObjects`` dictionary. This table is used to publish objects to the outside world. These objects are the targets of the ``pb.getRemoteURL`` and ``pb.callRemoteURL`` functions. To access these, a ``URLRemoteReference`` must be created that refers to a string ``clid`` instead of a numeric one. This is a simple subclass of ``RemoteReference`` : it behaves exactly the same. The ``URLRemoteReference`` is created manually by ``pb.getRemoteURL`` , rather than being generated automatically upon the receipt of a ``my-reference`` sequence. It also assumes a list of RemoteInterface names (which are usually provided by the holder of the ``Referenceable`` ). To invoke methods on a URL-indexed object, a string token is used as the ``clid`` in the "your-reference-id" argument of a ``call`` sequence. In addition, the ``clid`` of a ``your-reference`` sequence can be a string to use URL-indexed objects as arguments or return values of method invocations. This allows one side to send a ``URLRemoteReference`` to the other and have it turn into the matching ``Referenceable`` when it arrives. Of course, if it is invalid, the method call that tried to send it will fail. Note that these ``URLRemoteReference`` objects wil not survive a roundtrip like regular ``RemoteReference`` s do. The ``URLRemoteReference`` turns into a ``Referenceable`` , but the ``Referenceable`` will turn into a regular numeric (implicit) ``RemoteReference`` when it comes back. This may change in the future as the URL-based referencing scheme is developed. It might also become possible for string ``clid`` s to appear in ``my-reference`` sequences, giving ``Referenceable`` -holders the ability to publish URL references explicitly. It might also become possible to have these URLs point to other servers. In this case, a ``remote`` sequence will probably be used, rather than the ``my-reference`` sequence used for implicit references. Note that these URL-endpoints are per-Factory, so they are shared between multiple connections (the implicitly-created references are only available on the connection that created them). The PBServerFactory is created with a "root object" , which is a URL-endpoint with a ``clid`` of an empty string. .. rubric:: Footnotes .. [#] This isn't quite true: for some objects, serialization is scoped to the connection as a whole. Referenceables and RemoteReferences are like this. .. [#] unless they do something crazy like remembering the ``id(obj)`` of the old object and check to see if it is the same as that of the new one. But ``id(obj)`` is only unique among live objects anyway .. [#] and note that I think there is a race condition here, in which the reference is sent over the wire at the same time the other end forgets about it foolscap-0.13.1/doc/todo.txt0000644000076500000240000016400111477236333016311 0ustar warnerstaff00000000000000-*- outline -*- non-independent things left to do on newpb. These require deeper magic or can not otherwise be done casually. Many of these involve fundamental protocol issues, and therefore need to be decided sooner rather than later. * summary ** protocol issues *** negotiation *** VOCABADD/DEL/SET sequences *** remove 'copy' prefix from RemoteCopy type sequences? *** smaller scope for OPEN-counter reference numbers? ** implementation issues *** cred *** oldbanana compatibility *** Copyable/RemoteCopy default to __getstate__ or self.__dict__ ? *** RIFoo['bar'] vs RIFoo.bar (should RemoteInterface inherit from Interface?) *** constrain ReferenceUnslicer *** serialize target.remote_foo usefully * decide whether to accept positional args in non-constrained methods DEFERRED until after 2.0 warner: that would be awesome but let's do it _later_ This is really a backwards-source-compatibility issue. In newpb, the preferred way of invoking callRemote() is with kwargs exclusively: glyph's felt positional arguments are more fragile. If the client has a RemoteInterface, then they can convert any positional arguments into keyword arguments before sending the request. The question is what to do when the client is not using a RemoteInterface. Until recently, callRemote("bar") would try to find a matching RI. I changed that to have callRemote("bar") never use an RI, and instead you would use callRemote(RIFoo['bar']) to indicate that you want argument-checking. That makes positional arguments problematic in more situations than they were before. The decision to be made is if the OPEN(call) sequence should provide a way to convey positional args to the server (probably with numeric "names" in the (argname, argvalue) tuples). If we do this, the server (which always has the RemoteInterface) can do the positional-to-keyword mapping. But putting this in the protocol will oblige other implementations to handle them too. * change the method-call syntax to include an interfacename DONE Scope the method name to the interface. This implies (I think) one of two things: callRemote() must take a RemoteInterface argument each RemoteReference handles just a single Interface Probably the latter, maybe have the RR keep both default RI and a list of all implemented ones, then adapting the RR to a new RI can be a simple copy (and change of the default one) if the Referenceable knows about the RI. Otherwise something on the local side will need to adapt one RI to another. Need to handle reference-counting/DECREF properly for these shared RRs. From glyph: callRemote(methname, **args) # searches RIs callRemoteInterface(remoteinterface, methname, **args) # single RI getRemoteURL(url, *interfaces) URL-RRefs should turn into the original Referenceable (in args/results) (map through the factory's table upon receipt) URL-RRefs will not survive round trips. leave reference exchange for later. (like def remote_foo(): return GlobalReference(self) ) move method-invocation code into pb.Referenceable (or IReferenceable adapter). Continue using remote_ prefix for now, but make it a property of that code so it can change easily. ok, for today I'm just going to stick with remote_foo() as a low-budget decorator, so the current restrictions are 1: subclass pb.Referenceable, 2: implements() a RemoteInterface with method named "foo", 3: implement a remote_foo method and #1 will probably go away within a week or two, to be replaced by #1a: subclass pb.Referenceable OR #1b: register an IReferenceable adapter try serializing with ISliceable first, then try IReferenceable. The IReferenceable adapter must implements() some RemoteInterfaces and gets serialized with a MyReferenceSlicer. http://svn.twistedmatrix.com/cvs/trunk/pynfo/admin.py?view=markup&rev=44&root=pynfo ** use the methods of the RemoteInterface as the "method name" DONE (provisional), using RIFoo['add'] rr.callRemote(RIFoo.add, **args) Nice and concise. However, #twisted doesn't like it, adding/using arbitrary attributes of Interfaces is not clean (think about IFoo.implements colliding with RIFoo.something). rr.callRemote(RIFoo['add'], **args) RIFoo(rr).callRemote('add', **args) adaptation, or narrowing? glyph: I'm adding callRemote(RIFoo.bar, **args) to newpb right now wow. seemed like a simpler interface than callRemoteInterface("RIFoo", "bar", **args) warner: Does this mean that IPerspective can be parameterized now? warner: bad idea warner: Zope hates you! warner: zope interfaces don't support that syntax zi does support multi-adapter syntax but i don't really know what that is warner: callRemote(RIFoo.getDescriptionFor("bar"), *a, **k) glyph: yeah, I fake it. In RemoteInterfaceClass, I remove those attributes, call InterfaceClass, and then put them all back in warner: don't add 'em as attributes warner: just fix the result of __getitem__ to add a slot actually refer back to the interface radix: the problem is that IFoo['bar'] doesn't point back to IFoo warner: even better, make them callable :-) glyph: IFoo['bar'].interface == 'IFoo' RIFoo['bar']('hello') glyph: I was thinking of doing that in a later version of RemoteInterface exarkun: >>> type(IFoo['bar'].interface) right 'IFoo' Just look through all the defined interfaces for ones with matching names exarkun: ... e.g. *NOT* __main__.IFoo exarkun: AAAA you die hee hee * warner struggles to keep up with his thoughts and those of people around him * glyph realizes he has been given the power to whine glyph: ok, so with RemoteInterface.__getitem__, you could still do rr.callRemote(RIFoo.bar, **kw), right? was your objection to the interface or to the implementation? I really don't think you should add attributes to the interface ok I need to stash a table of method schemas somewhere just make __getitem__ return better type of object and ideally if this is generic we can get it into upstream Is there a reason Method.interface isn't a fully qualified name? not necessarily I have commit access to zope.interface if you have any features you want added, post to interface-dev@zope.org mailing list and if Jim Fulton is ok with them I can add them for you hmm does using RIFoo.bar to designate a remote method seem reasonable? I could always adapt it to something inside callRemote something PB-specific, that is but that adapter would have to be able to pull a few attributes off the method (name, schema, reference to the enclosing RemoteInterface) and we're really talking about __getattr__ here, not __getitem__, right? for x.y yes no, I don't think that's a good idea interfaces have all kinds od methods on them already, for introspection purposes namespace clashes are the suck unless RIFoo isn't really an Interface hm how about if it were a wrapper around a regular Interface? yeah, RemoteInterfaces are kind of a special case RIFoo(IFoo, publishedMethods=['doThis', 'doThat']) s/RIFoo/RIFoo = RemoteInterface(/ I'm confused. Why should you have to specify which methods are published? SECURITY! not actually necessary though, no and may be overkill the only reason I have it derive from Interface is so that we can do neat adapter tricks in the future that's not contradictory RIFoo(x) would still be able to do magic you wouldn't be able to check if an object provides RIFoo, though which kinda sucks but in any case I am against RIFoo.bar pity, it makes the callRemote syntax very clean hm So how come it's a RemoteInterface and not an Interface, anyway? I mean, how come that needs to be done explicitly. Can't you just write a serializer for Interface itself? * warner goes to figure out where the RemoteInterface discussion went after he got distracted maybe I should make RemoteInterface a totally separate class and just implement a couple of Interface-like methods cause rr.callRemote(IFoo.bar, a=1) just feels so clean warner: why not IFoo(rr).bar(a=1) ? hmm, also a possibility well IFoo(rr).callRemote('bar') or RIFoo, or whatever hold on, what does rr inherit from? RemoteReference it's a RemoteReference then why not IFoo(rr) / I'm keeping a strong distinction between local interfaces and remote ones ah, oka.y warner: right, you can still do RIFoo ILocal(a).meth(args) is an immediate function call in that case, I prefer rr.callRemote(IFoo.bar, a=1) .meth( is definitely bad, we need callRemote rr.callRemote("meth", args) returns a deferred radix: I don't like from foo import IFoo, RIFoo you probably wouldn't have both an IFoo and an RIFoo warner: well, look at it this way: IFoo(rr).callRemote('foo') still makes it obvious that IFoo isn't local warner: you could implement RemoteReferen.__conform__ to implement it radix: I'm thinking of providing some kind of other class that would allow .meth() to work (without the callRemote), but it wouldn't be the default plus, IFoo(rr) is how you use interfaces normally, and callRemote is how you make remote calls normally, so it seems that's the best way to do interfaces + PB hmm in that case the object returned by IFoo(rr) is just rr with a tag that sets the "default interface name" right and callRemote(methname) looks in that default interface before looking anywhere else for some reason I want to get rid of the stringyness of the method name and the original syntax (callRemoteInterface('RIFoo', 'methname', args)) felt too verbose warner: well, isn't that what your optional .meth thing is for? yes, I don't like that either using callRemote(RIFoo.bar, args) means I can just switch on the _name= argument being either a string or a (whatever) that's contained in a RemoteInterface a lot of it comes down to how adapters would be most useful when dealing with remote objects and to what extent remote interfaces should be interchangeable with local ones good point. I have never had a use case where I wanted to adapt a remote object, I don't think however, I have had use cases to send interfaces across the wire e.g. having a parameterized portal.login() interface that'll be different, just callRemote('foo', RIFoo) yeah. the current issue is whether to pass them by reference or by value eugh Can you explain it without using those words? :) hmm Do you mean, Referenceable style vs Copyable style? at the moment, when you send a Referenceable across the wire, the id-number is accompanied with a list of strings that designate which RemoteInterfaces the original claims to provide the receiving end looks up each string in a local table, and populates the RemoteReference with a list of RemoteInterface classes the table is populated by metaclass magic that runs when a 'class RIFoo(RemoteInterface)' definition is complete ok so a RemoteInterface is simply serialized as its qual(), right? so as long as both sides include the same RIFoo definition, they'll wind up with compatible remote interfaces, defining the same method names, same method schemas, etc effectively you can't just send a RemoteInterface across the wire right now, but it would be easy to add the places where they are used (sending a Referenceable across the wire) all special case them ok, and you're considering actually writing a serializer for them that sends all the information to totally reconstruct it on the other side without having the definiton yes or having some kind of debug method which give you that I'd say, do it the way you're doing it now until someone comes up with a use case for actually sending it... right the only case I can come up with is some sort of generic object browser debug tool everything else turns into a form of version negotiation which is better handled elsewhere hmm so RIFoo(rr).callRemote('bar', **kw) I guess that's not too ugly That's my vote. :) one thing it lacks is the ability to cleanly state that if 'bar' doesn't exist in RIFoo then it should signal an error whereas callRemote(RIFoo.bar, **kw) would give you an AttributeError before callRemote ever got called i.e. "make it impossible to express the incorrect usage" mmmh warner: but you _can_ check it immediately when it's called in the direction I was heading, callRemote(str) would just send the method request and let the far end deal with it, no schema-checking involved warner: which, 99% of the time, is effectively the same time as IFoo.bar would happen whereas callRemote(RIFoo.bar) would indicate that you want schema checking yeah, true hm. (that last feature is what allowed callRemote and callRemoteInterface to be merged) or, I could say that the normal RemoteReference is "untyped" and does not do schema checking but adapting one to a RemoteInterface results in a TypedRemoteReference which does do schema checking and which refuses to be invoked with method names that are not in the schema warner: we-ell warner: doing method existence checking is cool warner: but I think tying any further "schema checking" to adaptation is a bad idea yeah, that's my hunch too which is why I'd rather not use adapters to express the scope of the method name (which RemoteInterface it is supposed to be a part of) warner: well, I don't think tying it to callRemote(RIFoo.methName) would be a good idea just the same hm so that leaves rr.callRemote(RIFoo['add']) and rr.callRemoteInterface(RIFoo, 'add') OTOH, I'm inclined to think schema checking should happen by default It's just a the matter of where it's parameterized yeah, it's just that the "default" case (rr.callRemote('name')) needs to work when there aren't any RemoteInterfaces declared warner: oh but if we want to encourage people to use the schemas, then we need to make that case simple and concise * radix goes over the issue in his head again Yes, I think I still have the same position. which one? :) IFoo(rr).callRemote("foo"); which would do schema checking because schema checking is on by default when it's possible using an adaptation-like construct to declare a scope of the method name that comes later well, it _is_ adaptation, I think. Adaptation always has plugged in behavior, we're just adding a bit more :) heh it is a narrowing of capability hmm, how do you mean? rr.callRemote("foo") will do the same thing but rr.callRemote("foo") can be used without the remote interfaces I think I lost you. if rr has any RIs defined, it will try to use them (and therefore complain if "foo" does not exist in any of them, or if the schema is violated) Oh. That's strange. So it's really quite different from how interfaces regularly work... yeah except that if you were feeling clever you could use them the normal way Well, my inclination is to make them work as similarly as possible. "I have a remote reference to something that implements RIFoo, but I want to use it in some other way" s/possible/practical/ then IBar(rr) or RIBar(rr) would wrap rr in something that knows how to translate Bar methods into RIFoo remote methods Maybe it's not practical to make them very similar. I see. rr.callRemote(RIFoo.add, **kw) rr.callRemote(RIFoo['add'], **kw) RIFoo(rr).callRemote('add', **kw) I like the second one. Normal Interfaces behave like a dict, so IFoo['add'] gets you the method-describing object (z.i.i.Method). My RemoteInterfaces don't do that right now (because I remove the attributes before handing the RI to z.i), but I could probably fix that. I could either add attributes to the Method or hook __getitem__ to return something other than a Method (maybe a RemoteMethodSchema). Those Method objects have a .getSignatureInfo() which provides almost everything I need to construct the RemoteMethodSchema. Perhaps I should post-process Methods rather than pre-process the RemoteInterface. I can't tell how to use the return value trick, and it looks like the function may be discarded entirely once the Method is created, so this approach may not work. On the server side (Referenceable), subclassing Interface is nice because it provides adapters and implements() queries. On the client side (RemoteReference), subclassing Interface is a hassle: I don't think adapters are as useful, but getting at a method (as an attribute of the RI) is important. We have to bypass most of Interface to parse the method definitions differently. * create UnslicerRegistry, registerUnslicer DONE (PROVISIONAL), flat registry (therefore problematic for len(opentype)>1) consider adopting the existing collection API (getChild, putChild) for this, or maybe allow registerUnslicer() to take a callable which behaves kind of like a twisted.web isLeaf=1 resource (stop walking the tree, give all index tokens to the isLeaf=1 node) also some APIs to get a list of everything in the registry * use metaclass to auto-register RemoteCopy classes DONE ** use metaclass to auto-register Unslicer classes DONE ** and maybe Slicer classes too DONE with name 'slices', perhaps change to 'slicerForClasses'? class FailureSlicer(slicer.BaseSlicer): classname = "twisted.python.failure.Failure" slicerForClasses = (failure.Failure,) # triggers auto-register ** various registry approaches DONE There are currently three kinds of registries used in banana/newpb: RemoteInterface <-> interface name class/type -> Slicer (-> opentype) -> Unslicer (-> class/type) Copyable subclass -> copyable-opentype -> RemoteCopy subclass There are two basic approaches to representing the mappings that these registries implement. The first is implicit, where the local objects are subclassed from Sliceable or Copyable or RemoteInterface and have attributes to define the wire-side strings that represent them. On the receiving side, we make extensive use of metaclasses to perform automatic registration (taking names from class attributes and mapping them to the factory or RemoteInterface used to create the remote version). The second approach is explicit, where pb.registerRemoteInterface, pb.registerRemoteCopy, and pb.registerUnslicer are used to establish the receiving-side mapping. There isn't a clean way to do it explicitly on the sending side, since we already have instances whose classes can give us whatever information we want. The advantage of implicit is simplicity: no more questions about why my pb.RemoteCopy is giving "not unserializable" errors. The mere act of importing a module is enough to let PB create instances of its classes. The advantage of doing it explicitly is to remind the user about the existence of those maps, because the factory classes in the receiving map is precisely equal to the user's exposure (from a security point of view). See the E paper on secure-serialization for some useful concepts. A disadvantage of implicit is that you can't quite be sure what, exactly, you're exposed to: the registrations take place all over the place. To make explicit not so painful, we can use quotient's .wsv files (whitespace-separated values) which map from class to string and back again. The file could list fully-qualified classname, wire-side string, and receiving factory class on each line. The Broker (or rather the RootSlicer and RootUnslicer) would be given a set of .wsv files to define their mapping. It would get all the registrations at once (instead of having them scattered about). They could also demand-load the receive-side factory classes. For now, go implicit. Put off the decision until we have some more experience with using newpb. * move from VocabSlicer sequence to ADDVOCAB/DELVOCAB tokens Requires a .wantVocabString flag in the parser, which is kind of icky but fixes the annoying asymmetry between set (vocab sequence) and get (VOCAB token). Might want a CLEARVOCAB token too. On second thought, this won't work. There isn't room for both a vocab number and a variable-length string in a single token. It must be an open sequence. However, it could be an add/del/set-vocab sequence, allowing the vocab to be modified incrementally. ** VOCABize interface/method names One possibility is to make a list of all strings used by all known RemoteInterfaces and all their methods, then send it at broker connection time as the initial vocab map. A better one (maybe) is to somehow track what we send and add a word to the vocab once we've sent it more than three times. Maybe vocabize the pairs, as "ri/name1","ri/name2", etc, or maybe do them separately. Should do some handwaving math to figure out which is better. * nail down some useful schema syntaxes This has two parts: parsing something like a __schema__ class attribute (see the sketches in schema.xhtml) into a tree of FooConstraint objects, and deciding how to retrieve schemas at runtime from things like the object being serialized or the object being called from afar. To be most useful, the syntax needs to mesh nicely (read "is identical to") things like formless and (maybe?) atop or whatever has replaced the high-density highly-structured save-to-disk scheme that twisted.world used to do. Some lingering questions in this area: When an object has a remotely-invokable method, where does the appropriate MethodConstraint come from? Some possibilities: an attribute of the method itself: obj.method.__schema__ from inside a __schema__ attribute of the object's class from inside a __schema__ attribute of an Interface (which?) that the object implements Likewise, when a caller holding a RemoteReference invokes a method on it, it would be nice to enforce a schema on the arguments they are sending to the far end ("be conservative in what you send"). Where should this schema come from? It is likely that the sender only knows an Interface for their RemoteReference. When PB determines that an object wants to be copied by value instead of by reference (pb.Copyable subclass, Copyable(obj), schema says so), where should it find a schema to define what exactly gets copied over? A class attribute of the object's class would make sense: most objects would do this, some could override jellyFor to get more control, and others could override something else to push a new Slicer on the stack and do streaming serialization. Whatever the approach, it needs to be paralleled by the receiving side's unjellyableRegistry. * RemoteInterface instances should have an "RI-" prefix instead of "I-" DONE * merge my RemoteInterface syntax with zope.interface's I hacked up a syntax for how method definitions are parsed in RemoteInterface objects. That syntax isn't compatible with the one zope.interface uses for local methods, so I just delete them from the attribute dictionary to avoid causing z.i indigestion. It would be nice if they were compatible so I didn't have to do that. This basically translates into identifying the nifty extra flags (like priority classes, no-response) that we want on these methods and finding a z.i-compatible way to implement them. It also means thinking of SOAP/XML-RPC schemas and having a syntax that can represent everything at once. * use adapters to enable pass-by-reference or pass-by-value It should be possible to pass a reference with variable forms: rr.callRemote("foo", 1, Reference(obj)) rr.callRemote("bar", 2, Copy(obj)) This should probably adapt the object to IReferenceable or ICopyable, which are like ISliceable except they can pass the object by reference or by value. The slicing process should be: look up the type() in a table: this handles all basic types else adapt the object to ISliceable, use the result else raise an Unsliceable exception (and point the user to the docs on how to fix it) The adapter returned by IReferenceable or ICopyable should implement ISliceable, so no further adaptation will be done. * remove 'copy' prefix from remotecopy banana type names? warner: did we ever finish our conversation on the usefulness of the (copy foo blah) namespace rather than just (foo blah)? glyph: no, I don't think we did warner: do you still have (copy foo blah)? glyph: yup so far, it seems to make some things easier glyph: the sender can subclass pb.Copyable and not write any new code, while the receiver can write an Unslicer and do a registerRemoteCopy glyph: instead of the sender writing a whole slicer and the receiver registering at the top-level warner: aah glyph: although the fact that it's easier that way may be an artifact of my sucky registration scheme warner: so the advantage is in avoiding registration of each new unslicer token? warner: yes. I'm thinking that a metaclass will handily remove the need for extra junk in the protocol ;) well, the real reason is my phobia about namespace purity, of course warner: That's what the dots are for but ease of dispatch is also important warner: I'm concerned about it because I consider my use of the same idiom in the first version of PB to be a serious wart * warner nods I will put together a list of my reasoning warner: I think it's likely that PB implementors in other languages are going to want to introduce new standard "builtin" types; our "builtins" shouldn't be limited to python's provided data structures glyph: wait ok glyph: are you talking of banana types glyph: or really PB in which case (copy blah blah) is a non-builtin type, while (type-foo) is a builtin type warner: plus, our namespaces are already quite well separated, I can tell you I will never be declaring new types outside of quotient.* and twisted.* :) moshez: this is mostly banana (or what used to be jelly, really) warner: my inclination is to standardize by convention warner: *.* is a non-builtin type, [~.] is a builtin glyph: ? sorry [^.]* my regular expressions and shell globs are totally confused but you know what I mean moshez: yes glyph: hrm glyph: you're making crazy anime faces glyph: why do we need any non-Python builtin types moshez: because I want to destroy SOAP, and doing that means working with people I don't like moshez: outside of python glyph: I meant, "what specific types" I'd appreciate a blog on that * have Copyable/RemoteCopy default to __getstate__/__setstate__? At the moment, the default implementations of getStateToCopy() and setCopyableState() get and set __dict__ directly. Should the default instead be to call __getstate__() or __setstate__()? * make slicer/unslicers for pb.RemoteInterfaces exarkun's use case requires these Interfaces to be passable by reference (i.e. by name). It would also be interesting to let them be passed (and requested!) by value, so you can ask a remote peer exactly what their objects will respond to (the method names, the argument values, the return value). This also requires that constraints be serializable. do this, should be referenceable (round-trip should return the same object), should use the same registration lookup that RemoteReference(interfacelist) uses * investigate decref/Referenceable race Any object that includes some state when it is first sent across the wire needs more thought. The far end could drop the last reference (at time t=1) while a method is still pending that wants to send back the same object. If the method finishes at time t=2 but the decref isn't received until t=3, the object will be sent across the wire without the state, and the far end will receive it for the "first" time without that associated state. This kind of conserve-bandwidth optimization may be a bad idea. Or there might be a reasonable way to deal with it (maybe request the state if it wasn't sent and the recipient needs it, and delay delivery of the object until the state arrives). DONE, the RemoteReference is held until the decref has been acked. As long as the methods are executed in-order, this will prevent the race. TODO: third-party references (and other things that can cause out-of-order execution) could mess this up. * sketch out how to implement glyph's crazy non-compressed sexpr encoding * consider a smaller scope for OPEN-counter reference numbers For newpb, we moved to implicit reference numbers (counting OPEN tags instead of putting a number in the OPEN tag) because we didn't want to burn so much bandwidth: it isn't feasible to predict whether your object will need to be referenced in the future, so you always have to be prepared to reference it, so we always burn the memory to keep track of them (generally in a ScopedSlicer subclass). If we used explicit refids then we'd have to burn the bandwidth too. The sorta-problem is that these numbers will grow without bound as long as the connection remains open. After a few hours of sending 100-byte objects over a 100MB connection, you'll hit 1G-references and will have to start sending them as LONGINT tokens, which is annoying and slightly verbose (say 3 or 4 bytes of number instead of 1 or 2). You never keep track of that many actual objects, because the references do not outlive their parent ScopedSlicer. The fact that the references themselves are scoped to the ScopedSlicer suggests that the reference numbers could be too. Each ScopedSlicer would track the number of OPEN tokens emitted (actually the number of slicerForObject calls made, except you'd want to use a different method to make sure that children who return a Slicer themselves don't corrupt the OPEN count). This requires careful synchronization between the ScopedSlicers on one end and the ScopedUnslicers on the other. I suspect it would be slightly fragile. One sorta-benefit would be that a somewhat human-readable sexpr-based encoding would be even more human readable if the reference numbers stayed small (you could visually correlate objects and references more easily). The ScopedSlicer's open-parenthesis could be represented with a curly brace or something, then the refNN number would refer to the NN'th left-paren from the last left-brace. It would also make it clear that the recipient will not care about objects outside that scope. * implement the FDSlicer Over a unix socket, you can pass fds. exarkun had a presentation at PyCon04 describing the use of this to implement live application upgrade. I think that we could make a simple FDSlicer to hide the complexity of the out-of-band part of the communication. class Server(unix.Server): def sendFileDescriptors(self, fileno, data="Filler"): """ @param fileno: An iterable of the file descriptors to pass. """ payload = struct.pack("%di" % len(fileno), *fileno) r = sendmsg(self.fileno(), data, 0, (socket.SOL_SOCKET, SCM_RIGHTS, payload)) return r class Client(unix.Client): def doRead(self): if not self.connected: return try: msg, flags, ancillary = recvmsg(self.fileno()) except: log.msg('recvmsg():') log.err() else: buf = ancillary[0][2] fds = [] while buf: fd, buf = buf[:4], buf[4:] fds.append(struct.unpack("i", fd)[0]) try: self.protocol.fileDescriptorsReceived(fds) except: log.msg('protocol.fileDescriptorsReceived') log.err() return unix.Client.doRead(self) * implement AsyncDeferred returns dash wanted to implement a TransferrableReference object with a scheme that would require creating a new connection (to a third-party Broker) during ReferenceUnslicer.receiveClose . This would cause the object deserialization to be asynchronous. At the moment, Unslicers can return a Deferred from their receiveClose method. This is used by immutable containers (like tuples) to indicate that their object cannot be created yet. Other containers know to watch for these Deferreds and add a callback which will update their own entries appropriately. The implicit requirement is that all these Deferreds fire before the top-level parent object (usually a CallUnslicer) finishes. This allows for circular references involving immutable containers to be resolved into the final object graph before the target method is invoked. To accomodate Deferreds which will fire at arbitrary points in the future, it would be useful to create a marker subclass named AsyncDeferred. If an unslicer returns such an object, the container parent starts by treating it like a regular Deferred, but it also knows that its object is not "complete", and therefore returns an AsyncDeferred of its own. When the child completes, the parent can complete, etc. The difference between the two types: Deferred means that the object will be complete before the top-level parent is finished, AsyncDeferred makes claims about when the object will be finished. CallUnslicer would know that if any of its arguments are Deferreds or AsyncDeferreds then it need to hold off on the broker.doCall until all those Deferreds have fired. Top-level objects are not required to differentiate between the two types, because they do not return an object to an enclosing parent (the CallUnslicer is a child of the RootUnslicer, but it always returns None). Other issues: we'll need a schema to let you say whether you'll accept these late-bound objects or not (because if you do accept them, you won't be able to impose the same sorts of type-checks as you would on immediate objects). Also this will impact the in-order-invocation promises of PB method calls, so we may need to implement the "it is ok to run this asynchronously" flag first, then require that TransferrableReference objects are only passed to methods with the flag set. Also, it may not be necessary to have a marker subclass of Deferred: perhaps _any_ Deferred which arrives from a child is an indication that the object will not be available until an unknown time in the future, and obligates the parent to return another Deferred upwards (even though their object could be created synchronously). Or, it might be better to implement this some other way, perhaps separating "here is my object" from "here is a Deferred that will fire when my object is complete", like a call to parent.addDependency(self.deferred) or something. DONE, needs testing * TransferrableReference class MyThing(pb.Referenceable): pass r1 = MyThing() r2 = Facet(r1) g1 = Global(r1) class MyGlobalThing(pb.GloballyReferenceable): pass g2 = MyGlobalThing() g3 = Facet(g2) broker.setLocation("pb://hostname.com:8044") rem.callRemote("m1", r1) # limited to just this connection rem.callRemote("m2", Global(r1)) # can be published g3 = Global(r1) rem.callRemote("m3", g1) # can also be published.. g1.revoke() # but since we remember it, it can be revoked too g1.restrict() # and, as a Facet, we can revoke some functionality but not all rem.callRemote("m1", g2) # can be published E tarball: jsrc/net/captp/tables/NearGiftTable issues: 1: when A sends a reference on B to C, C's messages to the object referenced must arrive after any messages A sent before the reference forks in particular, if A does: B.callRemote("1", hugestring) B.callRemote("2_makeYourSelfSecure", args) C.callRemote("3_transfer", B) and C does B.callRemote("4_breakIntoYou") as soon as it gets the reference, then the A->B queue looks like (1, 2), and the A->C queue looks like (3). The transfer message can be fast, and the resulting 4 message could be delivered to B before the A->B queue manages to deliver 2. 2: an object which get passed through multiple external brokers and eventually comes home must be recognized as a local object 3: Copyables that contain RemoteReferences must be passable between hosts E cannot do all three of these at once http://www.erights.org/elib/distrib/captp/WormholeOp.html I think that it's ok to tell people who want this guarantee to explicitly serialize it like this: B.callRemote("1", hugestring) d = B.callRemote("2_makeYourSelfSecure", args) d.addCallback(lambda res: C.callRemote("3_transfer", B)) Note that E might not require that method calls even have a return value, so they might not have had a convenient way to express this enforced serialization. ** more thoughts To enforce the partial-ordering, you could do the equivalent of: A: B.callRemote("1", hugestring) B.callRemote("2_makeYourSelfSecure", args) nonce = makeNonce() B.callRemote("makeYourSelfAvailableAs", nonce) C.callRemote("3_transfer", (nonce, B.name)) C: B.callRemote("4_breakIntoYou") C uses the nonce when it connects to B. It knows the name of the reference, so it can compare it against some other reference to the same thing, but it can't actually use that name alone to get access. When the connection request arrives at B, it sees B.name (which is also unguessable), so that gives it reason to believe that it should queue C's request (that it isn't just a DoS attack). It queues it until it sees A's request to makeYourSelfAvailableAs with the matching nonce. Once that happens, it can provide the reference back to C. This implies that C won't be able to send *any* messages to B until that handshake has completed. It might be desireable to avoid the extra round-trip this would require. ** more thoughts url = PBServerFactory.registerReference(ref, name=None) creates human-readable URLs or random identifiers the factory keeps a bidirectional mapping of names and Referenceables when a Referenceable gets serialized, if the factory's table doesn't have a name for it, the factory creates a random one. This entry in the table is kept alive by two things: a live reference by one of the factory's Brokers an entry in a Broker's "gift table" When a RemoteReference gets serialized (and it doesn't point back to the receiving Broker, and thus get turned into a your-reference sequence), A->C: "I'm going to send somebody a reference to you, incref your gift table", C->A: roger that, here's a gift nonce A->B: "here's Carol's reference: URL plus nonce" B->C: "I want a liveref to your 'Carol' object, here's my ticket (nonce)", C->B: "ok, ticket redeemed, here's your liveref" once more, without nonces: A->C: "I'm going to send somebody a reference to you, incref your gift table", C->A: roger that A->B: "here's Carol's reference: URL" B->C: "I want a liveref to your 'Carol' object", C->B: "ok, here's your liveref" really: on A: c.vat.callRemote("giftYourReference", c).addCallback(step2) c is serialized as (your-reference, clid) on C: vat.remote_giftYourReference(which): self.table[which] += 1; return on A: step2: b.introduce(c) c is serialized as (their-reference, url) on B: deserialization sees their-reference newvat = makeConnection(URL) newvat.callRemote("redeemGift", URL).addCallback(step3) on C: vat.remote_redeemGift(URL): ref = self.urls[URL]; self.table[ref] -= 1; return ref ref is serialized as (my-reference, clid) on B: step3(c): b.remote_introduce(c) problem: if alice sends a thousand copies, that means these 5 messages are each send a thousand times. The makeConnection is cached, but the rest are not. We don't rememeber that we've already made this gift before, that the other end probably still has it. Hm, but we also don't know that they didn't lose it already. ** ok, a plan: concern 1: objects must be kept alive as long as there is a RemoteReference to them. concern 2: we should be able to tell when an object is being sent for the first time, to add metadata (interface list, public URL) that would be expensive to add to every occurrence. each (my-reference) sent over the wire increases the broker's refcount on both ends. the receiving Broker retains a weakref to the RemoteReference, and retains a copy of the metadata necessary to create it in the clid table (basically the entire contents of the RemoteReference). When the weakref expires, it marks the clid entry as "pending-free", and sends a decref(clid,N) to the other Broker. The decref is actually sent with broker.callRemote("decref", clid, N), so it can be acked. the sending broker gets the decref and reduces its count by N. If another reference was sent recently, this count may not drop all the way to zero, indicating there is a reference "in flight" and the far end should be ready to deal with it (by making a new RemoteReference with the same properties as the old one). If N!=0, it returns False to indicate that this was not the last decref message for the clid. If N==0, it returns True, since it is the last decref, and removes the entry from its table. Once remote_decref returns True, the clid is retired. the receiving broker receives the ack from the decref. If the ack says last==True, the clid table entry is freed. If it says last==False, then there should have been another (my-reference) received before the ack, so the refcount should be non-zero. message sequence: A-> : (my-reference clid metadata) [A.myrefs[clid].refcount++ = 1] A-> : (my-reference clid) [A.myrefs[clid].refcount++ = 2] ->B: receives my-ref, creates RR, B.yourrefs[clid].refcount++ = 1 ->B: receives my-ref, B.yourrefs[clid].refcount++ = 2 : time passes, B sees the reference go away <-B: d=brokerA.callRemote("decref", clid, B.yourrefs[clid].refcount) B.yourrefs[clid].refcount = 0; d.addCallback(B.checkref, clid) A-> : (my-reference clid) [A.myrefs[clid].refcount++ = 3] A<- : receives decref, A.myrefs[clid].refcount -= 2, now =1, returns False ->B: receives my-ref, re-creates RR, B.yourrefs[clid].refcount++ = 1 ->B: receives ack(False), B.checkref asserts refcount != 0 : time passes, B sees the reference go away again <-B: d=brokerA.callRemote("decref", clid, B.yourrefs[clid].refcount) B.yourrefs[clid].refcount = 0; d.addCallback(B.checkref, clid) A<- : receives decref, A.myrefs[clid].refcount -= 1, now =0, returns True del A.myrefs[clid] ->B: receives ack(True), B.checkref asserts refcount==0 del B.yourrefs[clid] B retains the RemoteReference data until it receives confirmation from A. Therefore whenever A sends a reference that doesn't already exist in the clid table, it is sending it to a B that doesn't know about that reference, so it needs to send the metadata. concern 3: in the three-party exchange, Carol must be kept alive until Bob has established a reference to her, even if Alice drops her carol-reference immediately after sending the introduction to Bob. (my-reference, clid, [interfaces, public URL]) (your-reference, clid) (their-reference, URL) Serializing a their-reference causes an entry to be placed in the Broker's .theirrefs[URL] table. Each time a their-reference is sent, the entry's refcount is incremented. Receiving a their-reference may initiate a PB connection to the target, followed by a getNamedReference request. When this completes (or if the reference was already available), the recipient sends a decgift message to the sender. This message includes a count, so multiple instances of the same gift can be acked as a group. The .theirrefs entry retains a reference to the sender's RemoteReference, so it cannot go away until the gift is acked. DONE, gifts are implemented, we punted on partial-ordering *** security, DoS Bob can force Alice to hold on to a reference to Carol, as long as both connections are open, by never acknowledging the gift. Alice can cause Bob to open up TCP connections to arbitrary hosts and ports, by sending third-party references to him, although the only protocol those connections will speak is PB. Using yURLs and StartTLS should be enough to secure and authenticate the connections. *** partial-ordering If we need it, the gift (their-reference message) can include a nonce, Alice sends a makeYourSelfAvailableAs message to Carol with the nonce, and Bob must do a new getReference with the nonce. Kragen came up with a good use-case for partial-ordering: A: B.callRemote("updateDocument", bigDocument) C.callRemote("pleaseReviewLatest", B) C: B.callRemote("getLatestDocument") * PBService / Tub Really, PB wants to be a Service, since third-party references mean it will need to make connections to arbitrary targets, and it may want to re-use those connections. s = pb.PBService() s.listenOn(strport) # provides URL base swissURL = s.registerReference(ref) # creates unguessable name publicURL = s.registerReference(ref, "name") # human-readable name s.unregister(URL) # also revokes all clids s.unregisterReference(ref) d = s.getReference(URL) # Deferred which fires with the RemoteReference d = s.shutdown() # close all servers and client connections DONE, this makes things quite clean * promise pipelining Even without third-party references, we can do E-style promise pipelining. hmm. subclass of Deferred that represents a Promise, can be serialized if it's being sent to the same broker as the RemoteReference it was generated for warner: hmmm. how's that help us? oh, pipelining? maybe a flag on the callRemote to say that "yeah, I want a DeferredPromise out of you, but I'm only going to include it as an argument to another method call I'm sending you, so don't bother sending *me* the result" aah yeah that sounds like a reasonable approach that would actually work dash: do you know if E makes any attempt to handle >2 vats in their pipelining implementation? seems to me it could turn into a large network optimization problem pretty quickly warner: Mmm hmm I do not think you have to so you have: t1=a.callRemote("foo",args1); t2=t1.callRemote("bar",args2), where callRemote returns a Promise, which is a special kind of Deferred that remembers the Broker its answer will eventually come from. If args2 consists of entirely immediate things (no Promises) or Promises that are coming from the same broker as t1 uses, then the "bar" call is eligible for pipelining and gets sent to the remote broker in the resulting newpb banana sequence, the clid of the target method is replaced by another kind of clid, which means "the answer you're going to send to method call #N", where N comes from t1 mmm yep using that new I-can't-unserialize-this-yet hook we added, the second call sequence doesn't finish unserializing until the first call finishes and sends the answer. Sending answer #N fires the hook's deferred. that triggers the invocation of the second method yay hm, of course that totally blows away the idea of using a Constraint on the arguments to the second method because you don't even know what the object is until after the arguments have arrived but well the first method has a schema, which includes a return constraint okay you can't fail synchronously so you *can* assert that, whatever the object will be, it obeys that constraint but you can return a failure like everybody else and since the constraint specifies an Interface, then the Interface plus mehtod name is enough to come up with an argument constraint so you can still enforce one this is kind of cool the big advantage of pipelining is that you can have a lot of composable primitives on your remote interfaces rather than having to smush them together into things that are efficient to call remotely hm, yeah, as long as all the arguments are either immediate or reference something on the recipient as soon as a third party enters the equation, you have to decide whether to wait for the arguments to resolve locally or if it might be faster to throw them at someone else that's where the network-optimization thing I mentioned before comes into play mmm you send messages to A and to B, once you get both results you want to send the pair to C to do something with them spin me an example scenario Hmm if all three are close to each other, and you're far from all of them, it makes more sense to tell C about A and B how _does_ E handle that or maybe tell A and B about C, tell them "when you get done, send your results to C, who will be waiting for them" warner: yeah, i think that the right thing to do is to wait for them to resolve locally assuming that C can talk to A and B is bad no it isn't well, depends on whether you live in this world or not :) warner: if you want other behaviour then you should have to set it up explicitly, i think I'm not even sure how you would describe that sort of thing. It'd be like routing protocols, you assign a cost to each link and hope some magical omniscient entity can pick an optimal solution ** revealing intentions Now suppose I say "B.your_fired(C.revoke_his_rights())", or such. A->C: sell all my stock. A->B: declare bankruptcy If B has access to C, and the promises are pipelined, then B has a window during which they know something's about to happen, and they still have full access to C, so they can do evil. Zooko tried to explain the concern to MarkM years ago, but didn't have a clear example of the problem. The thing is, B can do evil all the time, you're just trying to revoke their capability *before* they get wind of your intentions. Keeping intentions secret is hard, much harder than limiting someone's capabilities. It's kind of the trailing edge of the capability, as opposed to the leading edge. Zooko feels the language needs clear support for expressing how the synchronization needs to take place, and which domain it needs to happen in. * web-calculus integration Tyler pointed out that it is vital for a node to be able to grant limited access to some held object. Specifically, Alice may want to give Bob a reference not to Carol as a whole, but to just a specific Carol.remote_foo method (and not to any other methods that Alice might be allowed to invoke). I had been thinking of using RemoteInterfaces to indicate method subsets, something like this: bob.callRemote("introduce", Facet(self, RIMinimal)) but Tyler thinks that this is too coarse-grained and not likely to encourage the right kinds of security decisions. In his web-calculus, recipients can grant third-parties access to individual bound methods. bob.callRemote("introduce", carol.getMethod("howdy")) If I understand it correctly, his approach makes Referenceables into a copy-by-value object that is represented by a dictionary which maps method names to these RemoteMethod objects, so there is no actual callRemote(methname) method. Instead you do something like: rr = tub.getReference(url) d = rr['introduce'].call(args) These RemoteMethod objects are top-level, so unguessable URLs must be generated for them when they are sent, and they must be reference-counted. It must not be possible to get from the bound method to the (unrestricted) referenced object. TODO: how does the web-calculus maintain reference counts for these? It feels like there would be an awful lot of messages being thrown around. To implement this, we'll need: banana sequences for bound methods ('my-method', clid, url) ('your-method', clid) ('their-method', url, RI+methname?) syntax to carve a single method out of a local Referenceable A: self.doFoo (only if we get rid of remote_) B: self.remote_doFoo C: self.getMethod("doFoo") D: self.getMethod(RIFoo['doFoo']) leaning towards C or D syntax to carve a single method out of a RemoteReference A: rr.doFoo B: rr.getMethod('doFoo') C: rr.getMethod(RIFoo['doFoo']) D: rr['doFoo'] E: rr[RIFoo['doFoo']] leaning towards B or C decide whether to do getMethod early or late early means ('my-reference') includes a big dict of my-method values and a whole bunch of DECREFs when that dict goes away late means there is a remote_tub.getMethod(your-ref, methname) call and an extra round-trip to retrieve them dash thinks late is better We could say that the 'my-reference' sequence for any RemoteInterface-enabled Referenceable will include a dictionary of bound methods. The receiving end will just stash the whole thing. * do implicit "doFoo" -> RIFoo["doFoo"] conversion I want rr.callRemote("doFoo", args) to take advantage of a RemoteInterface, if one is available. RemoteInterfaces aren't supposed to be overlapping (at least not among RemoteInterfaces that are shared by a single Referenceable), so there shouldn't be any ambiguity. If there is, we can raise an error. * accept Deferreds as arguments? bob.callRemote("introduce", target=self.tub.getReference(pburl)) or bob.callRemote("introduce", carol.getMethod("doFoo")) instead of carol.getMethod("doFoo").addCallback(lambda r: bob.callRemote("introduce", r)) If one of the top-level arguments to callRemote is a Deferred, don't send the method request until all the arguments resolve. If any of the arguments errback, the callRemote will fail with some new exception (that can contain a reference to the argument's exception). however, this would mean the method would be invoked out-of-order w.r.t. an immediately-following bob.callRemote put this off until we get some actual experience. * batch decrefs? If we implement the copy-by-value Referenceable idea, then a single gc may result in dozens of simultaneous decrefs. It would be nice to reduce the traffic generated by that. * promise pipelining Promise(Deferred).__getattr__ DoS prevention techniques in CapIDL (MarkM) pb://key@ip,host,[ipv6],localhost,[/unix]/swissnumber tubs for lifetime management separate listener object, share tubs between listeners distinguish by key number actually, why bother with separate keys? Why allow the outside world to distinguish between these sub-Tubs? Use them purely for lifetime management, not security properties. That means a name->published-object table for each SubTub, maybe a hierarchy of them, and the parent-most Tub gets the Listeners. Incoming getReferenceByURL requests require a lookup in all Tubs that descend from the one attached to that listener. So one decision is whether to have implicitly-published objects have a name that lasts forever (well, until the Tub is destroyed), or if they should be reference-counted. If they are reference counted, then outstanding Gifts need to maintain a reference, and the gift must be turned into a live RemoteReference right away. It has bearing on how/if we implement SturdyRefs, so I need to read more about them in the E docs. Hrm, and creating new Tubs from within a remote_foo method.. to make that useful, you'd need to have a way to ask for the Tub through which you were being invoked. hrm. * creating new Tubs Tyler suggests using Tubs for namespace management. Tubs can share TCP listening ports, but MarkS recommends giving them all separate keys (which means separate SSL sessions, so separate TCP connections). Bill Frantz discourages using a hierarchy of Tubs, says it's not the sort of thing you want to be locked into. That means I'll need a separate Listener object, where the rule is that the last Tub to be stopped makes the Listener stop too.. probably abuse the Service interface in some wacky way to pull this off. Creating a new Tub.. how to conveniently create it with the same Listeners as the current one? If the method that's creating the Tub is receiving a reference, the Tub can be an attribute of the inbound RemoteReference. If not, that's trickier.. the _tub= argument may still be a useful way to go. Once you've got a source tub, then tub.newTub() should create a new one with the same Listeners as the source (but otherwise unassociated with it). Once you have the new Tub, registering an object in it should return something that can be directly serialized into a gift. class Target(pb.Referenceable): def remote_startGame(self, player_black, player_white): tub = player_black.tub.newTub() game = self.createGame() gameref = tub.register(game) game.setPlayer("black", tub.something(player_black)) game.setPlayer("white", tub.something(player_white)) return gameref Hmm. So, create a SturdyRef class, which remembers the tubid (key), list of location hints, and object name. These have a url() method that renders out a URL string, and a compare method which compares the tubid and object name but ignores the location hints. Serializing a SturdyRef creates a their-reference sequence. Tub.register takes an object (and maybe a name) and returns a SturdyRef. Tub.getReference takes either a URL or a SturdyRef. RemoteReferences should have a .getSturdyRef method. Actually, I think SturdyRefs should be serialized as Copyables, and create SturdyRefs on the other side. The new-tub sequence should be: create new tub, using the Listener from an existing tub register the objects in the new tub, obtaining a SturdyRef send/return SendLiveRef(sturdyref) to the far side SendLiveRef is a wrapper that causes a their-reference sequence to be sent. The alternative is to obtain an actual live reference (via player_black.tub.getReference(sturdyref) first), then send that, but it's kind of a waste if you don't actually want to use the liveref yourself. Note that it becomes necessary to provide for local references here: ones in different Tubs which happen to share a Listener. These can use real TCP connections (unless the Listener hint is only valid from the outside world). It might be possible to use some tricks cut out some of the network overhead, but I suspect there are reasons why you wouldn't actually want to do that. foolscap-0.13.1/doc/use-cases.txt0000644000076500000240000001052711477236333017237 0ustar warnerstaff00000000000000This file contains a collection of pb wishlist items, things it would be nice to have in newpb. * Log in, specifying desired interfaces The server can provide several different interfaces, each of which inherit from pb.IPerspective. The client can specify which of these interfaces that it desires. This change requires Jellyable interfaces, which in turn requires being able to "register jelliers sanely" (exarkun 2004-05-10). An example, in oldpb lingo: # client factory = PBClientFactory() reactor.connectTCP('localhost', portNum, factory) d = factory.login(creds, mind, IBusiness) # <-- API change d.addCallbacks(connected, disaster) # server class IBusiness(pb.IPerspective): def perspective_foo(self, bar): "Does something" class Business(pb.Avatar): __implements__ = (IBusinessInterface, pb.Avatar.__implements__) def perspective_foo(self, bar): return bar class Finance(pb.Avatar): def perspective_cash(self): """do cash""" class BizRealm: __implements__ = portal.IRealm def requestAvatar(self, avatarId, mind, *interfaces): if IBusiness in interfaces: return IBusiness, Business(avatarId, mind), lambda : None elif pb.IPerspective in interfaces: return pb.IPerspective, Finance(avatarId), lambda : None else: raise NotImplementedError * data schemas in Zope3 http://svn.zope.org/Zope3/trunk/src/zope/schema/README.txt?rev=13888&view=auto * objects that are both Referenceable and Copyable -warner I have a music player which can be controlled remotely via PB. There are server-side objects (corresponding to songs or albums) which contain both public attributes (song name, artist name) and private state (pathname to the local .ogg file, whether or not it is present in the cache). These objects may be sent to the remote end (the client) in response to either a "what are you playing right now" query, or a "tell me about all of your music" query. When they are sent down, the remote end should get an object which contains the public attributes. If the remote end sends that object back (in a "please play this song" method), the local end (the server) should get back a reference to the original Song or Album object. This requires that the original object be serialized with both some public state and a reference ID. The remote end must create a representation that contains both pieces. That representation will be serialized with just the reference ID. Ideally this should be as easy to express as marking the source object as implementing both pb.Referenceable and pb.Copyable, and the receiving object as both a pb.RemoteReference and a pb.RemoteCopy. Without this capability, my workaround is to manually assign a sequential integer to each of these referenceable objects, then send a dict of the public attributes and the index number. The recipient sends back the whole dict, and the server end only pays attention to the .index attribute. Note that I don't care about doing .callRemote on the remote object. This is a case where it might make sense to split pb.Referenceable into two pieces, one that talks about referenceability and the other that talks about callablilty (pb.Callable?). * both Callable and Copyable buildbot: remote version of BuildStatus. When a build starts, the buildmaster sends the current build to all status clients. It would be handy for them to get some static data (name, number, reason, changes) about the build at that time, plus a reference that can be used to query it again later (through callRemote). This can be done manually, but requires knowing all the places where a BuildStatus might be sent over the wire and wrapping them. I suppose it could be done with a Slicer/Unslicer pair: class CCSlicer: def slice(self, obj): yield obj yield obj.getName() yield obj.getNumber() yield obj.getReason() yield obj.getChanges() class CCUnslicer: def receiveChild(self, obj): if state == 0: self.obj = makeRemoteReference(obj); state += 1; return if state == 1: self.obj.name = obj; state += 1; return if state == 2: self.obj.reason = obj; state += 1; return if state == 3: self.obj.changes = obj; state += 1; return plus some glue to make sure the object gets added to the per-Broker references list: this makes sure the object is not sent (in full) twice, and that the receiving side keeps a reference to the slaved version. foolscap-0.13.1/doc/using-foolscap.rst0000644000076500000240000015353513204160675020273 0ustar warnerstaff00000000000000Introduction to Foolscap ======================== Introduction ------------ Suppose you find yourself in control of both ends of the wire: you have two programs that need to talk to each other, and you get to use any protocol you want. If you can think of your problem in terms of objects that need to make method calls on each other, then chances are good that you can use the Foolscap protocol rather than trying to shoehorn your needs into something like HTTP, or implementing yet another RPC mechanism. Foolscap is based upon a few central concepts: - *serialization* : taking fairly arbitrary objects and types, turning them into a chunk of bytes, sending them over a wire, then reconstituting them on the other end. By keeping careful track of object ids, the serialized objects can contain references to other objects and the remote copy will still be useful. - *remote method calls* : doing something to a local proxy and causing a method to get run on a distant object. The local proxy is called a ``RemoteReference``, and you "do something" by running its ``.callRemote`` method. The distant object is called a ``Referenceable`` , and it has methods like ``remote_foo`` that will be invoked. Foolscap is the descendant of Perspective Broker (which lived in the twisted.spread package). For many years it was known as "newpb". A lot of the API still has the name "PB" in it somewhere. These will probably go away sooner or later. A "foolscap" is a size of paper, probably measuring 17 by 13.5 inches. A twisted foolscap of paper makes a good fool's cap. Also, "cap" makes me think of capabilities, and Foolscap is a protocol to implement a distributed object-capabilities model in python. Getting Started --------------- Any Foolscap application has at least two sides: one which hosts a remotely-callable object, and another which calls (remotely) the methods of that object. We'll start with a simple example that demonstrates both ends. Later, we'll add more features like RemoteInterface declarations, and transferring object references. The most common way to make an object with remotely-callable methods is to subclass ``Referenceable``. Let's create a simple server which does basic arithmetic. You might use such a service to perform difficult mathematical operations, like addition, on a remote machine which is faster and more capable than your own [#]_ . .. code-block:: python from foolscap.api import Referenceable class MathServer(Referenceable): def remote_add(self, a, b): return a+b def remote_subtract(self, a, b): return a-b def remote_sum(self, args): total = 0 for a in args: total += a return total myserver = MathServer() On the other end of the wire (which you might call the "client" side), the code will have a ``RemoteReference`` to this object. The ``RemoteReference`` has a method named ``callRemote`` which you will use to invoke the method. It always returns a Deferred, which will fire with the result of the method. Assuming you've already acquired the ``RemoteReference`` , you would invoke the method like this: .. code-block:: python def gotAnswer(result): print "result is", result def gotError(err): print "error:", err d = remote.callRemote("add", 1, 2) d.addCallbacks(gotAnswer, gotError) Ok, now how do you acquire that ``RemoteReference`` ? How do you make the ``Referenceable`` available to the outside world? For this, we'll need to discuss the "Tub" , and the concept of a "FURL" . Tubs: The Foolscap Service -------------------------- The ``Tub`` is the container that you use to publish ``Referenceable`` s, and is the middle-man you use to access ``Referenceable`` s on other systems. It is known as the"Tub" , since it provides similar naming and identification properties as the `E language `_ 's "Vat" [#]_ . If you want to make a ``Referenceable`` available to the world, you create a Tub, tell it to listen on a TCP port, and then register the ``Referenceable`` with it under a name of your choosing. If you want to access a remote ``Referenceable`` , you create a Tub and ask it to acquire a ``RemoteReference`` using that same name. The ``Tub`` is a Twisted ``twisted.application.service.Service`` subclass, so you use it in the same way: once you've created one, you attach it to a parent Service or Application object. Once the top-level Application object has been started, the Tub will start listening on any network ports you've requested. When the Tub is shut down, it will stop listening and drop any connections it had established since last startup. If you have no parent to attach it to, you can use ``startService`` and ``stopService`` on the Tub directly. Note that no network activity will occur until the Tub's ``startService`` method has been called. This means that any ``getReference`` or ``connectTo`` requests that occur before the Tub is started will be deferred until startup. If the program forgets to start the Tub, these requests will never be serviced. A message to this effect is added to the twistd.log file to help developers discover this kind of problem. Making your Tub remotely accessible ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To make any of your ``Referenceable`` s available, you must make your Tub available. There are three parts: give it an identity, have it listen on a port, and tell it the protocol/hostname/portnumber at which that port is accessibly to the outside world. The Tub will generate its own identity, the *TubID* , by creating an SSL public key certificate and hashing it into a suitably-long random-looking string. This is the primary identifier of the Tub: everything else is just a *location hint* that suggests how the Tub might be reached. The fact that the TubID is tied to the public key allows FURLs to be "secure" references (meaning that no third party can cause you to connect to the wrong reference). You can also create a Tub with a pre-existing certificate, which is how Tubs can retain a persistent identity over multiple executions. Having the Tub listen on a TCP port is as simple as calling ``Tub.listenOn`` with a ``twisted.application.strports`` -formatted port specification string. The simplest such string would be "tcp:12345" , to listen on port 12345 on all interfaces. Using "tcp:12345:interface=127.0.0.1" would cause it to only listen on the localhost interface, making it available only to other processes on the same host. The ``strports`` module provides many other possibilities. The Tub needs to be told how it can be reached, so it knows what host and port to put into the FURLs it creates. This location is simply a string in the format "host:port" , using the host name by which that TCP port you've just opened can be reached. Foolscap cannot, in general, guess what this name is, especially if there are NAT boxes or port-forwarding devices in the way. If your machine is reachable directly over the internet as "myhost.example.com" , then you could use something like this: .. code-block:: python from foolscap.api import Tub tub = Tub() tub.listenOn("tcp:12345") # start listening on TCP port 12345 tub.setLocation("myhost.example.com:12345") If your Tub is client-only, and you don't want it to be remotely accessible, you should skip the ``listenOn`` and ``setLocation`` calls. You will be able to connect to remote objects, and objects you send over the wire will be available to the remote system, but ``registerReference`` will throw an error. Registering the Referenceable ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Once the Tub has a Listener and a location, you can publish your ``Referenceable`` to the entire world by picking a name and registering it: .. code-block:: python furl = tub.registerReference(myserver, "math-service") This returns the "FURL" for your ``Referenceable`` . Remote systems will use this FURL to access your newly-published object. The registration just maps a per-Tub name to the ``Referenceable`` : technically the same ``Referenceable`` could be published multiple times, under different names, or even be published by multiple Tubs in the same application. But in general, each program will have exactly one Tub, and each object will be registered under only one name. In this example (if we pretend the generated TubID was "ABCD" ), the FURL returned by ``registerReference`` would be ``"pb://ABCD@myhost.example.com:12345/math-service"`` . If you do not provide a name, a random (and unguessable) name will be generated for you. This is useful when you want to give access to your ``Referenceable`` to someone specific, but do not want to make it possible for someone else to acquire it by guessing the name. Note that the FURL can come from anywhere: typed in by the user, retrieved from a web page, or hardcoded into the application. Using a persistent certificate ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The Tub uses a TLS public-key certificate as the base of all its cryptographic operations. If you don't give it one when you create the Tub, it will generate a brand-new one. The TubID is simply the hash of this certificate, so if you are writing an application that should have a stable long-term identity, you will need to insure that the Tub uses the same certificate every time your app starts. The easiest way to do this is to pass the ``certFile=`` argument into your ``Tub()`` constructor call. This argument provides a filename where you want the Tub to store its certificate. The first time the Tub is started (when this file does not exist), the Tub will generate a new certificate and store it here. On subsequent invocations, the Tub will read the earlier certificate from this location. Make sure this filename points to a writable location, and that you pass the same filename to ``Tub()`` each time. Using a Persistent FURL ^^^^^^^^^^^^^^^^^^^^^^^ It is often useful to insure that a given Referenceable's FURL is both unguessable and stable, remaining the same from one invocation of the program that hosts it to the next. One (bad) way to do this is to have the programmer choose an unguessable name, embed it in the program, and pass it into ``registerReference`` each time the program runs, but of course this means that the name will be visible to anyone who sees the source code for the program, and the same name will be used by all copies of the program everywhere. A better approach is to use the ``furlFile=`` argument. This argument provides a filename that is used to hold the stable FURL for this object. If the furlfile exists when ``registerReference`` is called, the Tub will use the name inside it when constructing the new FURL. If it doesn't exist, it will create a new (unguessable) name. The new FURL will always be written into the furlfile afterwards. In addition, the tubid in the old FURL will be checked against the current Tub's tubid to make sure it matches. (this means that if you use furlFile=, you should also use the certFile= argument when constructing the Tub). Retrieving a RemoteReference ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ On the "client" side, you also need to create a Tub, although you don't need to perform the (``listenOn`` , ``setLocation`` , ``registerReference`` ) sequence unless you are also publishing` `Referenceable`` s to the world. To acquire a reference to somebody else's object, just use ``Tub.getReference`` : .. code-block:: python from foolscap.api import Tub tub = Tub() tub.startService() d = tub.getReference("pb://ABCD@myhost.example.com:12345/math-service") def gotReference(remote): print "Got the RemoteReference:", remote def gotError(err): print "error:", err d.addCallbacks(gotReference, gotError) ``getReference`` returns a Deferred which will fire with a ``RemoteReference`` that is connected to the remote ``Referenceable`` named by the FURL. It will use an existing connection, if one is available, and it will return an existing ``RemoteReference`` , it one has already been acquired. Since ``getReference`` requests are queued until the Tub starts, the following will work too. But don't forget to call ``tub.startService()`` eventually, otherwise your program will hang forever. .. code-block:: python from foolscap.api import Tub tub = Tub() d = tub.getReference("pb://ABCD@myhost.example.com:12345/math-service") def gotReference(remote): print "Got the RemoteReference:", remote def gotError(err): print "error:", err d.addCallbacks(gotReference, gotError) tub.startService() Complete example ~~~~~~~~~~~~~~~~ Here are two programs, one implementing the server side of our remote-addition protocol, the other behaving as a client. When running this example, you must copy the FURL printed by the server and provide it as an argument to the client. Both of these are standalone programs (you just run them), but normally you would create an ``twisted.application.service.Application`` object and pass the file to ``twistd -noy`` . An example of that usage will be provided later. (doc/listings/pb2server.py) .. code-block:: python #! /usr/bin/python from twisted.internet import reactor from foolscap.api import Referenceable, Tub class MathServer(Referenceable): def remote_add(self, a, b): return a+b def remote_subtract(self, a, b): return a-b myserver = MathServer() tub = Tub(certFile="pb2server.pem") tub.listenOn("tcp:12345") tub.setLocation("localhost:12345") url = tub.registerReference(myserver, "math-service") print "the object is available at:", url tub.startService() reactor.run() (doc/listings/pb2client.py) .. code-block:: python #! /usr/bin/python import sys from twisted.internet import reactor from foolscap.api import Tub def gotError1(why): print "unable to get the RemoteReference:", why reactor.stop() def gotError2(why): print "unable to invoke the remote method:", why reactor.stop() def gotReference(remote): print "got a RemoteReference" print "asking it to add 1+2" d = remote.callRemote("add", a=1, b=2) d.addCallbacks(gotAnswer, gotError2) def gotAnswer(answer): print "the answer is", answer reactor.stop() if len(sys.argv) < 2: print "Usage: pb2client.py URL" sys.exit(1) url = sys.argv[1] tub = Tub() tub.startService() d = tub.getReference(url) d.addCallbacks(gotReference, gotError1) reactor.run() (server output) .. code-block:: console % doc/listings/pb2server.py the object is available at: pb://j7oxoz3qzdkpgxgefsqp6xgdqeq4pvad@localhost:12345/math-service (client output) .. code-block:: console % doc/listings/pb2client.py pb://j7oxoz3qzdkpgxgefsqp6xgdqeq4pvad@localhost:12345/math-service got a RemoteReference asking it to add 1+2 the answer is 3 % FURLs ~~~~~ In Foolscap, each world-accessible Referenceable has one or more FURLs which are "secure" , where we use the capability-security definition of the term, meaning those FURLs have the following properties: - The only way to acquire the FURL is either to get it from someone else who already has it, or to be the person who published it in the first place. - Only that original creator of the FURL gets to determine which Referenceable it will connect to. If your ``tub.getReference(url)`` call succeeds, the Referenceable you will be connected to will be the right one. To accomplish the first goal, FURLs must be unguessable. You can register the reference with a human-readable name if your intention is to make it available to the world, but in general you will let ``tub.registerReference`` generate a random name for you, preserving the unguessability property. To accomplish the second goal, the cryptographically-secure TubID is used as the primary identifier, and the "location hints" are just that: hints. If DNS has been subverted to point the hostname at a different machine, or if a man-in-the-middle attack causes you to connect to the wrong box, the TubID will not match the remote end, and the connection will be dropped. These attacks can cause a denial-of-service, but they cannot cause you to mistakenly connect to the wrong target. The format of a FURL, like ``pb://abcd123@example.com:5901,backup.example.com:8800/math-server`` , is as follows [#]_ : #. The literal string ``pb://`` #. The TubID (as a base32-encoded hash of the SSL certificate) #. A literal ``@`` sign #. A comma-separated list of "location hints" . Each is one of the following: - TCP over IPv4 via DNS: ``HOSTNAME:PORTNUM`` - TCP over IPv4 without DNS: ``A.B.C.D:PORTNUM`` - TCP over IPv6: (TODO, maybe ``tcp6:HOSTNAME:PORTNUM`` ? - TCP over IPv6 w/o DNS: (TODO, maybe ``tcp6:[X:Y::Z]:PORTNUM``) - Unix-domain socket: (TODO) Each location hint is attempted in turn. Servers can return a "redirect" , which will cause the client to insert the provided redirect targets into the hint list and start trying them before continuing with the original list. #. A literal ``/`` character #. The reference's name (Unix-domain sockets are represented with only a single location hint, in the format ``pb://ABCD@unix/path/to/socket/NAME`` , but this needs some work) Clients vs Servers, Names and Capabilities ------------------------------------------ It is worthwhile to point out that Foolscap is a symmetric protocol. ``Referenceable`` instances can live on either side of a wire, and the only difference between "client" and "server" is who publishes the object and who initiates the network connection. In any Foolscap-using system, the very first object exchanged must be acquired with a ``tub.getReference(url)`` call [#]_ , which means it must have been published with a call to ``tub.registerReference(ref, name)`` . After that, other objects can be passed as an argument to (or a return value from) a remotely-invoked method of that first object. Any suitable ``Referenceable`` object that is passed over the wire will appear on the other side as a corresponding ``RemoteReference`` . It is not necessary to ``registerReference`` something to let it pass over the wire. The converse of this property is thus: if you do *not* ``registerReference`` a particular ``Referenceable`` , and you do *not* give it to anyone else (by passing it in an argument to somebody's remote method, or return it from one of your own), then nobody else will be able to get access to that ``Referenceable`` . This property means the ``Referenceable`` is a "capability" , as holding a corresponding ``RemoteReference`` gives someone a power that they cannot acquire in any other way [#]_ In the following example, the first program creates an RPN-style ``Calculator`` object which responds to "push" , "pop" ,"add" , and "subtract" messages from the user. The user can also register an ``Observer`` , to which the Calculator sends an ``event`` message each time something happens to the calculator's state. When you consider the ``Calculator`` object, the first program is the server and the second program is the client. When you think about the ``Observer`` object, the first program is a client and the second program is the server. It also happens that the first program is listening on a socket, while the second program initiated a network connection to the first. It *also* happens that the first program published an object under some well-known name, while the second program has not published any objects. These are all independent properties. Also note that the Calculator side of the example is implemented using ``twisted.application.service.Application`` object, which is the way you'd normally build a real-world application. You therefore use ``twistd`` to launch the program. The User side is written with the same ``reactor.run()`` style as the earlier example. The server registers the Calculator instance and prints the FURL at which it is listening. You need to pass this FURL to the client program so it knows how to contact the server. (doc/listings/pb3calculator.py) .. code-block:: python #! /usr/bin/python from twisted.application import service from twisted.internet import reactor from foolscap.api import Referenceable, Tub class Calculator(Referenceable): def __init__(self): self.stack = [] self.observers = [] def remote_addObserver(self, observer): self.observers.append(observer) def log(self, msg): for o in self.observers: o.callRemote("event", msg=msg) def remote_removeObserver(self, observer): self.observers.remove(observer) def remote_push(self, num): self.log("push(%d)" % num) self.stack.append(num) def remote_add(self): self.log("add") arg1, arg2 = self.stack.pop(), self.stack.pop() self.stack.append(arg1 + arg2) def remote_subtract(self): self.log("subtract") arg1, arg2 = self.stack.pop(), self.stack.pop() self.stack.append(arg2 - arg1) def remote_pop(self): self.log("pop") return self.stack.pop() tub = Tub() tub.listenOn("tcp:12345") tub.setLocation("localhost:12345") url = tub.registerReference(Calculator(), "calculator") print "the object is available at:", url application = service.Application("pb2calculator") tub.setServiceParent(application) if __name__ == '__main__': raise RuntimeError("please run this as 'twistd -noy pb3calculator.py'") (doc/listings/pb3user.py) .. code-block:: python #! /usr/bin/python import sys from twisted.internet import reactor from foolscap.api import Referenceable, Tub class Observer(Referenceable): def remote_event(self, msg): print "event:", msg def printResult(number): print "the result is", number def gotError(err): print "got an error:", err def gotRemote(remote): o = Observer() d = remote.callRemote("addObserver", observer=o) d.addCallback(lambda res: remote.callRemote("push", num=2)) d.addCallback(lambda res: remote.callRemote("push", num=3)) d.addCallback(lambda res: remote.callRemote("add")) d.addCallback(lambda res: remote.callRemote("pop")) d.addCallback(printResult) d.addCallback(lambda res: remote.callRemote("removeObserver", observer=o)) d.addErrback(gotError) d.addCallback(lambda res: reactor.stop()) return d url = sys.argv[1] tub = Tub() tub.startService() d = tub.getReference(url) d.addCallback(gotRemote) reactor.run() (server output) .. code-block:: console % twistd -noy doc/listings/pb3calculator.py 15:46 PDT [-] Log opened. 15:46 PDT [-] twistd 2.4.0 (/usr/bin/python 2.4.4) starting up 15:46 PDT [-] reactor class: twisted.internet.selectreactor.SelectReactor 15:46 PDT [-] Loading doc/listings/pb3calculator.py... 15:46 PDT [-] the object is available at: pb://5ojw4cv4u4d5cenxxekjukrogzytnhop@localhost:12345/calculator 15:46 PDT [-] Loaded. 15:46 PDT [-] foolscap.pb.Listener starting on 12345 15:46 PDT [-] Starting factory (client output) .. code-block:: console % doc/listings/pb3user.py \ pb://5ojw4cv4u4d5cenxxekjukrogzytnhop@localhost:12345/calculator event: push(2) event: push(3) event: add event: pop the result is 5 % Invoking Methods, Method Arguments ---------------------------------- As you've probably already guessed, all the methods with names that begin with ``remote_`` will be available to anyone who manages to acquire a corresponding ``RemoteReference`` . ``remote_foo`` matches a ``ref.callRemote("foo")`` , etc. This name lookup can be changed by overriding ``Referenceable`` (or, perhaps more usefully, implementing an ``foolscap.ipb.IRemotelyCallable`` adapter). The arguments of a remote method may be passed as either positional parameters (``foo(1,2)`` ), or as keyword args (``foo(a=1,b=2)`` ), or a mixture of both. The usual python rules about not duplicating parameters apply. You can pass all sorts of normal objects to a remote method: strings, numbers, tuples, lists, and dictionaries. The serialization of these objects is handled by the "Banana" protocol, defined in (doc/specifications/banana), which knows how to convey arbitrary object graphs over the wire. Things like containers which contain multiple references to the same object, and recursive references (cycles in the object graph) are all handled correctly [#]_ . Passing instances is handled specially. Foolscap will not send anything over the wire that it does not know how to serialize, and (unlike the standard ``pickle`` module) it will not make assumptions about how to handle classes that that have not been explicitly marked as serializable. This is for security, both for the sender (making sure you don't pass anything over the wire that you didn't intend to let out of your security perimeter), and for the recipient (making sure outsiders aren't allowed to create arbitrary instances inside your memory space, and therefore letting them run somewhat arbitrary code inside *your* perimeter). Sending ``Referenceable`` s is straightforward: they always appear as a corresponding ``RemoteReference`` on the other side. You can send the same ``Referenceable`` as many times as you like, and it will always show up as the same ``RemoteReference`` instance. A distributed reference count is maintained, so as long as the remote side hasn't forgotten about the ``RemoteReference`` , the original ``Referenceable`` will be kept alive. Sending ``RemoteReference`` s fall into two categories. If you are sending a ``RemoteReference`` back to the Tub that you got it from, they will see their original ``Referenceable`` . If you send it to some other Tub, they will (eventually) see a ``RemoteReference`` of their own. This last feature is called an "introduction" , and has a few additional requirements: see the "Introductions" section of this document for details. Sending instances of other classes requires that you tell Banana how they should be serialized. ``Referenceable`` is good for copy-by-reference semantics [#]_ . For copy-by-value semantics, the easiest route is to subclass ``foolscap.copyable.Copyable`` . See the "Copyable" section for details. Note that you can also register an ``ICopyable`` adapter on third-party classes to avoid subclassing. You will need to register the ``Copyable`` 's name on the receiving end too, otherwise Banana will not know how to unserialize the incoming data stream. When returning a value from a remote method, you can do all these things, plus two more. If you raise an exception, the caller's Deferred will have the errback fired instead of the callback, with a ``foolscap.call.CopiedFailure`` instance that describes what went wrong. The ``CopiedFailure`` is not quite as useful as a local ``twisted.python.failure.Failure`` object would be: see the "failures" document for details. The other alternative is for your method to return a ``Deferred`` . If this happens, the caller will not actually get a response until you fire that Deferred. This is useful when the remote operation being requested cannot complete right away. The caller's Deferred will fire with whatever value you eventually fire your own Deferred with. If your Deferred is errbacked, their Deferred will be errbacked with a ``CopiedFailure`` . Constraints and RemoteInterfaces -------------------------------- One major feature introduced by Foolscap (relative to oldpb) is the serialization ``foolscap.schema.Constraint`` . This lets you place limits on what kind of data you are willing to accept, which enables safer distributed programming. Typically python uses "duck typing" , wherein you usually just throw some arguments at the method and see what happens. When you are less sure of the origin of those arguments, you may want to be more circumspect. Enforcing type checking at the boundary between your code and the outside world may make it safer to use duck typing inside those boundaries. The type specifications also form a convenient remote API reference you can publish for prospective clients of your remotely-invokable service. In addition, these Constraints are enforced on each token as it arrives over the wire. This means that you can calculate a (small) upper bound on how much received data your program will store before it decides to hang up on the violator, minimizing your exposure to DoS attacks that involve sending random junk at you. There are three pieces you need to know about: Tokens, Constraints, and RemoteInterfaces. Tokens ~~~~~~ The fundamental unit of serialization is the Banana Token. These are thoroughly documented in the Banana Specification, but what you need to know here is that each piece of non-container data, like a string or a number, is represented by a single token. Containers (like lists and dictionaries) are represented by a special OPEN token, followed by tokens for everything that is in the container, followed by the CLOSE token. Everything Banana does is in terms of these nested OPEN/stuff/stuff/CLOSE sequences of tokens. Each token consists of a header, a type byte, and an optional body. The header is always a base-128 number with a maximum of 64 digits, and the type byte is always a single byte. The length of the body (if present) is indicated by the number encoded in the header. The length-first token format means that the receiving system never has to accept more than 65 bytes before it knows the type and size of the token, at which point it can make a decision about accepting or rejecting the rest of it. Constraints ~~~~~~~~~~~ The schema ``foolscap.schema`` module has a variety of ``foolscap.schema.Constraint`` classes that can be applied to incoming data. Most of them correspond to typical Python types, e.g. ``foolscap.schema.ListOf`` matches a list, with a certain maximum length, and a child ``Constraint`` that gets applied to the contents of the list. You can nest ``Constraint`` s in this way to describe the "shape" of the object graph that you are willing to accept. At any given time, the receiving Banana protocol has a single ``Constraint`` object that it enforces against the inbound data stream [#]_ . RemoteInterfaces ~~~~~~~~~~~~~~~~ The ``foolscap.remoteinterface.RemoteInterface`` is how you describe your constraints. You can provide a constraint for each argument of each method, as well as one for the return value. You can also specify additional flags on the methods. The convention (which is actually enforced by the code) is to name ``RemoteInterface`` objects with an "RI" prefix, like ``RIFoo`` . ``RemoteInterfaces`` are created and used a lot like the usual ``zope.interface`` -style ``Interface`` . They look like class definitions, inheriting from ``RemoteInterface`` . For each method, the default value of each argument is used to create a ``Constraint`` for that argument. Basic types (``int`` , ``str`` , ``bool`` ) are converted into a ``Constraint`` subclass (``IntegerConstraint`` , ``StringConstraint`` , ``BooleanConstraint``). You can also use instances of other ``Constraint`` subclasses, like ``foolscap.schema.ListOf`` and ``foolscap.schema.DictOf`` . This ``Constraint`` will be enforced against the value for the given argument. Unless you specify otherwise, remote callers must match all the ``Constraint`` s you specify, all arguments listed in the RemoteInterface must be present, and no arguments outside that list will be accepted. Note that, like zope.interface, these methods should **not** include "``self``" in their argument list. This is because you are documenting how *other* people invoke your methods. ``self`` is an implementation detail. ``RemoteInterface`` will complain if you forget. The "methods" in a ``RemoteInterface`` should return a single value with the same format as the default arguments: either a basic type (``int`` , ``str`` , etc) or a ``Constraint`` subclass. This ``Constraint`` is enforced on the return value of the method. If you are calling a method in somebody else's process, the argument constraints will be applied as a courtesy ("be conservative in what you send"), and the return value constraint will be applied to prevent the server from doing evil things to you. If you are running a method on behalf of a remote client, the argument constraints will be enforced to protect *you* , while the return value constraint will be applied as a courtesy. Attempting to send a value that does not satisfy the Constraint will result in a ``foolscap.Violation`` exception being raised. You can also specify methods by defining attributes of the same name in the ``RemoteInterface`` object. Each attribute value should be an instance of ``foolscap.schema.RemoteMethodSchema`` [#]_ . This approach is more flexible: there are some constraints that are not easy to express with the default-argument syntax, and this is the only way to set per-method flags. Note that all such method-defining attributes must be set in the ``RemoteInterface`` body itself, rather than being set on it after the fact (i.e. ``RIFoo.doBar = stuff`` ). This is required because the ``RemoteInterface`` metaclass magic processes all of these attributes only once, immediately after the ``RemoteInterface`` body has been evaluated. The ``RemoteInterface`` "class" has a name. Normally this is the (short) classname [#]_ . You can override this name by setting a special ``__remote_name__`` attribute on the ``RemoteInterface`` (again, in the body). This name is important because it is externally visible: all ``RemoteReference`` s that point at your ``Referenceable`` s will remember the name of the ``RemoteInterface`` s it implements. This is what enables the type-checking to be performed on both ends of the wire. In the future, this ought to default to the **fully-qualified** classname (like ``package.module.RIFoo`` ), so that two RemoteInterfaces with the same name in different modules can co-exist. In the current release, these two RemoteInterfaces will collide (and provoke an import-time error message complaining about the duplicate name). As a result, if you have such classes (e.g. ``foo.RIBar`` and``baz.RIBar`` ), you **must** use ``__remote_name__`` to distinguish them (by naming one of them something other than``RIBar`` to avoid this error. Hopefully this will be improved in a future version, but it looks like a difficult change to implement, so the standing recommendation is to use ``__remote_name__`` on all your RemoteInterfaces, and set it to a suitably unique string (like a URI). Here's an example: .. code-block:: python from foolscap.api import RemoteInterface, schema class RIMath(RemoteInterface): __remote_name__ = "RIMath.using-foolscap.docs.foolscap.twistedmatrix.com" def add(a=int, b=int): return int # declare it with an attribute instead of a function definition subtract = schema.RemoteMethodSchema(a=int, b=int, _response=int) def sum(args=schema.ListOf(int)): return int Using RemoteInterface ~~~~~~~~~~~~~~~~~~~~~ To declare that your ``Referenceable`` responds to a particular ``RemoteInterface`` , use the normal ``implements()`` annotation: .. code-block:: python class MathServer(foolscap.Referenceable): implements(RIMath) def remote_add(self, a, b): return a+b def remote_subtract(self, a, b): return a-b def remote_sum(self, args): total = 0 for a in args: total += a return total To enforce constraints everywhere, both sides will need to know about the ``RemoteInterface`` , and both must know it by the same name. It is a good idea to put the ``RemoteInterface`` in a common file that is imported into the programs running on both sides. It is up to you to make sure that both sides agree on the interface. Future versions of Foolscap may implement some sort of checksum-verification or Interface-serialization as a failsafe, but fundamentally the ``RemoteInterface`` that *you* are using defines what *your* program is prepared to handle. There is no difference between an old client accidentally using a different version of the RemoteInterface by mistake, and a malicious attacker actively trying to confuse your code. The only promise that Foolscap can make is that the constraints you provide in the RemoteInterface will be faithfully applied to the incoming data stream, so that you don't need to do the type checking yourself inside the method. When making a remote method call, you use the ``RemoteInterface`` to identify the method instead of a string. This scopes the method name to the RemoteInterface: .. code-block:: python d = remote.callRemote(RIMath["add"], a=1, b=2) # or d = remote.callRemote(RIMath["add"], 1, 2) Pass-By-Copy ------------ You can pass (nearly) arbitrary instances over the wire. Foolscap knows how to serialize all of Python's native data types already: numbers, strings, unicode strings, booleans, lists, tuples, dictionaries, sets, and the None object. You can teach it how to serialize instances of other types too. Foolscap will not serialize (or deserialize) any class that you haven't taught it about, both for security and because it refuses the temptation to guess your intentions about how these unknown classes ought to be serialized. The simplest possible way to pass things by copy is demonstrated in the following code fragment: .. code-block:: python from foolscap.api import Copyable, RemoteCopy class MyPassByCopy(Copyable, RemoteCopy): typeToCopy = copytype = "MyPassByCopy" def __init__(self): # RemoteCopy subclasses may not accept any __init__ arguments pass def setCopyableState(self, state): self.__dict__ = state If the code on both sides of the wire import this class, then any instances of ``MyPassByCopy`` that are present in the arguments of a remote method call (or returned as the result of a remote method call) will be serialized and reconstituted into an equivalent instance on the other side. For more complicated things to do with pass-by-copy, see the documentation on ``Copyable`` . This explains the difference between ``Copyable`` and ``RemoteCopy`` , how to control the serialization and deserialization process, and how to arrange for serialization of third-party classes that are not subclasses of ``Copyable`` . Third-party References ---------------------- Another new feature of Foolscap is the ability to send ``RemoteReference`` s to third parties. The classic scenario for this is illustrated by the `three-party Granovetter diagram `_ . One party (Alice) has RemoteReferences to two other objects named Bob and Carol. She wants to share her reference to Carol with Bob, by including it in a message she sends to Bob (i.e. by using it as an argument when she invokes one of Bob's remote methods). The Foolscap code for doing this would look like: .. code-block:: python bobref.callRemote("foo", intro=carolref) When Bob receives this message (i.e. when his ``remote_foo`` method is invoked), he will discover that he's holding a fully-functional ``RemoteReference`` to the object named Carol [#]_ . He can start using this RemoteReference right away: .. code-block:: python class Bob(foolscap.Referenceable): def remote_foo(self, intro): self.carol = intro carol.callRemote("howdy", msg="Pleased to meet you", you=intro) return carol If Bob sends this ``RemoteReference`` back to Alice, her method will see the same ``RemoteReference`` that she sent to Bob. In this example, Bob sends the reference by returning it from the original ``remote_foo`` method call, but he could almost as easily send it in a separate method call. .. code-block:: python class Alice(foolscap.Referenceable): def start(self, carol): self.carol = carol d = self.bob.callRemote("foo", intro=carol) d.addCallback(self.didFoo) def didFoo(self, result): assert result is self.carol # this will be true Moreover, if Bob sends it back to *Carol* (completing the three-party round trip), Carol will see it as her original ``Referenceable`` . .. code-block:: python class Carol(foolscap.Referenceable): def remote_howdy(self, msg, you): assert you is self # this will be true In addition to this, in the four-party introduction sequence as used by the `Grant Matcher Puzzle `_ , when a Referenceable is sent to the same destination through multiple paths, the recipient will receive the same ``RemoteReference`` object from both sides. For a ``RemoteReference`` to be transferrable to third-parties in this fashion, the original ``Referenceable`` must live in a Tub which has a working listening port, and an established base FURL. It is not necessary for the Referenceable to have been published with ``registerReference`` first: if it is sent over the wire before a name has been associated with it, it will be registered under a new random and unguessable name. The ``RemoteReference`` will contain the resulting FURL, enabling it to be sent to third parties. When this introduction is made, the receiving system must establish a connection with the Tub that holds the original Referenceable, and acquire its own RemoteReference. These steps must take place before the remote method can be invoked, and other method calls might arrive before they do. All subsequent method calls are queued until the one that involved the introduction is performed. Foolscap guarantees (by default) that the messages sent to a given Referenceable will be delivered in the same order. In the future there may be options to relax this guarantee, in exchange for higher performance, reduced memory consumption, multiple priority queues, limited latency, or other features. There might even be an option to turn off introductions altogether. Also note that enabling this capability means any of your communication peers can make you create TCP connections to hosts and port numbers of their choosing. The fact that those connections can only speak the Foolscap protocol may reduce the security risk presented, but it still lets other people be annoying. If this property bothers you, you can instruct the Tub to disable these introductions. When disabled, attempts to send or receive an introduction will fail (with a Violation error). .. code-block:: python tub = Tub() tub.setOption("accept-gifts", False) Note that you should set this option before your Tub has an opportunity to connect to any other Tub. Doing this before `tub.startService()` is one approach. Connection Progress/Status -------------------------- Several steps must take place between the time your application calls ``Tub.getReference()`` and when the Deferred finally fires with the ``RemoteReference``: * the FURL must be parsed for connection hints * each hint is mapped to a connection handler * the handler yields an endpoint * Foolscap establishes a connection to that endpoint * protocol negotiation takes place * finally, the first connection that passes negotiation wins, and the others are abandoned In addition, a reference might be satisfied by an inbound connection (where the Tub is listening on some port, and the target Tub happens to make a connection to us, in response to some ``getReference()`` call made on the far side that referenced our own TubID). Many of these steps can take a long amount of time. The most obviously slow step is connection establishment, but negotiation requires a few roundtrips, and handlers can defer yielding an endpoint if they need to spin up something like Tor first. Connections can also fail: the hint might not map to any connection handler, the handler might not be able to parse the hint, the endpoint might not respond to the connection attempt, the server might fail negotiation, or the connection might be abandoned in favor of a faster one. Note, of course, that "connections" are an illusion: in a distributed system, reality consists solely of messages, which are either successfully delivered or possibly lost. The only way to know that a message has been delivered is to receive a second message which proves knowledge of the first. The "Two Generals Problem" is persistently relevant, as are the information-flow consequences of Relativity (causality among partially-overlapping light-cones, and the fact that "simultaneous" lacks meaning in a global setting). However, references in Foolscap (and its E/CapTP/VatTP ancestors) are defined in terms of connection establishment and loss. They have a monotonic lifetime: a reference starts out as pending, then becomes established, then is "live" for some period of time, then is lost. Once lost, the RemoteReference is "dead", and all attempts to send messages on it yield a DeadReferenceError. When a Foolscap Tub says a reference is "connected", it means "at some point in the past, I received acknowledgment from the far end, and I have not yet seen any evidence that a new message would be rejected or ignored". When it says a reference is "dead", it means "I have seen a message be rejected or ignored, so I have decided that I will send no further messages to this target". With those caveats, Foolscap provides a few ways to obtain information about the status of connection attempts, and to describe how any established connection was made. ``Tub.getConnectionInfoForFURL(furl)`` will give you a ``ConnectionInfo`` object for the TubID referenced by ``furl`` if our Tub is connected to the Tub that hosts the given FURL, or if it has a connection in progress to that Tub. It will return None if our Tub has never heard of the target, or if it used to have a connection in the past, but that connection was lost. It is most useful to call this just after calling ``Tub.getReference(furl)`` (e.g. when an application status display is refreshed, and you want to display information about all pending or established connections). You can also get a ``ConnectionInfo`` for an established reference by calling ``rref.getConnectionInfo()`` on the RemoteReference. Finally, if your call to ``Tub.getReference()`` fails, the Failure object will probably have a ``._connectionInfo`` attribute. Some failure pathways do not populate this attribute, so applications should use ``getattr()``, guard access with a ``hasattr()`` check, or catch-and-tolerate ``AttributeError``. The ``ConnectionInfo`` object has attributes to tell you the following: * ``ci.connected``: is False until the target is "connected" (meaning that a ``.callRemote()`` might succeed), then is True until the connection is lost, then is False again. If the connection has been lost, ``callRemote()`` is sure to fail (until a new connection is established). These methods can help track progress of outbound connections: * ``ci.connectorStatuses``: is a dictionary, where the keys are connection hints, one for each hint in the FURL that provoked the outbound connection attempt. Each value is a string that describes the current status of this hint: "no handler", "resolving hint", "connecting", "ConnectionRefusedError", "cancelled", "InvalidHintError", "negotiation", "negotiation failed:" (and an exception string), "successful", or some other error string. * ``ci.connectionHandlers``: a dictionary, where the keys are connection hints (like ``connectorStatuses()``, and each value is a string description of the connection handler that is managing that hint (e.g. "tcp" or "tor"). If not connection handler could be found for the hint, the value will be None. Once connected, the following attributes become useful to tell you when and how the connection was established: * ``ci.establishedAt``: is None until the connection is established, then is a unix timestamp (seconds since epoch) of the connection time. That timestamp will remain set (non-None) even after the connection is subsequently lost. * ``ci.winningHint``: is None until an outbound connection is successfully negotiated, then is a string with the connection hint that succeeded. If the connection was created by an inbound socket, this will remain None (in which case ``ci.listenerStatus`` will help). * ``ci.listenerStatus``: is (None, None) until an inbound connection is accepted, then is a tuple of (listener, status), both strings. This provides a description of the listener and its status ("negotiating", "successful", or "negotiation failed:" and an exception string, except that the only observable value is "successful"). If the connection was established by an *outbound* connection, this will remain (None, None). Finally, when the connection is lost, this attribute becomes useful: * ``ci.lostAt``: is None until the connection is established and then lost, then is a unix timestamp (seconds since epoch) of the connection-loss time. Note that the ``ConnectionInfo`` object is not "live": connection establishment or loss may cause the object to be replaced with a new copy. So applications should re-obtain a new object each time they want to display the current status. However ``ConnectionInfo`` objects are also not static: the Tub may keep mutating a given object (and returning the same object to ``getConnectionInfo() calls``) until it needs to replace it. Application code should obtain the ``ConnectionInfo`` object, read the necessary attributes, render them to a status display, then drop the object reference. Reconnector Status ------------------ The Reconnector (e.g. ``reconnector = Tub.connectTo(furl, callback)``) has a separate object, known as the ``ReconnectionInfo``, which describes the state of the reconnection process. Reconnectors react to a connection loss by waiting a random delay, then attempting to re-establish the connection. The delay increases with each failure, to avoid overwhelming the target with useless traffic. This results in a repeating cycle of states: * "connecting" (while a connection attempt is underway) * "connected" (once a connection is established) * "waiting" (after connection loss, during the delay before a new attempt is started) After the delay has passed, the Reconnector moves to "connecting". If the attempt succeeds, it moves to "connected". If not, it moves to "waiting". A fourth state, "unstarted", is present before the Reconnector's Tub has been started. The ``ReconnectionInfo`` object can be obtained by calling ``reconnector.getReconnectionInfo()``. It provides the following API: * ``ri.state``: a string: "unstarted", "connecting", "connected", or "waiting" * ``ri.connectionInfo``: provides the current ``ConnectionInfo`` object, which describes the most recent connection attempt or establishment. This will be None if the Reconnector is unstarted. * ``ri.lastAttempt``: provides the time (as seconds since epoch) of the start of the most recent connection attempt, specifically the timestamp of the last transition to "connecting". This will be None if the Reconnector is in the "unstarted" state. * ``ri.nextAttempt``: provides the time of the next scheduled connection establishment attempt (as seconds since epoch). This will be None if the Reconnector is not in the "waiting" state. .. rubric:: Footnotes .. [#] although really, if your client machine is too slow to perform this kind of math, it is probably too slow to run python or use a network, so you should seriously consider a hardware upgrade .. [#] but they do not provide quite the same insulation against other objects as E's Vats do. In this sense, Tubs are leaky Vats. .. [#] note that the FURL uses the same format as an `HTTPSY `_ URL .. [#] in fact, the very *very* first object exchanged is a special implicit RemoteReference to the remote Tub itself, which implements an internal protocol that includes a method named ``remote_getReference`` . The ``tub.getReference(url)`` call is turned into one step that connects to the remote Tub, and a second step which invokes remotetub.callRemote("getReference", refname) on the result .. [#] of course, the Foolscap connections must be secured with SSL (otherwise an eavesdropper or man-in-the-middle could get access), and the registered name must be unguessable (or someone else could acquire a reference), but both of these are the default. .. [#] you may not want to accept shared objects in your method arguments, as it could lead to surprising behavior depending upon how you have written your method. The ``foolscap.schema.Shared`` constraint will let you express this, and is described in the "Constraints" section of this document .. [#] In fact, if all you want is referenceability (and not callability), you can use ``foolscap.referenceable.OnlyReferenceable`` . Strictly speaking, ``Referenceable`` is both "Referenceable" (meaning it is sent over the wire using pass-by-reference semantics, and it survives a round trip) and "Callable" (meaning you can invoke remote methods on it). ``Referenceable`` should really be named ``Callable`` , but the existing name has a lot of historical weight behind it. .. [#] to be precise, each ``Unslicer`` on the receive stack has a ``Constraint`` , and the idea is that all of them get to pass judgement on the inbound token. A useful syntax to describe this sort of thing is still being worked out. .. [#] although technically it can be any object which implements the ``IRemoteMethodConstraint`` interface .. [#] ``RIFoo.__class__.__name__`` , if ``RemoteInterface`` s were actually classes, which they're not .. [#] and since Tubs are authenticated, Foolscap offers a guarantee, in the cryptographic sense, that Bob will wind up with a reference to the same object that Alice intended. The authenticated FURLs prevent DNS-spoofing and man-in-the-middle attacks. foolscap-0.13.1/LICENSE0000644000076500000240000000204511477236333015042 0ustar warnerstaff00000000000000Copyright (c) 2006-2008 Brian Warner Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. foolscap-0.13.1/Makefile0000644000076500000240000000234612766553111015477 0ustar warnerstaff00000000000000 PYTHON=python TRIAL=trial TEST=foolscap .PHONY: build test build: $(PYTHON) setup.py build test: $(TRIAL) $(TEST) test-poll: $(MAKE) test TRIAL="trial -r poll" api-docs: rm -rf doc/api PYTHONPATH=. epydoc -v -o doc/api --html -n Foolscap -u http://foolscap.lothar.com --exclude foolscap.test foolscap pyflakes: pyflakes setup.py src |sort |uniq find-trailing-spaces: find-trailing-spaces -r src setup-test-from-tarball: rm -rf sdist-test $(PYTHON) setup.py sdist -d sdist-test cd sdist-test && tar xf *.tar.gz rm sdist-test/*.tar.gz cd sdist-test && ln -s * srcdir test-from-tarball: cd sdist-test/srcdir && trial foolscap .PHONY: release _release release: @if [ "X${VERSION}" = "X" ]; then echo "must pass VERSION="; else $(MAKE) _release; fi _release: git tag -s -u AF1B4A2A -m "release foolscap-${VERSION}" foolscap-${VERSION} python setup.py sdist bdist_wheel cd dist && gpg -u AF1B4A2A -ba foolscap-${VERSION}.tar.gz cd dist && gpg -u AF1B4A2A -ba foolscap-${VERSION}-py2-none-any.whl echo "manual steps:" @echo "git push warner master foolscap-${VERSION}" @echo "update 'latest-release' tag, push -f" @echo "twine register dist/foolscap-${VERSION}-py2-none-any.whl" @echo "twine upload dist/foolscap-${VERSION}*" foolscap-0.13.1/MANIFEST.in0000644000076500000240000000043712766553111015574 0ustar warnerstaff00000000000000include ChangeLog.0.6.4 MANIFEST.in NEWS LICENSE README.packagers include doc/*.txt doc/*.rst include doc/listings/*.py include doc/specifications/*.rst include Makefile include tox.ini .coveragerc include misc/classify_foolscap.py include versioneer.py include src/foolscap/_version.py foolscap-0.13.1/misc/0000755000076500000240000000000013204747603014764 5ustar warnerstaff00000000000000foolscap-0.13.1/misc/classify_foolscap.py0000755000076500000240000000056511477236333021055 0ustar warnerstaff00000000000000 # this is a plugin for "flogtool classify-incident" or the Incident Gatherer import re TUBCON_RE = re.compile(r'^Tub.connectorFinished: WEIRD, is not in \[') def classify_incident(trigger): m = trigger.get('message', '') if TUBCON_RE.search(m): return 'foolscap-tubconnector' return None foolscap-0.13.1/NEWS0000644000076500000240000036402313204747336014543 0ustar warnerstaff00000000000000User visible changes in Foolscap -*- outline -*- * Release 0.13.1 (20-Nov-2017) This release adds a minor feature to "flappclient": it now pays attention to a pair of environment variables named $FOOLSCAP_TOR_CONTROL_PORT and $FOOLSCAP_TOR_SOCKS_PORT. If set, the client will install a connection handler that routes "tor:" -type FURLs through a Tor daemon at the given ports (both of which are endpoint descriptors, e.g. "tcp:localhost:9050"). To use this, install the "tor" extra, like "pip install foolscap[tor]". If this extra was not installed (e.g. "txtorcon" is not importable), the environment variables will be ignored. This release also improves the reliability of the unit test suite (specifically test_reconnector) on slower systems. * Release 0.13.0 (20-Nov-2017) This release fixes compatibility with the latest Twisted-17.9.0 and changes the way logfiles are encoded. Foolscap's "flogtool" event-logging system can be configured to serialize log events into "Incident Files". In previous versions, these were serialized with the stdlib "pickle" module. However a recent change to Twisted's "Failure" class made them unpickleable, causing Foolscap's unit test suite to fail, and also affect applications which foolscap.logging.log.msg() with Failures as arguments. And untrusted pickles were unsafe to load anyways. This release replaces pickle with JSON, making it safe to use "flogtool" utilities on untrusted incident files. All new incident files created by this version will use JSON, and all tools (e.g. "flogtool dump") can only handle JSON-based files. #247 This also resolves a problem with tox-2.9.0, which caused tests to not run at all because nothing was installed into the test environment. * Release 0.12.7 (26-Jul-2017) This is a minor bugfix release to help Tahoe-LAFS. It depends upon a newer version of I2P, which should handle Tahoe storage servers that listen on I2P sockets (the Tahoe executable makes an outbound connection to the local I2P daemon, however it then accepts inbound TLS connections on that same socket, which confuses the TLS negotiation because both sides appear to be "clients", and TLS requires exactly one "client" and one "server"). It also fixes a minor Tub shutdown behavior to let unit tests work more reliably. #274 * Release 0.12.6 (12-Jan-2017) This is a minor release to improve compatibility with Twisted and I2P. In this release, the Foolscap test suite no longer uses several deprecated and/or internal Twisted attributes, so it should pass cleanly on the next release of Twisted (which will probably be named Twisted-17.0.0). In addition, the I2P connection handler was enhanced to let applications pass arbitrary kwargs through to the underlying "SAM" API. Finally connection-status error messages should be slightly cleaner and provide more useful information in the face of unrecogized exceptions. * Release 0.12.5 (07-Dec-2016) ** Connection Status Reporting This release adds an object named `ConnectionInfo`, which encapsulates information about a connection (both progress while being established, and the outcome once connected). This includes which connection hint was successful, what happened with the other hints, which handlers were used for each, and when the connection was made or lost. To get one of these, use `tub.getConnectionInfoForFURL(furl)` any time after `getReference()` is called, or `rref.getConnectionInfo()` after it resolves. #267 It also adds `ReconnectionInfo`, a similar object for Reconnectors. These capture the state of reconnection process (trying, established, waiting), and will provide a `ConnectionInfo` for the most recent (possibly successful) connection attempt. The API is `reconnector.getReconnectionInfo()`. #268 For details, see "Connection Progress/Status" and "Reconnector Status" in `doc/using-foolscap.rst`. ** Connection Handler API Changes To support `ConnectionInfo`, the Connection Handler API was changed. The one backwards-incompatible change was that the `hint_to_endpoint()` method now takes a third argument, to update the status as the handler makes progress. External handler functions will need to be modified to accept this new argument, and applications which use them should declare a dependency upon the latest Foolscap version, to avoid runtime breakage. Several backwards-compatible changes were made too: handlers can provide a `describe()` method (which feeds `ConnectionInfo.connectionHandlers`), and they can now set a special attribute on any exception they raise, to further influence the status string. In addition, the `tor.control_endpoint_maker()` handler now accepts an optional second argument, which causes the maker function to be called with a additional `update_status` argument. This backwards-compatible change allows the maker function to influence the `ConnectionInfo` status too. The Tor connection handler was enhanced to report distinct statuses for the different phases of connection: launching a new copy of Tor, connecting to an existing Tor daemon, etc. ** Minor Fixes Foolscap-0.12.0 broke `flappserver create`, causing the command to hang rather than exiting cleanly (although the flappserver directory itself was probably created properly). This release finally fixes it. #271 * Release 0.12.4 (27-Sep-2016) ** Improvements The TCP connection-hint handler can now accept square-bracket-wrapped IPv6 addresses in colon-hex format. You can produce FURLs with such hints by doing this: tub.setLocation("tcp:[2001:0DB8:f00e:eb00::1]:9900") Foolscap Tubs have been using the IPv6-capable `HostnameEndpoint` since 0.11.0, so this completes the IPv6 support. Note that there are no provisions for automatically detecting the host's IPv6 addresses: applications that wish to use addresses (instead of hostnames) must discover those addresses on their own. #155 A new `tor.control_endpoint_maker()` handler function was added, which is just like `tor.control_endpoint()` but accepts a callable function, which will be invoked only when a `tor:` hint is encountered. The function can return a Deferred which yields the control endpoint. This allows lazy launching of a Tor daemon, which can also be shared with other application needs, such as listening on an Onion service. #270 * Release 0.12.3 (01-Sep-2016) ** Improvements The `tor.socks_port()` handler was replaced by `tor.socks_endpoint()`, which takes an Endpoint object (just like `tor.control_endpoint()` does). This enables applications to speak SOCKS to the Tor daemon over e.g. a Unix-domain socket. The `tor.socks_port()` API was removed, so applications using it must upgrade. #265 The `allocate_tcp_port()` utility function would occasionally return ports that were in use by other applications, when those applications bound their own port to the loopback interface (127.0.0.1). allocate_tcp_port should no longer do this. * Release 0.12.2 (28-Aug-2016) ** Improved Tor Connection Handler The `tor.control_endpoint` connection handler now properly handles the config.SocksPort response provided by the debian Tor daemon (and possibly others), which included a confusing unix-domain socket in its response. The `tor.socks_port` handler was changed to accept both hostname and port number. Using anything but "localhost" or "127.0.0.1" is highly discouraged, as it would reveal your IP address to (possibly hostile) external hosts. This change was made to support applications (e.g. Tahoe-LAFS) which accept endpoint strings to configure socks_port, but then parse them and reject anything but TCP endpoints (to match Foolscap's current limitations). Such applications ought to warn their users to use only localhost. * Release 0.12.1 (20-Aug-2016) ** Connection Handlers for SOCKS, Tor, I2P Foolscap now includes importable connection handlers for SOCKS(5a), Tor, and I2P. #242, #246, #261 These handlers require additional supporting libraries, so they must be imported separately, and a setuptools "extra feature" declaration must be used to ask for the supporting libs. For example, applications which want to use `tor:` hints (on a host with a Tor daemon running) should have a setup.py with: install_requires=["foolscap[tor]"], and the Tub setup code should do: from foolscap.connections import tor tub.addConnectionHintHandler("tor", tor.default_socks()) Full examples and docs are available in docs/connection-handlers.rst. The default connection-negotiation timeout was increased from 60s to 120s, to accomodate tor/i2p daemon startup times. * Release 0.12.0 (20-Jul-2016) ** API changes: no more automatic configuration Foolscap has moved from automatic listener configuration (randomly-allocated TCP ports, automatically-determined IP address) to using more predictable manual configuration. In our experience, the automatic configuration only worked on hosts which had external IP addresses, which (sadly) is not the case for most computers attached to the modern internet. #252 Applications must now explicitly provide Foolscap with port numbers (for Tub.listenOn) and hostnames (for Tub.setLocation). Applications are encouraged to give users configuration controls to teach Foolscap what hostname and port number it should advertise to external hosts in the FURLs it creates. See https://tahoe-lafs.org/trac/tahoe-lafs/ticket/2773 for ideas. The specific API changes were: - Tub.setLocationAutomatically() has been deprecated - Listener.getPortnum() has been deprecated - calling Tub.listenOn("tcp:0") is also deprecated: callers should allocate a port themselves (the foolscap.util.allocate_tcp_port utility function, which does not block, has been added for this purpose). Foolscap tools like "flappserver create" and "flogtool create-gatherer" will no longer try to deduce their external IP address in an attempt to build externally-reachable FURLs, and will no longer accept "tcp:0" as a listening port (they now default to specific port numbers). Instead, they have --location= and --port arguments. The user must provide '--location' with a connection-hint string like 'tcp:hostname.example.org:3117' (which is put into the server's FURLs). This must match the corresponding '--port' argument, if provided. - for all tools, if '--port' is provided, it must not be tcp:0 - 'flappserver create' now requires --location, and '--port' defaults to tcp:3116 - 'flogtool create-gatherer' requires --location, default port is tcp:3117 - 'flogtool create-incident-gatherer' does too, default is tcp:3118 For backwards-compatibility, old flappservers will have "tcp:0" written into their "BASEDIR/port" file, and an empty string in "BASEDIR/location": these must then be edited to allow the flappserver to start. For example, write "tcp:12345" into "BASEDIR/port" to assign a portnumber, and "tcp:HOSTNAME:12345" into "BASEDIR/location" to expose it in the generated FURL. ** Other API changes Tub.listenOn() now takes a string or an Endpoint (something that implements twisted.internet.interfaces.IStreamServerEndpoint). This makes it possible to listen on non-IPv4 sockets (e.g. IPv6-only sockets, or unix-domain sockets, or more exotic endpoints), as long as Tub.setLocation() is set to something which the other end's connection handlers can deal with. #203 #243 The "DefaultTCP" handler (which manages normal "tcp:HOST:PORT" connection hints) has been moved to foolscap.connections.tcp . This makes room for new Tor/I2P/SOCKS handlers to live in e.g. foolscap.connections.tor . #260 Connection handlers are now allowed to return a Deferred from hint_to_endpoint(), which should make some handlers easier to write. #262 Note that RemoteReference.notifyOnDisconnect() will be deprecated in the next release (once all internal uses have been removed from Foolscap). Applications should stop using it as soon as possible. #42 #140 #207 ** Compatibility Changes This release removes support for the old (py2.4) "sets" module. This was retained to support applications which were trying to maintain py2.4 compatibility, but it's been so long since this was necessary, it's time to remove it. ** Other Changes The internal `allocate_tcp_port()` function was fixed: unexpected kernel behavior meant that sometimes it would return a port that was actually in use. This caused unit tests to fail randomly about 5% of the time. #258 IPv6 support is nearly complete: listening on a plain TCP port will typically accept connections via both IPv4 and IPv6, and the DefaultTCP handler will do a hostname lookup that can use both A and AAAA records. So as long as your server has a DNS entry that points at its IPv6 address, and you provide the hostname to Tub.setLocation(), Foolscap will connect over IPv6. There is one piece missing for complete support: the DefaultTCP connection handler must be modified to accept square-bracketed numeric IPv6 addresses, for rare situations where the host has a known (stable) IPv6 address, but no DNS name. * Release 0.11.0 (23-Mar-2016) ** Packaging Fixes Foolscap now declares a dependency on "twisted[tls]" instead of just "twisted": the "[tls]" extra means "we need Twisted and its TLS support". That's how we ask for Twisted to depend upon service_identity and other supporting packages. By using "[tls]", we no longer need to manually depend upon service_identity ourselves. If Twisted switches to some other scheme for TLS support, this will correctly ask for that to be included. (#249) Note that we still depend on pyOpenSSL ourselves, because we need its code to control certificate validation (if Twisted actually moved away from pyOpenSSL for TLS, Foolscap might break altogether). The Twisted dependency was updated to >=16.0.0 (the current version), to get an important HostnameEndpoint fix (#155). The "flogtool", "flappserver", and "flappclient" executables are now provided as "entry_points" on all platforms, not just windows. The old bin/* scripts have been removed. The "flogtool" entrypoint was fixed (a one-character typo in the setup.py specification): apparently it was always broken on windows and nobody noticed. We now use "tox" to run tests, instead of "trial foolscap", although the latter is still fine when run in a virtualenv into which Foolscap has been installed (and is what "tox" does under the hood). This release also moves all source code from "foolscap/" to "src/foolscap/", which should avoid some confusion as to which code is being tested. Developers who work from a git checkout should manually "rm -rf foolscap" after pulling this change, because otherwise the leftover .pyc files are likely to cause spurious test failures. (#250, #251) ** partial IPv6 support Foolscap's outbound connections now use HostnameEndpoint, which means that connection hints which contain DNS names which map to AAAA (and maybe A6) records should successfully connect to those IPv6 addresses. There is not yet any support to *listen* on IPv6 ports, so this probably does not enable IPv6 completely. But a client running this release may be able to connect to server running some future IPv6-capable release and advertising v6-based hostnames. (#155) * Release 0.10.1 (21-Jan-2016) ** Packaging Fixes This release fixes a version-string management failure when the "log publisher" feature was used in a tree built from a release tarball (rather than from a git checkout). This caused a unit test failure, as well as operational failures when using `flogtool tail`. Thanks to Ramakrishnan Muthukrishnan (vu3rdd) for the catch and the patch. (#248) * Release 0.10.0 (15-Jan-2016) ** Compatibility Fixes This release is compatible with Twisted-15.3.0 through 15.5.0. A change in 15.3.0 triggered a bug in Foolscap which produced a somewhat-infinite series of log messages when run under `twistd`. This release fixes that bug, and slightly changes the semantics of calling `log.msg()` with additional parameters. (#244) Foolscap no longer claims compatibility with python-2.6.x . Twisted-15.5.0 was the last release to offer 2.6 support, and subsequent releases actively throw errors when run against 2.6, so we've turned off Foolscap's automated testing for 2.6. It may remain compatible by accident for a while. (#245) * Release 0.9.1 (21-Sep-2015) Point release to deal with PyPI upload problems. No code changes. * Release 0.9.0 (21-Sep-2015) ** Plugins for Connection Handlers (#236) New types of connection hints can now be used, by installing a suitable connection handler into the Tub. These hints could point to I2P servers or Tor hidden-service (.onion) addresses. The built-in TCP handler can be replaced entirely to protect a client's IP address by routing all connections through Tor. Implementation of these plugins are left as exercise for the reader: Foolscap only provides the built-in "DefaultTCP" handler. See doc/connection-handlers.rst for details. ** Shared Listeners are removed (#239) Until this version, it was possible to create a single Listener that serviced multiple Tubs (by passing the Listener returned from `l=tubA.listenOn(where)` into `tubB.listenOn(l)`). This seemed useful a long time ago, but in fact was not, and the implementation caused irreparable problems that were exposed while testing the new connection handlers. So support for shared Listeners has been removed: Tubs can still use multiple Listeners, but each Listener now services at most one Tub. In particular, `Tub.listenOn()` now only accepts a string, not a Listener instance. Note that relays and redirects are still on the roadmap, but neither feature requires sharing a Listener between multiple local Tubs. ** Extended-Form Connection Hints are removed Support for extended-form connection hints has been removed. These were hints with explicit key names like "tcp:host=example.org:port=12345", or "tcp:example.org:timeout=30". They were added in the 0.7.0 release, but since then we've realized that this is power that should not be granted to external FURL providers. The parser now only accepts "tcp:example.org:12345" and "example.org:12345". Foolscap has never particularly encouraged applications to call Tub.setLocation() with anything other than these two forms, so we do not expect any compatibility problems. ** Option to Disable Gifts (#126) "Gifts", more precisely known as "third-party reference introductions", occur when one Tub sends you a message that includes a reference to some object on a third Tub. This allows references to be passed around transparently, without regard to which Tub they live on (yours, mine, or theirs), but allows other Tubs to cause you to create network connections to hosts and ports of their choosing. If this bothers you, the new `tub.setOption("accept-gifts", False)` option instructs your Tub to reject these third-party references, causing the calls that used them to signal a Violation error instead. ** Unreachable Tubs now fully supported (#208) Unreachable "client-only" Tubs can be created by simply not calling either `tub.listenOn()` nor `tub.setLocation()`. These Tubs can make outbound connections, but will not accept inbound ones. `tub.registerReference()` will throw an error, and Gifts delivered to third parties will not work. Previous versions suggested using `tub.setLocation("")`: this is no longer recommended. ** new util.allocate_tcp_port() function To support a future deprecation of `Tub.listenOn("tcp:0")`, the new allocate_tcp_port() function was added to return (synchronously) a currently-unused TCP port integer. This can be used during app configuration to decide on a listening port, which can then be passed into `Tub.listenOn("tcp:%d" % portnum)`. This may allow Tub.setLocation() to be called *before* the reactor is started, simplifying application startup code (this also requires a suitable hostname or IP address, which is a separate issue). ** Packaging/Dependency Changes Foolscap now requires Twisted 10.1.0 or newer, to use Endpoints and connection handler plugins. Foolscap's logging system (specifically the twisted-to-foolscap bridge) is now compatible with Twisted-15.2.0. The previous version had problems with the new contents of twisted.logger's "eventDict" objects. (#235) * Release 0.8.0 (15-Apr-2015) ** UnauthenticatedTub is gone As announced in the previous release, UnauthenticatedTub has been removed. All Tubs are fully authenticated now. ** Security Improvements Foolscap now generates better TLS certificates, with 2048-bit RSA keys and SHA256 digests. Previous versions used OpenSSL's defaults, which typically meant 1024-bit MD5. To benefit from the new certificates, you must regenerate your Tubs, which means creating new FURLs (with new TubIDs). Previously-created Tubs will continue to work normally: only new Tubs will be different. ** Packaging/Dependency Changes setup.py now requires setuptools Foolscap now requires pyOpenSSL unconditionally, because all Tubs are authenticated. We now recommend "pip install ." to install Foolscap and all its dependencies, instead of "python setup.py install". See #231 for details. * Release 0.7.0 (23-Sep-2014) ** Security Fixes The "flappserver" feature was found to have a vulnerability in the service-lookup code which, when combined with an attacker who has the ability to write files to a location where the flappserver process could read them, would allow that attacker to obtain control of the flappserver process. Users who run flappservers should upgrade to 0.7.0, where this was fixed as part of #226. Each flappserver runs from a "base directory", and uses multiple files within the basedir to track the services that have been configured. The format of these files has changed. The flappserver tool in 0.7.0 remains capable of reading the old format (safely), but will upgrade the basedir to the new format when you use "flappserver add" to add a new service. Brand new servers, created with "flappserver create", will use the new format. The flappserver tool in 0.6.5 (or earlier) cannot handle this new format, and will believe that no services have been configured. Therefore downgrading to an older version of Foolscap will require manual reconstruction of the configured services. ** Major Changes UnauthenticatedTub has been deprecated, and will be removed in the next release (0.8.0). This seldom-used feature provides Foolscap's RPC semantics without any of the security, and was included to enable the use of Foolscap without depending upon the (challenging-to-install) PyOpenSSL library. However, in practice, the lack of a solid dependency on PyOpenSSL has made installation more difficult for applications that *do* want the security, and UnauthenticatedTub is a footgun waiting to go off. Foolscap's code and packaging will be simpler without it. (#67) ** Minor Changes The "git-foolscap" tools, which make it possible to publish and clone Git repositories over a Foolscap (flappserver) connection, have been moved from their hiding place in doc/examples/ into their own project, hosted at https://github.com/warner/git-foolscap . They will also be published on PyPI, to enable "pip install git-foolscap". The documentation was converted from Lore to ReStructuredText (.rst). Thanks to Koblaid for the patient work. (#148) The connection-hint parser in 0.7.0 has been changed to handle all TCP forms of Twisted's "Client Endpoint Descriptor" syntax, including the short "tcp:127.0.0.1:9999" variant. A future version should handle arbitrary endpoint descriptors (including Tor and i2p, see #203), but this small step should improve forward compatibility. (#216, #217) * Release 0.6.5 (12-Aug-2014) ** Compatibility Fixes This release is compatible with Twisted-14.0.0. Foolscap no longer claims compatability with python-2.4.x or 2.5.x . These old versions might still work, but there are no longer automated tests to ensure this. Future versions will almost certainly *not* work with anything older than python-2.6.x . Foolscap remains incompatible with py3, sorry. ** Forward Compatibility When parsing FURLs, the connection hints can now use TCP sockets described with the Twisted Endpoints syntax (e.g. "tcp:host=127.0.0.1:port=9999"), in addition to the earlier host:port "127.0.0.1:9999" form. Foolscap-0.6.5 ignores any hint that is not in one of these two forms. This should make it easier to introduce new hint types in the future. ** Minor Changes The "ChangeLog" file is no longer updated. Violation reports now include the method name. (#201) The "flappserver" tool explicitly rejects unicode input, rather than producing hard-to-diagnose errors later. (#209) * Release 0.6.4 (18-Jun-2012) ** Minor Changes The unreliable 'extras_require' property in setup.py, which allowed other python programs to declare a dependency on foolscap's "secure_connections" feature, was removed. See README.packagers for alternate instructions. (#174) 'flogtool' log-dumping commands (dump, tail, web-viewer) now accept a consistent --timestamps= argument to control how event times are displayed (UTC, local, seconds-since-epoch, etc). (#192, #193) Certain invalid "location" strings (accepted by Tub.setLocation and put into FURLs) are rejected earlier, and with better error messages. The error message produced when 'flogtool dump' is given a FURL-file (instead of an event log file) has been improved. The Incident Gatherer will tolerate incident-file errors better, fetching remaining incidents instead of halting. (#190) The git-over-foolscap tools were cleaned up, and the documentation was brought into line with the implementation. (#197) Other minor bugs were fixed: #179, #191, #194, #195, #196 * Release 0.6.3 (05-Jan-2012) ** Compatibility Fixes This release really is compatible with Twisted-11.1.0 . The previous Foolscap release (0.6.2), despite the changes described below, suffered mild incompatibilites with the new TLS code in the final Twisted-11.1.0 release. The most common symptom is a DirtyReactorError in unit tests that use Tub.stopService() in their tearDown() method (to coordinate shutdown and cleanup). Another symptom is tests overlapping with one another, causing port-already-in-use errors. This incompatibility did not generally affect normal operation, but only impacted unit tests. ** Other Changes The Debian packaging tools in misc/ were removed, as they were pretty stale. These days, both Debian and Ubuntu make their own Foolscap packages. * Release 0.6.2 (15-Oct-2011) ** Compatibility Fixes Foolscap-0.6.2 will be compatible with future versions of Twisted (>11.0.0). The 0.6.1 release will not: a TLS change went into Twisted trunk recently (after the 11.0.0 release) which broke Foolscap 0.6.1 and earlier. This release also fixes a minor incompatibility with newer versions of OpenSSL (0.9.8o was ok, 1.0.0d was not), which caused errors in the test suite (but normal runtime operation) on e.g. Ubuntu 11.10 "Oneiric". ** Git-Over-Foolscap Tools The doc/examples/ directory contains two executables (git-foolscap and git-remote-pb) which, when placed in your $PATH, make it easy to use Foolscap to access a Git repository. These use the flappserver/flappclient tools and let you build a FURL that provides read-only or read-write access to a single repository. This is somewhat like providing SSH access to a repo, but with a much smaller scope: the client will only be able to manipulate the one repository, and gets no other authority on the target system. See the tool's inline comments for usage instructions. ** Minor Fixes Using 'flappserver upload-file FILE1 FILE2 FILE3..' (with three or more files) now correctly uploads all files: previously it only managed to upload the first and last. 'flappserver' argument handling was improved slightly. A workaround was added to handle a Twisted stdio-closing bug which affected flappserver's run-command function and broke the git-foolscap tool. Several changes were made for the benefit of Windows: log filenames all use hyphens (not colons), log filtering tools tolerate the lack of atomic-rename filesystem operations, and some unixisms in the test suite were removed. The Tub.setLogGathererFURL() method can now accept a list (iterable) of log gatherer FURLs, not just a single one. * Release 0.6.1 (16-Jan-2011) ** Minor Fixes The old "sets" module is no longer imported without wrapping the import in a DeprecationWarning suppressor. We still import it from slicers.set for compatibility with older code, but that import will not produce a warning. This should make Foolscap quieter when used with Python 2.6 or later. A new RemoteReference method named getDataLastReceivedAt() was added, which will tell you when data was most recently received on the connection supporting that reference. This can be compared against time.time() to see how "live" the connection is. For performance reasons, this is only enabled when keepalives are turned on, otherwise it returns None. (#169) Some unreachable code was removed. (#165) * Release 0.6.0 (28-Dec-2010) ** API Changes *** "foolscap.api" now mandatory The old import names from foolscap/__init__.py have been removed, finishing the transition begun with 0.5.0 . Applications must now import Tub, Referenceable, and so on from "foolscap.api". (#122) ** Compatibility Fixes Foolscap-0.6.0 is compatible with Twisted-10.2 (released 29-Nov-2010). The 0.5.1 release was not: pb.Listener was depending upon the behavior of an internal Twisted function that changed, causing an AttributeError in "StreamServerEndpointService". This is fixed, but the code is still using an undocumented internal attribute to handle port=0 which will need to be replaced eventually. (#167) The first unit test ("test__versions") spuriously failed against Twisted-10.1 and 10.2, mistakenly believing that 10.1 was older than 8.1.0 due to a lexicographic comparison that should have been numeric. ** Other Changes Incident filenames are now like "2008-08-22--16:20:28Z.flog" which are in UTC and mostly ISO-8601 format (the real ISO-8601 would use "_" instead of "--"). This is also used for log-gatherer filenames. (#111) The logging code now honors FLOGLEVEL= when using FLOGTOTWISTED=1; previously FLOGLEVEL= was ignored when deciding which log events should be bridged to the twisted logger. (#154) Some minor packaging bugs were fixed. * Release 0.5.1 (25 Mar 2010) ** Bugfixes This release fixes a significant performance problem, causing receivers a very long time (over 10 seconds) to process large (>10MB) messages, for example when receiving a large string in method arguments. Receiver CPU time was quadratic in the size of the message. (#149) ** Other Changes This release removes some unused code involved in the now-abandoned resource-exhaustion defenses. (#127) * Release 0.5.0 (18 Jan 2010) ** Compatibility The wire format remains the same as in earlier releases. The preferred API import path has changed, see below. ** API changes: import statements, foolscap.api To reduce circular dependencies in Foolscap's internal code, a new "foolscap.api" module has been created. Applications should use: from foolscap.api import Tub instead of e.g. "from foolscap import Tub". Deprecation warnings will be raised for code which imports symbols directly from the "foolscap" module. These warnings will turn into errors in the 0.6.0 release. (see ticket #122 for details) The nearly-useless getRemoteURL_TCP() function was removed. ** setup.py is more windows-friendly The main setup.py script has been modified to use setuptools "entry_points=" on windows, which should help create runnable executables of "flogtool" and "flappserver", with proper extensions. Entry-point scripts are not used on non-windows platforms, but setuptools still creates fairly opaque executable scripts (which makes it hard to figure out that e.g. /usr/bin/flogtool wants to import the "foolscap" module). To get non-opaque scripts, install with "setup.py install --single-version-externally-managed". (#109) ** tool changes *** flappserver "flappserver create" now records the umask value from its environment, and uses it later when the server is started (since normally twistd resets the umask to a very restrictive value). A new --umask argument was added to override this. The server's base directory is chmod go-rwx to protect the private key from other users. The "flappserver start" command uses twisted.scripts.twistd.run(), instead of spawning an intermediate "twistd" process with os.execvp(). This should make things work better in environments where Twisted is not fully installed (especially on windows) and correctly launching "twistd" is non-trivial, such as when some other package is installing it as a setuptools dependency. "flappclient upload-file ~/foo.txt" will use os.path.expanduser() on the filename, even if your shell does not. This should make it easier to use from e.g. buildbot upload commands. (#134) *** logging The "flogtool dump" and "flogtool web-viewer" commands now have a --timestamps argument, which controls how timestamps are expressed (UTC vs localtime, ISO-9601, etc). The web-viewer HTML pages now have more timestamp and sorting options, and hyperlinks to select each. (#100) "flogtool web-viewer --open" will tell your local web browser to open to the correct page, using the Python stdlib "webbrowser" module. "flogtool dump" now emits a better error when told to open a missing file. *** examples Examples of running the Git version-control-system over a flappserver-based secure connection have been added to doc/examples/ . This enables remote-update authority to be expressed as a FURL with no other shell privileges. To accomplish the same with ssh "authorized_keys" command restrictions is annoying and error-prone. See doc/examples/git-proxy-flappclient for setup instructions. This will probably be simplified to a single "git-furl" executable in a later release. The xfer-client/xfer-server/command-client examples have been removed, obsoleted by the flappserver/flappclient tools. ** Other changes The DeprecationWarning for the obsolete "sets" module is now removed on python2.6 (#124) When a getReference() call fails because the remote Tub does not recognize the FURL, it now only emits the first two letters of the secret swissnum in the exception, instead of the whole thing. This reduces information leakage into e.g. stderr logs from a "flappclient --furlfile=X upload-file" command. DeadReferenceError now includes the remote tubid, interfacename, and remote method name of the message that was being sent when the lost connection was discovered, so log.err() calls which record a DeadReferenceError should include this information. This may make it easier to locate the code that provoked the error. * Release 0.4.2 (16 Jun 2009) ** Compatibility Same as 0.4.1 ** the Foolscap Application Server The big new feature in this release is the "Foolscap Application Server". This is both a demo of what you can do with Foolscap, and an easy way to deploy a few simple services that run over secure connections. You create and start a "flappserver" on one machine, and then use the new "flappclient" on the other side. The server can contain multiple services, each with a separate FURL. You give the client a FURL for a specific services, it connects, does a job, and shuts down. See doc/flappserver.xhtml for details. Two service types are provided in this release. The first is a simple file-uploader: the holder of the FURL gets to upload arbitrary files into a specific target directory, and nowhere else. The second is a pre-configured command runner: the service is configured with a shell command, and the client gets to make it run (but doesn't get to influence anything about what gets run). The run-command service defaults to sending stdout/stderr/exitcode to the client program, which will behave as if it were the command being run (stdout and stderr appear at right time, and it exits with the same exitcode). The service can be configured to accept stdin, or to turn off stdout or stderr. The service always runs in a preconfigured working directory. To do this with SSH, you'd need to create a new keypair, then set up an authorized_keys entry to limit that pubkey to a single command, and hope that environment variables and the working directory don't cause any surprises. Implementing the fixed-directory file-uploader would probably require a specialized helper program. The flappserver provides an easy-to-configure capability-based replacement those sorts of SSH setups. The first use-case is to allow buildslaves to upload newly-created debian packages to a central repository and then trigger a package-index rebuild script. By using FURLs instead of raw SSH keys, the buildslaves will be unable to affect any .debs in other directories, or any other files on the repository host, nor will they be able to run arbitrary commands on that host. By storing the FURLs in a file and using the --furlfile argument to "flappclient", a buildbot transcript of the upload step will not leak the upload authority. ** new RemoteReference APIs RemoteReference now features two new methods. rref.isConnected() returns a boolean, True if the remote connection is currently live, False if it has been lost. This is an immediate form of the rref.notifyOnDisconnect() callback-registration mechanism, and can make certain types of publish-subscribe code easier to write. The second is rref.getLocationHints(), which returns a list of location hints as advertised by the host Tub. Most hints are a ("ipv4",host,portnum) tuple, but other types may be defined in the future. Note that this is derived from the FURL that each Tub sends with its my-reference sequence (i.e. it is entirely controlled by the Tub in which that Referenceable lives), so getLocationHints() is quite distinct from rref.getPeer() (which returns an IPv4Address or LoopbackAddress instance describing the other end of the actual network connection). getLocationHints() indicates what the other Tub wants you to use for new connections, getPeer() indicates what was used for the existing connection (which might not accept new connections due to NAT or proxy issues). getLocationHints() is meant to make it easier to write connection-status display code, for example in a server which holds connections to a number of peers. A status web page can loop over the peer RemoteReferences and display location information for each one without needing to look deep inside the hidden RemoteReferenceTracker instance to find it. ** giving up on resource-consumtion defenses Ticket #127 contains more detail, but beginning with this release, Foolscap will be slowly removing the code that attempted to prevent memory-exhaustion attacks. Doing this in a single process is just too hard, and the limits that were enforced provided more problems than protection. To this end, an internal 200-byte limit on FURL length (applied in Gifts) has been removed. Later releases will remove more code, hopefully simplifying the deserization path. ** other bugfixes Previous releases would throw an immediate exception when Tub.getReference() or Tub.connectTo() was called with an unreachable FURL (one with a corrupted or empty set of location hints). In code which walks a list of FURLs and tries to initiate connections to all of them, this synchronous exception would bypass all FURLs beyond the troublesome one. This has been improved: Tub.getReference() now always returns a Deferred, even if the connection is doomed to fail because of a bad FURL. These problems are now indicated by a Deferred that errbacks instead of a synchronous exception. * Release 0.4.1 (22 May 2009) ** Compatibility Same as 0.4.0 ** Bug fixes The new RemoteException class was not stringifiable under python2.4 (i.e. str(RemoteException(f)) would raise an AttributeError), causing problems especially when callRemote errbacks attempted to record the received Failure with log.msg(failure=f). This has been fixed. * Release 0.4.0 (19 May 2009) ** Compatibility The wire protocol remains the same as before, unchanged since 0.2.6 . The main API entry point has moved to "foolscap.api": e.g. you should do "from foolscap.api import Tub" instead of "from foolscap import Tub". Importing symbols directly from the "foolscap" module is now deprecated. (this makes it easier to reorganize the internal structure of Foolscap without causing circular dependencies). (#122) A near-future release (probably 0.4.1) will add proper DeprecationWarnings-raising wrappers to all classes and functions in foolscap/__init__.py . The next major release (probably 0.5.0) will remove these symbols from foolscap/__init__.py altogether. Logging functions are still meant to be imported from foolscap.logging.* . ** expose-remote-exception-types (#105) Remote exception reporting is changing. Please see the new document docs/failures.xhtml for full details. This release adds a new option: tub.setOption("expose-remote-exception-types", BOOL) The default is True, which provides the same behavior as previous releases: remote exceptions are presented to look as much as possible like local exceptions. If you set it to False, then all remote exceptions will be collapsed into a single "foolscap.api.RemoteException" type, with an attribute named .failure that can be used to get more details about the remote exception. This means that callRemote will either fire its Deferred with a regular value, or errback with one of three exception types from foolscap.api: DeadReferenceError, Violation, or RemoteException. (When the option is True, it could errback with any exception type, limited only by what the remote side chose to raise) A future version of Foolscap may change the default value of this option. We're not sure yet: we need more experience to see which mode is safer and easier to code with. If the default is changed, the deprecation sequence will probably be: 0.5.0: require expose-remote-exception-types to be set 0.6.0: change the default to False, stop requiring the option to be set 0.7.0: remove the option ** major bugs fixed: Shared references now work after a Violation (#104) The tubid returned by rref.getSturdyRef() is now reliable (#84) Foolscap should work with python-2.6: Decimal usage fixed, sha/md5 deprecation warnings fixed, import of 'sets' still causes a warning. (#118, #121) Foolscap finally uses new-style classes everywhere (#96) bin/flogtool might work better on windows now (#108) logfiles now store library versions and process IDs (#80, #97) The "flogtool web-viewer" tool listens at a URL of "/" instead of "/welcome", making it slightly easier to use (#120) You can now setOption() on both log-gatherer-furl and log-gatherer-furlfile on the same Tub. Previously this caused an error. (#114) * Release 0.3.2 (14 Oct 2008) ** Compatibility: same as 0.2.6 Incident classifier functions (introduced in 0.3.0) have been changed: if you have written custom functions for an Incident Gatherer, you will need to modify them upon upgrading to this release. ** Logging Changes The log.msg counter now uses a regular Python integer/bigint. The counter in 0.3.1 used itertools.count(), which, despite its documentation, stores the counter in a C signed int, and thus throws an exception when the message number exceeds 2**31-1 . This exception would pretty much kill any program which ran long enough to emit this many messages, a situation which was observed in a busy production server with an uptime of about three or four weeks. The 0.3.2 counter will be promoted to a bigint when necessary, removing this limitation. (ticket #99) The Incident-Gatherer now imports classification functions from files named 'classify_*.py' in the gatherer's directory. This effectively adds "classifier plugins". The signature of the functions has changed since the 0.3.0 release, making them easier to use. If you have written custom functions (and edited the gatherer.tac file to activate them, using gs.add_classifier()), you will need to modify the functions to take a single 'trigger' argument. These same 'classify_*.py' plugins are used by a new "flogtool classify-incident" subcommand, which can be pointed at an incident file, and performs the same kind of classification as the Incident Gatherer. (#102). The logfiles produced by the "flogtool tail" command and the internal incident-reporter now include the PID of the reporting process. This can be seen by passing the --verbose option to "flogtool dump", and will be made more visible in later releases. (#80). The "flogtool web-viewer" tool now marks Incident triggers (#79), and features a "Reload Logfile" button to re-read the logfile on disk (#103). This is most useful when running unit tests, in conjunction with the FLOGFILE= environment variable. ** Other Changes When running unit tests, if the #62 bug is encountered (pyopenssl >= 0.7 and twisted <= 8.1.0 and selectreactor), the test process will emit a warning and pause for ten seconds to give the operator a chance to halt the test and re-run it with --reactor=poll. This may help to reduce the confusion of a hanging+failing test run. The xfer-client.py example tool (in doc/listings/) has been made more useable, by calling os.path.expanduser() on its input files, and by doing sys.exit(1) on failure (instead of hanging), so that external programs can react appropriately. * Release 0.3.1 (03 Sep 2008) ** Compatibility: same as 0.2.6 ** callRemote API changes: DeadReferenceError All partitioning exceptions are now mapped to DeadReferenceError. Previously there were three separate exceptions that might indicate a network partition: DeadReferenceError, ConnectionLost, and ConnectionDone. (a network partition is when one party cannot reach the other party, due to a variety of reasons: temporary network failure, the remote program being shut down, the remote host being taken offline, etc). This means that, if you want to send a message and don't care whether that message makes it to the recipient or not (but you *do* still care if the recipient raises an exception during processing of that message), you can set up the Deferred chain like this: d = rref.callRemote("message", args) d.addCallback(self.handle_response) d.addErrback(lambda f: f.trap(foolscap.DeadReferenceError)) d.addErrback(log.err) The first d.addErrback will use f.trap to catch DeadReferenceError, but will pass other exceptions through to the log.err() errback. This will cause DeadReferenceError to be ignored, but other errors to be logged. DeadReferenceError will be signalled in any of the following situations: 1: the TCP connection was lost before callRemote was invoked 2: the connection was lost after the request was sent, but before the response was received 3: when the active connection is dropped because a duplicate connection was established. This can occur when two programs are simultaneously connecting to each other. ** logging improvements *** bridge foolscap logs into twistd.log By calling foolscap.logging.log.bridgeLogsToTwisted(), or by setting the $FLOGTOTWISTED environment variable (to anything), a subset of Foolscap log events will be copied into the Twisted logging system. The default filter will not copy events below the log.OPERATIONAL level, nor will it copy internal foolscap events (i.e. those with a facility name that starts with "foolscap"). This mechanism is careful to avoid loops, so it is safe to use both bridgeLogsToTwisted() and bridgeTwistedLogs() at the same time. The events that are copied into the Twisted logging system will typically show up in the twistd.log file (for applications that are run under twistd). An alternate filter function can be passed to bridgeLogsToTwisted(). This feature provides a human-readable on-disk record of significant events, using a traditional one-line-per-event all-text sequential logging structure. It does not record parent/child relationships, structured event data, or causality information. *** Incident Gatherer improvements If an Incident occurs while a previous Incident is still being recorded (i.e. during the "trailing log period"), the two will be folded together. Specifically, the second event will not trigger a new Incident, but will be recorded in the first Incident as a normal log event. This serves to address some performance problems we've seen when incident triggers occur in clusters, which used to cause dozens of simultaneous Incident Recorders to swing into action. The Incident Gatherer has been changed to only fetch one Incident at a time (per publishing application), to avoid overloading the app with a large outbound TCP queue. The Incident Gatherer has also been changed to scan the classified/* output files and reclassify any stored incidents it has that are not mentioned in one of these files. This means that you can update the classification functions (to add a function for some previously unknown type of incident), delete the classified/unknown file, then restart the incident gatherer, and it will only reclassify the previously-unknown incidents. This makes it much easier to iteratively develop classification functions. *** Application Version data The table of application versions, previously displayed only by the 'flogtool tail' command, is now recorded in the header of both Incidents and the 'flogtool tail --save-to' output file. The API to add application versions has changed: now programs should call foolscap.logging.app_versions.add_version(name, verstr). * Release 0.3.0 (04 Aug 2008) ** Compatibility: same as 0.2.6 The wire-level protocol remains the same as other recent releases. The new incident-gatherer will only work with applications that use Foolscap 0.3.0 or later. ** logging improvements The "incident gatherer" has finally been implemented. This is a service, like the log-gatherer, which subscribes to an application's logport and collects incident reports: each is a dump of accumulated log messages, triggered by some special event (such as those above a certain severity threshold). The "flogtool create-incident-gatherer" command is used to create one, and twistd is used to start it. Please see doc/logging.xhtml for more details. The incident publishing API was changed to support the incident-gatherer. The incident-gatherer will only work with logports using foolscap 0.3.0 or newer. The log-publishing API was changed slightly, to encourage the use of subscription.unsubscribe() rather than publisher.unsubscribe(subscription). The old API remains in place for backwards compatibility with log-gatherers running foolscap 0.2.9 or earlier. The Tub.setOption("log-gatherer-furlfile") can accept a file with multiple FURLs, one per line, instead of just a single FURL. This makes the application contact multiple log gatherers, offering its logport to each independently, e.g. to connect to both a log-gatherer and an incident-gatherer. ** API Additions RemoteReferences now have a getRemoteTubID() method, which returns a string (base32-encoded) representing the secure Tub ID of the remote end of the connection. For any given Tub ID, only the possessor of the matching private key should be able to provide a RemoteReference for which getRemoteTubID() will return that value. I'm not yet sure if getRemoteTubID() is a good idea or not (the traditional object-capability model discourages making access-control decisions on the basis of "who", instead these decisions should be controlled by "what": what objects do they have access to). This method is intended for use by application code that needs to use TubIDs as an index into a table of some sort. It is used by Tahoe to securely compute shared cryptographic secrets for each remote server (by hashing the TubID together with some other string). Note that the rref.getSturdyRef() call (which has been present in Foolscap since forever) is *not* secure: the remote application controls all parts of the sturdy ref FURL, including the tubid. A future version of foolscap may remedy this. ** Bug fixes The log-gatherer FURL can now be set before Tub.setLocation (the connection request will be deferred until setLocation is called), and getLogPort/getLogPortFURL cannot be called until after setLocation. These two changes, in combination, resolve a problem (#55) in which the gatherer connection could be made before the logport was ready, causing the log-gatherer to fail to subscribe to receive log events. ** Dependent Libraries Foolscap uses PyOpenSSL for all of its cryptographic routines. A bug (#62) has been found in which the current version of Twisted (8.1.0) and the current version of PyOpenSSL (0.7) interact badly, causing Foolscap's unit tests to fail. This problem will affect application code as well (specifically, Tub.stopService will hang forever). The problem only appears to affect the selectreactor, so the current recommended workaround is to run unit tests (and applications that need to shut down Tubs) with --reactor=poll (or whatever other reactor is appropriate for the platform, perhaps iocp). A less-desireable workaround is to downgrade PyOpenSSL to 0.6, or Twisted to something older. The Twisted maintainers are aware of the problem and intend to fix it in an upcoming Twisted release. * Release 0.2.9 (02 Jul 2008) ** Compatibility: exactly the same as 0.2.6 ** logging bugs fixed The foolscap.logging.log.setLogDir() option would throw an exception if the directory already existed, making it unsuitable for use in an application which is expected to be run multiple times. This has been fixed. ** logging improvements 'flogtool tail' now displays the process ID and version information about the remote process. The tool will tolerate older versions of foolscap which do not offer the get_pid interface. (foolscap ticket #71) The remote logport now uses a size-limited queue for messages going to a gatherer or 'flogtool tail', to prevent the monitored process from using unbounded amounts of memory during overload situations (when it is generating messages faster than the receiver can handle them). This solves a runaway load problem we've seen in Tahoe production systems, in which a busy node sends log messages to a gatherer too quickly for it to absorb, using lots of memory to hold the pending messages, which causes swapping, which causes more load, making the problem worse. We frequently see an otherwise well-behaved process swell to 1.4GB due to this problem, occasionally failing due to VM exhaustion. Of course, a bounded queue means that new log events will be dropped during this overload situation. (#72) ** serialization added for the Decimal type (#50) ** debian packaging targets added for gutsy and hardy The Makefile now has 'make debian-gutsy' and 'make debian-hardy' targets. These do the same thing as 'make debian-feisty'. (#76) * Release 0.2.8 (04 Jun 2008) ** Compatibility: exactly the same as 0.2.6 ** setuptools dependencies updated Foolscap (when built with setuptools) now uses the "extras_require" feature to declare that it needs pyOpenSSL if you want to use the "secure_connections" feature. This makes easy_install easier to use in projects that depend upon Foolscap (and also insist upon using secure connections): they do not need to declare a dependency on pyOpenSSL themselves, instead they declare a dependency on "Foolscap[secure_connections]". See the following documentation for more details: http://peak.telecommunity.com/DevCenter/setuptools#declaring-extras-optional-features-with-their-own-dependencies ** New RemoteReference.getPeer() method The new rref.getPeer() method will return address information about the far end of the connection, allowing you to determine their IP address and port number. This may be useful for debugging or diagnostic purposes. ** Minor bugs fixed Tub.registerReference() with both name= and furlFile= arguments now works even when the furlFile= already exists. Tubs which have been shutdown now give more useful error messages when you (incorrectly) try to use them again. Previously a bug caused them to emit a TypeError. * Release 0.2.7 (13 May 2008) ** Compatibility: exactly the same as 0.2.6 ** flogtool works again The "flogtool" utility was completely non-functional in 0.2.6 . This has been fixed. ** Known Issues *** some debian packages are wrong When using the 'make debian-dapper' target (to build a .deb for a dapper system), the resulting .deb sometimes includes a full copy of Twisted, and is probably unsuitable for installation. This appears to be a result of installation behavior changing due to setuptools being imported (even though it is not explicitly used). No other platforms .deb files seem to be affected this way. Package builders are advised to examine the generated .deb closely before using it. * Release 0.2.6 (06 May 2008) ** Compatibility All releases between 0.1.3 and 0.2.6 (inclusive) are fully wire-compatible. The saved logfiles produced (by e.g. 'flogtool tail --save-to' and the log-gatherer) in 0.2.6 and beyond are not readable by tools (e.g. 'flogtool dump' and 'flogtool filter') from 0.2.5 and earlier. FURLs which contain "extensions" (described below) will not be tolerated by foolscap 0.2.5 or earlier. If, at some point in the future, we add such extensions to published FURLs, then such an application will require foolscap-0.2.6 or later to interpret those FURLs. ** Logging Changes *** "Incident" Logging This release finally implements the "strangeness-triggered logging" espoused in doc/logging.xhtml . By giving the foolscap logging code a directory to work with, the logging system will automatically save a compressed pickled logfile containing the messages that lead up to sufficiently-severe log event. The docs explain how to control what "sufficiently-severe" means. These events are retrievable through the logport, although no tools have been written yet to actually extract them. They are also retrievable by using 'flogtool dump' directly on the incident files. *** 'flogtool as a subcommand The implementation of the 'flogtool' executable has been rearranged to make it possible to add a 'flogtool' subcommand to some other executable. *** 'flogtool filter' now has --above LEVEL and --from TUBID options *** 'flogtool dump' has --rx-time option, also shows failure tracebacks *** gatherer: don't add erroneous UTC-implying "Z" suffix to filename timestamps *** 'flogtool tail': don't add spurious "0" to timestamps ** constraints no longer reject ('reference',) sequences The foolscap/banana serialization protocol responds to sending two separate copies of the same object in the same callRemote message by emitting one serialized object sequence and one 'reference' sequence: this is the standard way by which cycles are broken in the serialized data. Unfortunately, the wire-level constraint checkers in 0.2.5 and earlier would reject reference sequences with a Violation exception: if they were expecting a tuple, they would reject anything else, even a reference sequence that pointed at a tuple. Worse yet, python's normal constant-object folding code can produce shared references where you might not expect. In the following example, the two tuples are identical objects (and result in a 'reference' sequence on the wire), despite the programmer having typed them separately: rref.callRemote("haveTwoTuples", (0,1), (0,1) ) Foolscap-0.2.6 now allows reference sequence in all wire-level constraint checks, to avoid this false-negative Violation. The post-deserialization check will still enforce the constraint properly. It just does it late enough to be able to tell what the reference points to. ** Twisted/pyopenssl compatibility *** compatible with Twisted-8.0.x Some unit test failures under Twisted-8.0.x (the most recent release) were fixed: tests now pass against Twisted-8.0.x, and a buildbot is used to make sure compatibility is maintained in the future. *** incompatible with pyOpenSSL-0.7 An incompatibility has been discovered with the most recent version of PyOpenSSL. pyopenssl 0.6 works correctly, but pyopenssl 0.7 causes modern versions of Twisted (both 2.5.x and 8.0.x) to follow a code path that breaks the Foolscap unit tests. This may or may not cause a problem in actual application use (the symptom is that the non-winning parallel connections are not disconnected properly, and several negotiation timers are left running). Until a fix is developed for either Twisted or PyOpenSSL, the recommended workaround is to downgrade to PyOpenSSL-0.6 . Twisted bug #3218 and Foolscap bug #62 exist to track this problem. ** setup.py is more setuptools-friendly The foolscap version string is located with a regular expression rather than an import, allowing setuptools to install Foolscap as a build-dependency of some other project without having Twisted available first. If setuptools is available, we also declare a dependency upon Twisted (at least 2.4.0), to give more information to the setuptools dependency-auto-installer. ** Unauthenticated FURLs can now contain multiple connection hints Previously they were limited to a single one ** FURLs can now contain extensions, providing forwards-compatibility The parsing of FURLs has been refined to tolerate (and ignore) certain kinds of extensions. The "tubid" section will be able to have additional identifiers (perhaps stronger hashes for the public key, or an encryption-ready EC-DSA public key). In addition, the "connection hints" section will be able to contain alternate protocol specifiers (for TCP over IPv6, or a less connection-oriented UDP transport). By ignoring such extensions, foolscap-0.2.6 will tolerate FURLs produced (with extensions) by some future version of foolscap. This marks the beginning of a "transition period": when such extensions are introduced, 0.2.6 will be the oldest version still capable of interoperating with the extension-using new version. * Release 0.2.5 (25 Mar 2008) ** Compatibility All releases between 0.1.3 and 0.2.5 (inclusive) are fully wire-compatible. The new 'flogtool tail --catch-up' command requires a log publisher running 0.2.5 or later. 'flogtool tail' without the --catch-up option will work with earlier publishers. ** Licensing clarification Foolscap is distributed under the (very liberal) terms of the MIT license, which is the same license that Twisted uses. It's been like this since the beginning, but this is the first release to make this obvious by including a LICENSE file. ** foolscap.logging Changes 'flogtool tail' now has a --catch-up option, which prompts the remote publisher to deliver stored historical events to the subscribe, in proper sequential order. This allows you to connect to a process that has just done something interesting and grab a copy of the log events relevant to that event. 'flogtool tail' also has a --save-to option, which specifies a filename to which all captured events should be saved. This file can be processed further with 'flogtool dump', 'flogtool filter', or 'flogtool web-viewer'. This behaves much like the unix 'tee' utility, except that the saved data is recorded in a lossless binary format (whereas the text emitted to stdout is not particularly machine-readable). 'flogtool tail' and 'flogtool dump' both emit human-readable log messages by default. The --verbose option will emit raw event dictionaries, which contain slightly more information but are harder to read. 'flogtool create-gatherer' will create a log gatherer .tac file in a new working directory. This .tac file can be launched with 'twistd', the standard Twisted daemon-launching program. This is significantly easier to work with than the previous 'flogtool gather' command (which has been removed). The new --rotate option will cause the log-gatherer to switch to a new output file every N seconds. The --bzip option will make it compress the logfiles after rotating them. For example, a log gatherer that rotates and compresses log files once per day could be created and launched with: flogtool create-gatherer --rotate 86400 --bzip ./workdir (cd workdir && twistd -y gatherer.tac) ** New sample programs doc/listings/command-server.py and command-client.py are a pair of programs that let you safely grant access to a specific command. The server is configured with a command line to run, and a directory to run it from. The client gets a FURL: when the client is executed, the server will run its preconfigured command. The client gets to see stdout and stderr (and the exit status), but does not get to influence the command being run in any way. This is much like setting up an ssh server with a restricted command, but somewhat easier to configure. doc/listings/xfer-server.py and xfer-client.py are similar, but provide file transfer services instead of command execution. ** New Features Tub.setLocationAutomatically() will try to determine an externally-visible IP address and feed it to Tub.setLocation(). It does this by preparing to send a packet to a well-known public IP address (one of the root DNS servers) and seeing which network interface would be used. This will tend to find the outbound default route, which of course is only externally-visible if the host is externally-visible. Applications should not depend upon this giving a useful value, and should give the user a way to configure a list of hostname+portnumbers so that manually-configured firewalls, port forwarders, and NAT boxes can be dealt with. * Release 0.2.4 (28 Jan 2008) ** Compatibility All releases between 0.1.3 and 0.2.4 (inclusive) are fully wire-compatible. ** foolscap.logging Changes *** 'flogtool filter' command added This mode is used to take one event-log file and produce another with a subset of the events. There are several options to control the filtering: "--strip-facility=foolscap" would remove all the foolscap-related messages, and "--after=start --before=end" will retain events that occur within the given period. The syntax is still in flux, expect these options to change in the next few releases. The main idea is to take a very large logfile and turn it into a smaller, more manageable one. *** error logging Applications should avoid recording application-specific instances in log events. Doing so will forces the log viewer to access the original source code. The current release of foolscap uses pickle, so such instances will be loadable if the viewer can import the same code, but future versions will probably switch to using Banana, at which point trying to log such instances will cause an error. In this release, foolscap stringifies the type of an exception/Failure passed in through the failure= kwarg, to avoid inducing this import dependency in serialized Failures. It also uses the CopiedFailure code to improve portability of Failure instances, and CopiedFailures have been made pickleable. The preferred way to log a Failure instance is to pass it like so: def _oops(f): log.msg("Oh no, it failed", failure=f, level=log.BAD) d.addErrback(_oops) Finally, a 'log.err()' method was added, which behaves just like Twisted's normal log.err(): it can be used in a Deferred errback, or inside an exception handler. *** 'flogtool web-viewer' Adding a "sort=time" query argument to the all-events viewing page URL will turn off the default nested view, and instead will sort all events strictly by time of generation (note that unsynchronized clocks may confuse the relative ordering of events on different machines). "sort=number" sorts all events by their event number, which is of dubious utility (since these numbers are only scoped to the Tub). "sort=nested" is the default mode. The web-viewer now provides "summary views", which show just the events that occurred at a given severity level. Each event is a hyperlink to the line in the all-events page (using anchor/fragment tags), which may make them more convenient to bookmark or reference externally. * Release 0.2.3 (24 Dec 2007) ** Compatibility All releases between 0.1.3 and 0.2.3 (inclusive) are fully wire-compatible. ** Bug Fixes RemoteReference.getSturdyRef() didn't work (due to bitrot). It has been fixed. ** foolscap.logging Changes This release is mostly about flogging improvements: some bugs and misfeatures were fixed: *** tolerate '%' in log message format strings Dictionary-style kwarg formatting is now done with a twisted-style style format= argument instead of happening implicitly. That means the acceptable ways to call foolscap.logging.log.msg are: log.msg("plain string") log.msg("no args means use 0% interpolation") log.msg("pre-interpolated: %d apples" % apples) log.msg("posargs: %d apples and %d oranges", apples, oranges) log.msg(format="kwargs: %(numapples)d apples", numapples=numapples) The benefit of the latter two forms are that the arguments are recorded separately in the event dictionary, so viewing tools can filter on the structured data, think of something like: [e for e in allEvents if e.get("numapples",0) > 4] *** log facility names are now dot-separated, to match stdlib logging *** log levels are derived from numerical stdlib logging levels *** $FLOGFILE to capture flog events during 'trial' runs One challenge of the flogging system is that, once an application was changed to write events to flogging instead of twisted's log, those events do not show up in the normal places where twisted writes its logfiles. For full applications this will be less of an issue, because application startup will tell flogging where events should go (flogging is intended to supplant twisted logging for these applications). But for events emitted during unit tests, such as those driven by Trial, these events would get lost. To address this problem, the 0.2.3 flogging code looks for the "FLOGFILE" environment variable at import time. This specifies a filename where flog events (a series of pickled event dictionaries) should be written. The file is opened at import time, events are written during the lifetime of the process, then the file is closed at shutdown using a Twisted "system event trigger" (which happens to be enough to work properly under Trial: other environments may not work so well). If the FLOGFILE filename ends in .bz2, the event pickles will be compressed, which is highly recommended because it can result in a 30x space savings (and e.g. the Tahoe unit test run results in 90MB of uncompressed events). All 'flogtool' modes know how to handle a .bz2 compressed flogfile as well as an uncompressed one. The "FLOGTWISTED" environment variable, if set, will cause this same code to bridge twisted log messages into the flogfile. This makes it easier to see the relative ordering of Twisted actions and foolscap/application events. (without this it becomes very hard to correlate the two sources of events). The "FLOGLEVEL" variable specifies a minimum severity level that will be put into the flogfile. This defaults to "1", which puts pretty much everything into the file. The idea is that, for tests, you might as well record everything, and use the filtering tools to control the display and isolate the important events. Real applications will use more sophisticated tradeoffs between disk space and interpretation effort. The recommended way to run Trial on a unit test suite for an application that uses Foolscap is: FLOGFILE=flog.out FLOGTWISTED=1 trial PACKAGENAME Note that the logfile cannot be placed in _trial_temp/, because trial deletes that directory after flogging creates the logfile, so the logfile would get deleted too. Also note that the file created by $FLOGFILE is truncated on each run of the program. * Release 0.2.2 (12 Dec 2007) ** Compatibility All releases between 0.1.3 and 0.2.2 (inclusive) are fully wire-compatible. New (optional) negotiation parameters were added in 0.2.1 (really in 0.2.0). ** Bug Fixes The new duplicate-connection handling code in 0.2.1 was broken. This release probably fixes it. There were other bugs in 0.2.1 which were triggered when a duplicate connection was shut down, causing remote calls to never be retired, which would also prevent the Reconnector from doing its job. These should be fixed now. ** Other Changes Foolscap's connection-negotiation mechanism has been modified to use foolscap logging ("flog") instead of twisted.log . Setting the FLOGFILE= environment variable will cause a Foolscap-using program to write pickled log events to a file of that name. This is particularly useful when you want to record log events during 'trial' unit test run. The normal API for setting this file will be added later. The FLOGTWISTED= environment variable will cause the creation of a twisted.log bridge, to copy all events from the twisted log into the foolscap log. The 'flogtool web-view' mode has been enhanced to color-code events according to their severity, and to format Failure tracebacks in a more-readable way. * Release 0.2.1 (10 Dec 2007) ** Compatibility All releases between 0.1.3 and 0.2.1 (inclusive) are fully wire-compatible. 0.2.1 introduces some new negotiation parameters (to handle duplicate connections better), but these are ignored by older versions, and their lack is tolerated by 0.2.1 . ** New Features *** new logging support Foolscap is slowly acquiring advanced diagnostic event-logging features. See doc/logging.xhtml for the philosophy and design of this logging system. 0.2.1 contains the first few pieces, including a tool named bin/flogtool that can be used to watch events on a running system, or gather events from multiple applications into a single place for later analysis. This support is still preliminary, and many of the controls and interfaces described in that document are not yet implemented. *** better handling of duplicate connections / NAT problems The connection-management code in 0.1.7 and earlier interacted badly with programs that run behind NAT boxes (especially those with aggressive connection timeouts) or on laptops which get unplugged from the network abruptly. Foolscap seeks to avoid duplicate connections, and various situtations could cause the two ends to disagree about the viability of any given connection. The net result (no pun intended) was that a client might have to wait up to 35 minutes (depending upon various timeout values) before being able to reestablish a connection, and the Reconnector's exponential backoff strategy could easily push this into 90 minutes of downtime. 0.2.1 uses a different approach to accomplish duplicate-suppression, and should provide much faster reconnection after netquakes. To benefit from this, both ends must be running foolscap-0.2.1 or newer, however there is an additional setting (not enabled by default) to improve the behavior of pre-0.2.1 clients: tub.setOption("handle-old-duplicate-connections", True). *** new Reconnector methods The Reconnector object (as returned by Tub.connectTo) now has three utility methods that may be useful during debugging. reset() drops the backoff timer down to one second, causing the Reconnector to reconnect quickly: you could use this to avoid an hour-long delay if you've just restarted the server or re-enabled a network connection that was the cause of the earlier connection failures. getDelayUntilNextAttempt() returns the number of seconds remaining until the next connection attempt. And getLastFailure() returns a Failure object explaining why the last connection attempt failed, which may be a useful diagnostic in trying to resolve the connection problems. ** Bug Fixes There were other minor changes: an OS-X unit test failure was resolved, CopiedFailures are serialized in a way that doesn't cause constraint violations, and the figleaf code-coverage tools (used by foolscap developers to measure how well the unit tests exercise the code base) have been improved (including an emacs code-used/unused annotation tool). * Release 0.2.0 (10 Dec 2007) This release had a fatal bug that wasn't caught by the unit tests, and was superseded almost immediately by 0.2.1 . * Release 0.1.7 (24 Sep 2007) ** Compatibility All releases between 0.1.3 and 0.1.7 (inclusive) are fully wire-compatible. ** Bug Fixes *** slow remote_ methods shouldn't delay subsequent messages (#25) In earlier releases, a remote_ method which runs slowly (i.e. one which returns a Deferred and does not fire it right away) would have the unfortunate side-effect of delaying all subsequent calls from the same Broker. Those later calls would not be delivered until the first message had completed processing. If, for some reason, that Deferred were never fired, this Foolscap bug would prevent any other remote_ methods from ever being called. This is not how Foolscap's message-ordering logic is designed to work. Foolscap guarantees in-order *delivery* of messages, but does not require that they be completed/retired in that same order. This has now been fixed. The invocation of remote_* is done in-order: any further sequencing is up to the receiving application. For example, in the following code: sender: rref.callRemote("quick", 1) rref.callRemote("slow", 2) rref.callRemote("quick", 3) receiver: def remote_quick(self, num): print num def remote_slow(self, num): print num d = Deferred() def _done(): print "DONE" d.callback(None) reactor.callLater(5.0, _done) return d The intended order of printed messages is 1,2,3,DONE . This bug caused the ordering to be 1,2,DONE,3 instead. *** default size limits removed from all Constraints (#26) Constraints in Foolscap serve two purposes: DoS attack mitigation, and strong typing on remote interfaces to help developers find problems sooner. To support the former, most container-based Constraints had default size limits. For example, the default StringConstraint enforced a maximum length of 1000 characters, and the ListConstraint had a maxLength of 30 elements. In practice, these limits turned out to be more surprising than helpful. Applications which worked fine in testing would mysteriously break when subjected to data that was larger than expected. Developers who used Constraints for their type-checking properties were surprised to discover that they were getting size limitations as well. In addition, the DoS-mitigation code in foolscap is not yet complete, so the cost/benefit ratio of this feature was dubious. For these reasons, all default size limits have been removed from this release. The 0.1.7 StringConstraint() schema is equivalent to the 0.1.6 StringConstraint(maxLength=None) version. To get the 0.1.6 behavior, use StringConstraint(maxLength=1000). The same is true for ListConstraint, DictConstraint, SetConstraint, and UnicodeConstraint. ** New features *** Tub.registerReference(furlFile=) In the spirit of Tub(certFile=), a new argument was added to registerReference that instructs Foolscap to find and store a randomly-generated name in the given file. This makes it convenient to allow subsequent invocations of the same program to use the same stable (yet unguessable) identifier for long-lived objects. For example, a Foolscap-based server can make its Server object available under the same FURL from one run of the program to the next with the following startup code: s = MyServer() furl = tub.registerReference(s, furlFile=os.path.join(basedir, "server")) If the furlFile= exists before registerReference is called, a FURL will be read from it, and a name extracted to use for the object. If not, the file will be created and filled with a FURL that uses a randomly-generated name. *** Tub.serialize() Work is ongoing to implement E-style cryptographic Sealers/Unsealers in Foolscap (see ticket #20). Part of that work has made it into this release. The new Tub.serialize() and Tub.unserialize() methods provide access to the underlying object-graph-serialization code. Normally this code is used to construct a bytestream that is immediately sent over an SSL connection to a remote host; these methods return a string instead. Eventually, a Sealer will return an encrypted version of this string, and the corresponding Unsealer will take the encrypted string and build a new object graph. The foolscap.serialize() and .unserialize() functions have existed for a while. The new Tub.serialize()/.unserialize() methods are special in that you can serialize Referenceables and RemoteReferences. These are encoded with their FURLs, so that the unserializing Tub can establish a new live reference to their targets. foolscap.serialize() cannot handle referenceables. Note that both Tub.serialize() and foolscap.serialize() are currently "unsafe", in that they will serialize (and unserialize!) instances of arbitrary classes, much like the stdlib pickle module. This is a significant security problem, as this results in arbitrary object constructors being executed during deserialization. In a future release of Foolscap, this mode of operation will *not* be the default, and a special argument will have to be passed to enable such behavior. ** Other Improvements When methods fail, the error messages that get logged have been improved. The new messages contain information about which source+dest TubIDs were involved, and which RemoteInterface and method name was being used. A new internal method named Tub.debug_listBrokers() will provide information on which messages are waiting for delivery, either inbound or outbound. It is intended to help diagnose problems like #25. Any message which remains unresolved for a significant amount of time is likely to indicate a problem. * Release 0.1.6 (02 Sep 2007) ** Compatibility All releases between 0.1.3 and 0.1.6 (inclusive) are fully wire-compatible. ** Bug Fixes Using a schema of ChoiceOf(StringConstraint(2000), None) would fail to accept strings between 1000 and 2000 bytes: it would accept a short string, or None, but not a long string. This has been fixed. ChoiceOf() remains a troublesome constraint: having it is awfully nice, and things like ChoiceOf(str,None) seem to work, but it is unreliable. Using ChoiceOf with non-terminal children is not recommended (the garden-path problem is unlikely to be easy to solve): schemas are not regular expressions. The debian packaging rules have been fixed. The ones in 0.1.5 failed to run because of some renamed documentation files. ** Minor Fixes Several minor documentation errors have been corrected. A new 'make api-docs' target has been added to run epydoc and build HTML versions of the API documentation. When a remote method fails and needs to send a traceback over the wire, and when the traceback is too large, trim out the middle rather than the end, since usually it's the beginning and the end that are the most useful. * Release 0.1.5 (07 Aug 2007) ** Compatibility This release is fully compatible with 0.1.4 and 0.1.3 . ** CopiedFailure improvements When a remote method call fails, the calling side gets back a CopiedFailure instance. These instances now behave slightly more like the (local) Failure objects that they are intended to mirror, in that .type now behaves much like the original class. This should allow trial tests which result in a CopiedFailure to be logged without exploding. In addition, chained failures (where A calls B, and B calls C, and C fails, so C's Failure is eventually returned back to A) should work correctly now. ** Gift improvements Gifts inside return values should properly stall the delivery of the response until the gift is resolved. Gifts in all sorts of containers should work properly now. Gifts which cannot be resolved successfully (either because the hosting Tub cannot be reached, or because the name cannot be found) will now cause a proper error rather than hanging forever. Unresolvable gifts in method arguments will cause the message to not be delivered and an error to be returned to the caller. Unresolvable gifts in method return values will cause the caller to receive an error. ** IRemoteReference() adapter The IRemoteReference() interface now has an adapter from Referenceable which creates a wrapper that enables the use of callRemote() and other IRemoteReference methods on a local object. The situation where this might be useful is when you have a central introducer and a bunch of clients, and the clients are introducing themselves to each other (to create a fully-connected mesh), and the introductions are using live references (i.e. Gifts), then when a specific client learns about itself from the introducer, that client will receive a local object instead of a RemoteReference. Each client will wind up with n-1 RemoteReferences and a single local object. This adapter allows the client to treat all these introductions as equal. A client that wishes to send a message to everyone it's been introduced to (including itself) can use: for i in introductions: IRemoteReference(i).callRemote("hello", args) In the future, if we implement coercing Guards (instead of compliance-asserting Constraints), then IRemoteReference will be useful as a guard on methods that want to insure that they can do callRemote (and notifyOnDisconnect, etc) on their argument. ** Tub.registerNameLookupHandler This method allows a one-argument name-lookup callable to be attached to the Tub. This augments the table maintained by Tub.registerReference, allowing Referenceables to be created on the fly, or persisted/retrieved on disk instead of requiring all of them to be generated and registered at startup. * Release 0.1.4 (14 May 2007) ** Compatibility This release is fully compatible with 0.1.3 . ** getReference/connectTo can be called before Tub.startService() The Tub.startService changes that were suggested in the 0.1.3 release notes have been implemented. Calling getReference() or connectTo() before the Tub has been started is now allowed, however no action will take place until the Tub is running. Don't forget to start the Tub, or you'll be left wondering why your Deferred or callback is never fired. (A log message is emitted when these calls are made before the Tub is started, in the hopes of helping developers find this mistake faster). ** constraint improvements The RIFoo -style constraint now accepts gifts (third-party references). This also means that using RIFoo on the outbound side will accept either a Referenceable that implements the given RemoteInterface or a RemoteReference that points to a Referenceable that implements the given RemoteInterface. There is a situation (sending a RemoteReference back to its owner) that will pass the outbound constraint but be rejected by the inbound constraint on the other end. It remains to be seen how this will be fixed. ** foolscap now deserializes into python2.4-native 'set' and 'frozenset' types Since Foolscap is dependent upon python2.4 or newer anyways, it now unconditionally creates built-in 'set' and 'frozenset' instances when deserializing 'set'/'immutable-set' banana sequences. The pre-python2.4 'sets' module has non-built-in set classes named sets.Set and sets.ImmutableSet, and these are serialized just like the built-in forms. Unfortunately this means that Set and ImmutableSet will not survive a round-trip: they'll be turned into set and frozenset, respectively. Worse yet, 'set' and 'sets.Set' are not entirely compatible. This may cause a problem for older applications that were written to be compatible with both python-2.3 and python-2.4 (by using sets.Set/sets.ImmutableSet), for which the compatibility code is still in place (i.e. they are not using set/frozenset). These applications may experience problems when set objects that traverse the wire via Foolscap are brought into close proximity with set objects that remained local. This is unfortunate, but it's the cleanest way to support modern applications that use the native types exclusively. ** bug fixes Gifts inside containers (lists, tuples, dicts, sets) were broken: the target method was frequently invoked before the gift had properly resolved into a RemoteReference. Constraints involving gifts inside containers were broken too. The constraints may be too loose right now, but I don't think they should cause false negatives. The unused SturdyRef.asLiveRef method was removed, since it didn't work anyways. ** terminology shift: FURL The preferred name for the sort of URL that you get back from registerReference (and hand to getReference or connectTo) has changed from "PB URL" to "FURL" (short for Foolscap URL). They still start with 'pb:', however. Documentation is slowly being changed to use this term. * Release 0.1.3 (02 May 2007) ** Incompatibility Warning The 'keepalive' feature described below adds a new pair of banana tokens, PING and PONG, which introduces a compatibility break between 0.1.2 and 0.1.3 . Older versions would throw an error upon receipt of a PING token, so the version-negotiation mechanism is used to prevent banana-v2 (0.1.2) peers from connecting to banana-v3 (0.1.3+) peers. Our negotiation mechanism would make it possible to detect the older (v2) peer and refrain from using PINGs, but that has not been done for this release. ** Tubs must be running before use Tubs are twisted.application.service.Service instances, and as such have a clear distinction between "running" and "not running" states. Tubs are started by calling startService(), or by attaching them to a running service, or by starting the service that they are already attached to. The design rule in operation here is that Tubs are not allowed to perform network IO until they are running. This rule was not enforced completely in 0.1.2, and calls to getReference()/connectTo() that occurred before the Tub was started would proceed normally (initiating a TCP connection, etc). Starting with 0.1.3, this rule *is* enforced. For now, that means that you must start the Tub before calling either of these methods, or you'll get an exception. In a future release, that may be changed to allow these early calls, and queue or otherwise defer the network IO until the Tub is eventually started. (the biggest issue is how to warn users who forget to start the Tub, since in the face of such a bug the getReference will simply never complete). ** Keepalives Tubs now keep track of how long a connection has been idle, and will send a few bytes (a PING of the other end) if no other traffic has been seen for roughly 4 to 8 minutes. This serves two purposes. The first is to convince an intervening NAT box that the connection is still in use, to prevent it from discarding the connection's table entry, since that would block any further traffic. The second is to accelerate the detection of such blocked connections, specifically to reduce the size of a window of buggy behavior in Foolscap's duplicate-connection detection/suppression code. This problem arises when client A (behind a low-end NAT box) connects to server B, perhaps using connectTo(). The first connection works fine, and is used for a while. Then, for whatever reason, A and B are silent for a long time (perhaps as short as 20 minutes, depending upon the NAT box). During this silence, A's NAT box thinks the connection is no longer in use and drops the address-translation table entry. Now suppose that A suddenly decides to talk to B. If the NAT box creates a new entry (with a new outbound port number), the packets that arrive on B will be rejected, since they do not match any existing TCP connections. A sees these rejected packets, breaks the TCP connection, and the Reconnector initiates a new connection. Meanwhile, B has no idea that anything has gone wrong. When the second connection reaches B, it thinks this is a duplicate connection from A, and that it already has a perfectly functional (albeit quiet) connection for that TubID, so it rejects the connection during the negotiation phase. A sees this rejection and schedules a new attempt, which ends in the same result. This has the potential to prevent hosts behind NAT boxes from ever reconnecting to the other end, at least until the the program at the far end is restarted, or it happens to try to send some traffic of its own. The same problem can occur if a laptop is abruptly shut down, or unplugged from the network, then moved to a different network. Similar problems have been seen with virtual machine instances that were suspended and moved to a different network. The longer-term fix for this is a deep change to the way duplicate connections (and cross-connect race conditions) are handled. The keepalives, however, mean that both sides are continually checking to see that the connection is still usable, enabling TCP to break the connection once the keepalives go unacknowledged for a certain amount of time. The default keepalive timer is 4 minutes, and due to the way it is implemented this means that no more than 8 minutes will pass without some traffic being sent. TCP tends to time out connections after perhaps 15 minutes of unacknowledged traffic, which means that the window of unconnectability is probably reduced from infinity down to about 25 minutes. The keepalive-sending timer defaults to 4 minutes, and can be changed by calling tub.setOption("keepaliveTimeout", seconds). In addition, an explicit disconnect timer can be enabled, which tells Foolscap to drop the connection unless traffic has been seen within some minimum span of time. This timer can be set by calling tub.setOption("disconnectTimeout", seconds). Obviously it should be set to a higher value than the keepaliveTimeout. This will close connections faster than TCP will. Both TCP disconnects and the ones triggered by this disconnectTimeout run the risk of false negatives, of course, in the face of unreliable networks. ** New constraints When a tuple appears in a method constraint specification, it now maps to an actual TupleOf constraint. Previously they mapped to a ChoiceOf constraint. In practice, TupleOf appears to be much more useful, and thus better deserving of the shortcut. For example, a method defined as follows: def get_employee(idnumber=int): return (str, int, int) # (name, room_number, age) can only return a three-element tuple, in which the first element is a string (specifically it conforms to a default StringConstraint), and the second two elements are ints (which conform to a default IntegerConstraint, which means it fits in a 32-bit signed twos-complement value). To specify a constraint that can accept alternatives, use ChoiceOf: def get_record(key=str): """Return the record (a string) if it is present, or None if it is not present.""" return ChoiceOf(str, None) UnicodeConstraint has been added, with minLength=, maxLength=, and regexp= arguments. The previous StringConstraint has been renamed to ByteStringConstraint (for accuracy), and it is defined to *only* accept string objects (not unicode objects). 'StringConstraint' itself remains equivalent to ByteStringConstraint for now, but in the future it may be redefined to be a constraint that accepts both bytestrings and unicode objects. To accomplish the bytestring-or-unicode constraint now, you might try schema.AnyStringConstraint, but it has not been fully tested, and might not work at all. ** Bugfixes Errors during negotiation were sometimes delivered in the wrong format, resulting in a "token prefix is limited to 64 bytes" error message. Several error messages (including that one) have been improved to give developers a better chance of determining where the actual problem lies. RemoteReference.notifyOnDisconnect was buggy when called on a reference that was already broken: it failed to fire the callback. Now it fires the callback soon (using an eventual-send). This should remove a race condition from connectTo+notifyOnDisconnect sequences and allow them to operate reliably. notifyOnDisconnect() is now tolerant of attempts to remove something twice, which should make it easier to use safely. Remote methods which raise string exceptions should no longer cause Foolscap to explode. These sorts of exceptions are deprecated, of course, and you shouldn't use them, but at least they won't break Foolscap. The Reconnector class (accessed by tub.connectTo) was not correctly reconnecting in certain cases (which appeared to be particularly common on windows). This should be fixed now. CopyableSlicer did not work inside containers when streaming was enabled. Thanks to iacovou-AT-gmail.com for spotting this one. ** Bugs not fixed Some bugs were identified and characterized but *not* fixed in this release *** RemoteInterfaces aren't defaulting to fully-qualified classnames When defining a RemoteInterface, you can specify its name with __remote_name__, or you can allow it to use the default name. Unfortunately, the default name is only the *local* name of the class, not the fully-qualified name, which means that if you have an RIFoo in two different .py files, they will wind up with the same name (which will cause an error on import, since all RemoteInterfaces known to a Foolscap-using program must have unique names). It turns out that it is rather difficult to determine the fully-qualified name of the RemoteInterface class early enough to be helpful. The workaround is to always add a __remote_name__ to your RemoteInterface classes. The recommendation is to use a globally-unique string, like a URI that includes your organization's DNS name. *** Constraints aren't constraining inbound tokens well enough Constraints (and the RemoteInterfaces they live inside) serve three purposes. The primary one is as documentation, describing how remotely-accessible objects behave. The second purpose is to enforce that documentation, by inspecting arguments (and return values) before invoking the method, as a form of precondition checking. The third is to mitigate denial-of-service attacks, in which an attacker sends so much data (or carefully crafted data) that the receiving program runs out of memory or stack space. It looks like several constraints are not correctly paying attention to the tokens as they arrive over the wire, such that the third purpose is not being achieved. Hopefully this will be fixed in a later release. Application code can be unaware of this change, since the constraints are still being applied to inbound arguments before they are passed to the method. Continue to use RemoteInterfaces as usual, just be aware that you are not yet protected against certain DoS attacks. ** Use os.urandom instead of falling back to pycrypto Once upon a time, when Foolscap was compatible with python2.3 (which lacks os.urandom), we would try to use PyCrypto's random-number-generation routines when creating unguessable object identifiers (aka "SwissNumbers"). Now that we require python2.4 or later, this fallback has been removed, eliminating the last reference to pycrypto within the Foolscap source tree. * Release 0.1.2 (04 Apr 2007) ** Bugfixes Yesterday's release had a bug in the new SetConstraint which rendered it completely unusable. This has been fixed, along with some new tests. ** More debian packaging Some control scripts were added to make it easier to create debian packages for the Ubuntu 'edgy' and 'feisty' distributions. * Release 0.1.1 (03 Apr 2007) ** Incompatibility Warning Because of the technique used to implement callRemoteOnly() (specifically the commandeering of reqID=0), this release is not compatible with the previous release. The protocol negotiation version numbers have been bumped to avoid confusion, meaning that 0.1.0 Tubs will refuse to connect to 0.1.1 Tubs, and vice versa. Be aware that the errors reported when this occurs may not be ideal, in particular I think the "reconnector" (tub.connectTo) might not log this sort of connection failure in a very useful way. ** changes to Constraints Method specifications inside RemoteInterfaces can now accept or return 'Referenceable' to indicate that they will accept a Referenceable of any sort. Likewise, they can use something like 'RIFoo' to indicate that they want a Referenceable or RemoteReference that implements RIFoo. Note that this restriction does not quite nail down the directionality: in particular there is not yet a way to specify that the method will only accept a Referenceable and not a RemoteReference. I'm waiting to see if such a thing is actually useful before implementing it. As an example: class RIUser(RemoteInterface): def get_age(): return int class RIUserListing(RemoteInterface): def get_user(name=str): """Get the User object for a given name.""" return RIUser In addition, several constraints have been enhanced. StringConstraint and ListConstraint now accept a minLength= argument, and StringConstraint also takes a regular expression to apply to the string it inspects (the regexp can either be passed as a string or as the output of re.compile()). There is a new SetConstraint object, with 'SetOf' as a short alias. Some examples: HexIdConstraint = StringConstraint(minLength=20, maxLength=20, regexp=r'[\dA-Fa-f]+') class RITable(RemoteInterface): def get_users_by_id(id=HexIdConstraint): """Get a set of User objects; all will have the same ID number.""" return SetOf(RIUser, maxLength=200) These constraints should be imported from foolscap.schema . Once the constraint interface is stabilized and documented, these classes will probably be moved into foolscap/__init__.py so that you can just do 'from foolscap import SetOf', etc. *** UnconstrainedMethod To disable schema checking for a specific method, use UnconstrainedMethod in the RemoteInterface definition: from foolscap.remoteinterface import UnconstrainedMethod class RIUse(RemoteInterface): def set_phone_number(area_code=int, number=int): return bool set_arbitrary_data = UnconstrainedMethod The schema-checking code will allow any sorts of arguments through to this remote method, and allow any return value. This is like schema.Any(), but for entire methods instead of just specific values. Obviously, using this defeats te whole purpose of schema checking, but in some circumstances it might be preferable to allow one or two unconstrained methods rather than resorting to leaving the entire class left unconstrained (by not declaring a RemoteInterface at all). *** internal schema implementation changes Constraints underwent a massive internal refactoring in this release, to avoid a number of messy circular imports. The new way to convert a "shorthand" description (like 'str') into an actual constraint object (like StringConstraint) is to adapt it to IConstraint. In addition, all constraints were moved closer to their associated slicer/unslicer definitions. For example, SetConstraint is defined in foolscap.slicers.set, right next to SetSlicer and SetUnslicer. The constraints for basic tokens (like lists and ints) live in foolscap.constraint . ** callRemoteOnly A new "fire and forget" API was added to tell Foolscap that you want to send a message to the remote end, but do not care when or even whether it arrives. These messages are guaranteed to not fire an errback if the connection is already lost (DeadReferenceError) or if the connection is lost before the message is delivered or the response comes back (ConnectionLost). At present, this no-error philosophy is so strong that even schema Violation exceptions are suppressed, and the callRemoteOnly() method always returns None instead of a Deferred. This last part might change in the future. This is most useful for messages that are tightly coupled to the connection itself, such that if the connection is lost, then it won't matter whether the message was received or not. If the only state that the message modifies is both scoped to the connection (i.e. not used anywhere else in the receiving application) and only affects *inbound* data, then callRemoteOnly might be useful. It may involve less error-checking code on the senders side, and it may involve fewer round trips (since no response will be generated when the message is delivered). As a contrived example, a message which informs the far end that all subsequent messages on this connection will sent entirely in uppercase (such that the recipient should apply some sort of filter to them) would be suitable for callRemoteOnly. The sender does not need to know exactly when the message has been received, since Foolscap guarantees that all subsequently sent messages will be delivered *after* the 'SetUpperCase' message. And, the sender does not need to know whether the connection was lost before or after the receipt of the message, since the establishment of a new connection will reset this 'uppercase' flag back to some known initial-contact state. rref.callRemoteOnly("set_uppercase", True) # returns None! This method is intended to parallel the 'deliverOnly' method used in E's CapTP protocol. It is also used (or will be used) in some internal Foolscap messages to reduce unnecessary network traffic. ** new Slicers: builtin set/frozenset Code has been added to allow Foolscap to handle the built-in 'set' and 'frozenset' types that were introduced in python-2.4 . The wire protocol does not distinguish between 'set' and 'sets.Set', nor between 'frozenset' and 'sets.ImmutableSet'. For the sake of compatibility, everything that comes out of the deserializer uses the pre-2.4 'sets' module. Unfortunately that means that a 'set' sent into a Foolscap connection will come back out as a 'sets.Set'. 'set' and 'sets.Set' are not entirely interoperable, and concise things like 'added = new_things - old_things' will not work if the objects are of different types (but note that things like 'added = new_things.difference(old_things)' *do* work). The current workaround is for remote methods to coerce everything to a locally-preferred form before use. Better solutions to this are still being sought. The most promising approach is for Foolscap to unconditionally deserialize to the builtin types on python >= 2.4, but then an application which works fine on 2.3 (by using sets.Set) will fail when moved to 2.4 . ** Tub.stopService now indicates full connection shutdown, helping Trial tests Like all twisted.application.service.MultiService instances, the Tub.stopService() method returns a Deferred that indicates when shutdown has finished. Previously, this Deferred could fire a bit early, when network connections were still trying to deliver the last bits of data. This caused problems with the Trial unit test framework, which insist upon having a clean reactor between tests. Trial test writers who use Foolscap should include the following sequence in their twisted.trial.unittest.TestCase.tearDown() methods: def tearDown(self): from foolscap.eventual import flushEventualQueue d = tub.stopService() d.addCallback(flushEventualQueue) return d This will insure that all network activity is complete, and that all message deliveries thus triggered have been retired. This activity includes any outbound connections that were initiated (but not completed, or finished negotiating), as well as any listening sockets. The only remaining problem I've seen so far is with reactor.resolve(), which is used to translate DNS names into addresses, and has a window during which you can shut down the Tub and it will leave a cleanup timer lying around. The only solution I've found is to avoid using DNS names in URLs. Of course for real applications this does not matter: it only makes a difference in Trial unit tests which are making heavy use of short-lived Tubs and connections. * Release 0.1.0 (15 Mar 2007) ** usability improvements *** Tubs now have a certFile= argument A certFile= argument has been added to the Tub constructor to allow the Tub to manage its own certificates. This argument provides a filename where the Tub should read or write its certificate. If the file exists, the Tub will read the certificate data from there. If not, the Tub will generate a new certificate and write it to the file. The idea is that you can point certFile= at a persistent location on disk, perhaps in the application's configuration or preferences subdirectory, and then not need to distinguish between the first time the Tub has been created and later invocations. This allows the Tub's identity (derived from the certificate) to remain stable from one invocation to the next. The related problem of how to make (unguessable) object names persistent from one program run to the next is still outstanding, but I expect to implement something similar in the future (some sort of file to which object names are written and read later). certFile= is meant to be used somewhat like this: where = os.path.expanduser("~/.myapp.cert") t = Tub(certFile=where) t.registerReference(obj) # ... *** All eventual-sends are retired on each reactor tick, not just one. Applications which make extensive use of the eventual-send operations (in foolscap.eventual) will probably run more smoothly now. In previous releases, the _SimpleCallQueue class would only execute a single eventual-send call per tick, then take care of all pending IO (and any pending timers) before servicing the next eventual-send. This could probably lead to starvation, as those eventual-sends might generate more work (and cause more network IO), which could cause the event queue to grow without bound. The new approach finishes as much eventual-send work as possible before accepting any IO. Any new eventual-sends which are queued during the current tick will be put off until the next tick, but everything which was queued before the current tick will be retired in the current tick. ** bug fixes *** Tub certificates can now be used the moment they are created In previous releases, Tubs were only willing to accept SSL certificates that created before the moment of checking. If two systems A and B had unsynchronized clocks, and a Foolscap-using application on A was run for the first time to connect to B (thus creating a new SSL certificate), system B might reject the certificate because it looks like it comes from the future. This problem is endemic in systems which attempt to use the passage of time as a form of revocation. For now at least, to resolve the practical problem of certificates generated on demand and used by systems with unsynchronized clocks, Foolscap does not use certificate lifetimes, and will ignore timestamps on the certificates it examines. * Release 0.0.7 (16 Jan 2007) ** bug fixes *** Tubs can now connect to themselves In previous releases, Tubs were unable to connect to themselves: the following code would fail (the negotiation would never complete, so the connection attempt would eventually time out after about 30 seconds): url = mytub.registerReference(target) d = mytub.getReference(url) In release 0.0.7, this has been fixed by catching this case and making it use a special loopback transport (which serializes all messages but does not send them over a wire). There may be still be problems with this code, in particular connection shutdown is not completely tested and producer/consumer code is completely untested. *** Tubs can now getReference() the same URL multiple times A bug was present in the RemoteReference-unslicing code which caused the following code to fail: d = mytub.getReference(url) d.addCallback(lambda ref: mytub.getReference(url)) In particular, the second call to getReference() would return None rather than the RemoteReference it was supposed to. This bug has been fixed. If the previous RemoteReference is still alive, it will be returned by the subsequent getReference() call. If it has been garbage-collected, a new one will be created. *** minor fixes Negotiation errors (such as having incompatible versions of Foolscap on either end of the wire) may be reported more usefully. In certain circumstances, disconnecting the Tub service from a parent service might have caused an exception before. It might behave better now. * Release 0.0.6 (18 Dec 2006) ** INCOMPATIBLE PROTOCOL CHANGES Version 0.0.6 will not interoperate with versions 0.0.5 or earlier, because of changes to the negotiation process and the method-calling portion of the main wire protocol. (you were warned :-). There are still more incompatible changes to come in future versions as the feature set and protocol stabilizes. Make sure you can upgrade both ends of the wire until a protocol freeze has been declared. *** Negotiation versions now specify a range, instead of a single number The two ends of a connection will agree to use the highest mutually-supported version. This approach should make it much easier to maintain backwards compatibility in the future. *** Negotiation now includes an initial VOCAB table One of the outputs of connection negotiation is the initial table of VOCAB tokens to use for abbreviating commonly-used strings into short tokens (usually just 2 bytes). Both ends have the ability to modify this table at any time, but by setting the initial table during negotiation we same some protocol traffic. VOCAB-izing common strings (like 'list' and 'dict') have the potential to compress wire traffic by maybe 50%. *** remote methods now accept both positional and keyword arguments Previously you had to use a RemoteInterface specification to be able to pass positional arguments into callRemote(). (the RemoteInterface schema was used to convert the positional arguments into keyword arguments before sending them over the wire). In 0.0.6 you can pass both posargs and kwargs over the wire, and the remote end will pass them directly to the target method. When schemas are in effect, the arguments you send will be mapped to the method's named parameters in the same left-to-right way that python does it. This should make it easier to port oldpb code to use Foolscap, since you don't have to rewrite everything to use kwargs exclusively. ** Schemas now allow =None and =RIFoo You can use 'None' in a method schema to indicate that the argument or return value must be None. This is useful for methods that always return None. You can also require that the argument be a RemoteReference that provides a particular RemoteInterface. For example: class RIUser(RemoteInterface): def get_age(): return int def delete(): return None class RIUserDatabase(RemoteInterface): def get_user(username=str): return RIUser Note that these remote interface specifications are parsed at import time, so any names they refer to must be defined before they get used (hence placing RIUserDatabase before RIUser would fail). Hopefully we'll figure out a way to fix this in the future. ** Violations are now annotated better, might keep more stack-trace information ** Copyable improvements The Copyable documentation has been split out to docs/copyable.xhtml and somewhat expanded. The new preferred Copyable usage is to have a class-level attribute named "typeToCopy" which holds the unique string. This must match the class-level "copytype" attribute of the corresponding RemoteCopy class. Copyable subclasses (or ICopyable adapters) may still implement getTypeToCopy(), but the default just returns self.typeToCopy . Most significantly, we no longer automatically use the fully-qualified classname: instead we *require* that the class definition include "typeToCopy". Feel free to use any stable and globally-unique string here, like a URI in a namespace that you control, or the fully-qualified package/module/classname of the Copyable subclass. The RemoteCopy subclass must set the 'copytype' attribute, as it is used for auto-registration. These can set copytype=None to inhibit auto-registration. * Release 0.0.5 (04 Nov 2006) ** add Tub.setOption, add logRemoteFailures and logLocalFailures These options control whether we log exceptions (to the standard twisted log) that occur on other systems in response to messages that we've sent, and that occur on our system in response to messages that we've received (respectively). These may be useful while developing a distributed application. All such log messages have each line of the stack trace prefixed by REMOTE: or LOCAL: to make it clear where the exception is happening. ** add sarge packaging, improve dependencies for sid and dapper .debs ** fix typo that prevented Reconnector from actually reconnecting * Release 0.0.4 (26 Oct 2006) ** API Changes *** notifyOnDisconnect() takes args/kwargs RemoteReference.notifyOnDisconnect(), which registers a callback to be fired when the connection to this RemoteReference is lost, now accepts args and kwargs to be passed to the callback function. Without this, application code needed to use inner functions or bound methods to close over any additional state you wanted to get into the disconnect handler. notifyOnDisconnect() returns a "marker", an opaque values that should be passed into the corresponding dontNotifyOnDisconnect() function to deregister the callback. (previously dontNotifyOnDisconnect just took the same argument as notifyOnDisconnect). For example: class Foo: def _disconnect(self, who, reason): print "%s left us, because of %s" % (who, reason) def connect(self, url, why): d = self.tub.getReference(url) def _connected(rref): self.rref = rref m = rref.notifyOnDisconnect(self._disconnect, who, reason=why) self.marker = m d.addCallback(_connected) def stop_caring(self): self.rref.dontNotifyOnDisconnect(self.marker) *** Reconnector / Tub.connectTo() There is a new connection API for applications that want to connect to a target and to reconnect to it if/when that connection is lost. This is like ReconnectingClientFactory, but at a higher layer. You give it a URL to connect to, and a callback (plus args/kwargs) that should be called each time a connection is established. Your callback should use notifyOnDisconnect() to find out when it is disconnected. Reconnection attempts use exponential backoff to limit the retry rate, and you can shut off reconnection attempts when you no longer want to maintain a connection. Use it something like this: class Foo: def __init__(self, tub, url): self.tub = tub self.reconnector = tub.connectTo(url, self._connected, "arg") def _connected(self, rref, arg): print "connected" assert arg == "arg" self.rref = rref self.rref.callRemote("hello") self.rref.notifyOnDisconnect(self._disconnected, "blag") def _disconnected(self, blag): print "disconnected" assert blag == "blag" self.rref = None def shutdown(self): self.reconnector.stopConnecting() Code which uses this pattern will see "connected" events strictly interleaved with "disconnected" events (i.e. it will never see two "connected" events in a row, nor two "disconnected" events). The basic idea is that each time your _connected() method is called, it should re-initialize all your state by making method calls to the remote side. When the connection is lost, all that state goes away (since you have no way to know what is happening until you reconnect). ** Behavioral Changes *** All Referenceable object are now implicitly "giftable" In 0.0.3, for a Referenceable to be "giftable" (i.e. useable as the payload of an introduction), two conditions had to be satisfied. #1: the object must be published through a Tub with Tub.registerReference(obj). #2: that Tub must have a location set (with Tub.setLocation). Once those conditions were met, if the object was sent over a wire from this Tub to another one, the recipient of the corresponding RemoteReference could pass it on to a third party. Another side effect of calling registerReference() is that the Tub retains a strongref to the object, keeping it alive (with respect to gc) until either the Tub is shut down or the object is explicitly de-registered with unregisterReference(). Starting in 0.0.4, the first condition has been removed. All objects which pass through a setLocation'ed Tub will be usable as gifts. This makes it much more convenient to use third-party references. Note that the Tub will *not* retain a strongref to these objects (merely a weakref), so such objects might disappear before the recipient has had a chance to claim it. The lifecycle of gifts is a subject of much research. The hope is that, for reasonably punctual recipients, the gift will be kept alive until they claim it. The whole gift/introduction mechanism is likely to change in the near future, so this lifetime issue will be revisited in a later release. ** Build Changes The source tree now has some support for making debian-style packages (for both sid and dapper). 'make debian-sid' and 'make debian-dapper' ought to create a .deb package. * Release 0.0.3 (05 Oct 2006) ** API Changes The primary entry point for Foolscap is now the "Tub": import foolscap t = foolscap.Tub() d = t.getReference(pburl) d.addCallback(self.gotReference) ... The old "PBService" name is gone, use "Tub" instead. There are now separate classes for "Tub" and "UnauthenticatedTub", rather than using an "encrypted=" argument. Tubs always use encryption if available: the difference between the two classes is whether this Tub should use a public key for its identity or not. Note that you always need encryption to connect to an authenticated Tub. So install pyopenssl, really. ** eventual send operators Foolscap now provides 'eventually' and 'fireEventually', to implement the "eventual send" operator advocated by Mark Miller's "Concurrency Among Strangers" paper (http://www.erights.org/talks/promises/index.html). eventually(cb, *args, **kwargs) runs the given call in a later reactor turn. fireEventually(value=None) returns a Deferred that will be fired (with 'value') in a later turn. These behave a lot like reactor.callLater(0,..), except that Twisted doesn't actually promise that a pair of callLater(0)s will be fired in the right order (they usually do under unix, but they frequently don't under windows). Foolscap's eventually() *does* make this guarantee. In addition, there is a flushEventualQueue() that is useful for unit tests, it returns a Deferred that will only fire when the entire queue is empty. As long as your code only uses eventually() (and not callLater(0)), putting the following in your trial test cases should keep everything nice and clean: def tearDown(self): return foolscap.flushEventualQueue() ** Promises An initial implementation of Promises is in foolscap.promise for experimentation. Only "Near" Promises are implemented to far (promises which resolve to a local object). Eventually Foolscap will offer "Far" Promises as well, and you will be able to invoke remote method calls through Promises as well as RemoteReferences. See foolscap/test/test_promise.py for some hints. ** Bug Fixes Messages containing "Gifts" (third-party references) are now delivered in the correct order. In previous versions, the presence of these references could delay delivery of the containing message, causing methods to be executed out of order. The VOCAB-manipulating code used to have nasty race conditions, which should be all fixed now. This would be more important if we actually used the VOCAB-manipulating code yet, but we don't. Lots of internal reorganization (put all slicers in a subpackage), not really user-visible. Updated to work with recent Twisted HEAD, specifically changes to sslverify. This release of Foolscap ought to work with the upcoming Twisted-2.5 . ** Incompatible protocol changes There are now separate add-vocab and set-vocab sequences, which add a single new VOCAB token and replace the entire table, respectively. These replace the previous 'vocab' sequence which behaved like set-vocab does now. This would be an incompatible protocol change, except that previous versions never sent the vocab sequence anyways. This version doesn't send either vocab-changing sequence either, but when we finally do start using it, it'll be ready. * Release 0.0.2 (14 Sep 2006) Renamed to "Foolscap", extracted from underneat the Twisted packaged, consolidated API to allow a simple 'import foolscap'. No new features or bug fixes relative to pb2-0.0.1 . * Release 0.0.1 (29 Apr 2006) First release! All basic features are in place. The wire protocol will almost certainly change at some point, so compatibility with future versions is not guaranteed. foolscap-0.13.1/PKG-INFO0000644000076500000240000000204213204747603015124 0ustar warnerstaff00000000000000Metadata-Version: 1.1 Name: foolscap Version: 0.13.1 Summary: Foolscap contains an RPC protocol for Twisted. Home-page: http://foolscap.lothar.com/trac Author: Brian Warner Author-email: warner-foolscap@lothar.com License: MIT Description-Content-Type: UNKNOWN Description: Foolscap (aka newpb) is a new version of Twisted's native RPC protocol, known as 'Perspective Broker'. This allows an object in one process to be used by code in a distant process. This module provides data marshaling, a remote object reference system, and a capability-based security model. Platform: any Classifier: Development Status :: 3 - Alpha Classifier: Operating System :: OS Independent Classifier: License :: OSI Approved :: MIT License Classifier: Programming Language :: Python Classifier: Topic :: Internet Classifier: Topic :: Software Development :: Libraries :: Python Modules Classifier: Topic :: System :: Distributed Computing Classifier: Topic :: System :: Networking Classifier: Topic :: Software Development :: Object Brokering foolscap-0.13.1/README0000644000076500000240000000662613204160675014721 0ustar warnerstaff00000000000000 Foolscap Foolscap is an RPC/RMI (Remote Procedure Call / Remote Method Invocation) protocol for use with Twisted, derived/inspired by Twisted's built-in "Perspective Broker" package. If you have control of both ends of the wire, and are thus not constrained to use some other protocol like HTTP/XMLRPC/CORBA/etc, you might consider using Foolscap. Fundamentally, Foolscap allows you to make a python object in one process available to code in other processes, which means you can invoke its methods remotely. This includes a data serialization layer to convey the object graphs for the arguments and the eventual response, and an object reference system to keep track of which objects you are connecting to. It uses a capability-based security model, such that once you create a non-public object, it is only accessible to clients to whom you've given the (unguessable) FURL. You can of course publish world-visible objects that have well-known FURLs. Full documentation and examples are in the doc/ directory. DEPENDENCIES: * Python 2.7 * Twisted 16.0.0 or later * PyOpenSSL (tested against 16.0.0) INSTALLATION: To install foolscap into your system's normal python library directory, just run the following (you will probably have to do this as root): pip install . You can also just add the foolscap source tree to your PYTHONPATH, since there are no compile steps or .so/.dll files involved. COMPATIBILITY: Foolscap's wire protocol is unlikely to change in the near future. Foolscap has a built-in version-negotiation mechanism that allows the two processes to determine how to best communicate with each other. The two ends will agree upon the highest mutually-supported version for all their traffic. If they do not have any versions in common, the connection will fail with a NegotiationError. Please check the NEWS file for announcements of compatibility-breaking changes in any given release. Foolscap-0.9.1 was compatible with (and tested against) Python-2.6 . Since that release, Twisted-15.5.0 dropped py2.6 support (throwing an error at import time), so Foolscap is no longer tested against py2.6, nor against any version of Twisted earlier than 16.0.0. HISTORY: Foolscap is a rewrite of the Perspective Broker protocol provided by Twisted (in the twisted.pb package), with the goal of improving serialization flexibility and connection security. It also adds features to assist application development, such as distributed/incident-triggered logging, Service management, persistent object naming, and debugging tools. For a brief while, it was intended to replace Perspective Broker, so it had a name of "newpb" or "pb2". However we no longer expect Foolscap to ever be added to the Twisted source tree. A "foolscap" is a size of paper, probably measuring 17 by 13.5 inches. A twisted foolscap of paper makes a good fool's cap. Also, "cap" implies capabilities, and Foolscap is a protocol to implement a distributed object-capabilities model in python. AUTHOR: Brian Warner is responsible for this thing. Please discuss it on the twisted-python mailing list. The Foolscap home page is a Trac instance at . It contains pointers to the latest release, bug reports, patches, documentation, and other resources. Foolscap is distributed under the same license as Twisted itself, namely the MIT license. Details are in the LICENSE file. foolscap-0.13.1/README.packagers0000644000076500000240000000122512766553111016651 0ustar warnerstaff00000000000000Notes to Packagers: Foolscap depends on PyOpenSSL. All packaged versions should include a dependency on whatever package your distribution uses to provide PyOpenSSL ("python-openssl" on Debian). That way, other programs can depend upon "python-foolscap" and get full support for secure connections. To silence a warning that Twisted emits otherwise, the foolscap package should also depend upon some packaging of the "service_identity" module. Python programs (using distutils/setuptools/distribute metadata) that want to declare their dependency on Foolscap can do so with install_requires=["foolscap"], which will ensure that PyOpenSSL also gets installed. foolscap-0.13.1/setup.cfg0000644000076500000240000000031113204747603015645 0ustar warnerstaff00000000000000[versioneer] vcs = git versionfile_source = src/foolscap/_version.py versionfile_build = foolscap/_version.py tag_prefix = foolscap- parentdir_prefix = foolscap- [egg_info] tag_build = tag_date = 0 foolscap-0.13.1/setup.py0000755000076500000240000000461613204160675015553 0ustar warnerstaff00000000000000#!/usr/bin/env python from setuptools import setup, Command import versioneer commands = versioneer.get_cmdclass().copy() class Trial(Command): description = "run trial" user_options = [] def initialize_options(self): pass def finalize_options(self): pass def run(self): import sys from twisted.scripts import trial sys.argv = ["trial", "--rterrors", "foolscap.test"] trial.run() # does not return commands["trial"] = Trial commands["test"] = Trial setup_args = { "name": "foolscap", "version": versioneer.get_version(), "description": "Foolscap contains an RPC protocol for Twisted.", "author": "Brian Warner", "author_email": "warner-foolscap@lothar.com", "url": "http://foolscap.lothar.com/trac", "license": "MIT", "long_description": """\ Foolscap (aka newpb) is a new version of Twisted's native RPC protocol, known as 'Perspective Broker'. This allows an object in one process to be used by code in a distant process. This module provides data marshaling, a remote object reference system, and a capability-based security model. """, "classifiers": [ "Development Status :: 3 - Alpha", "Operating System :: OS Independent", "License :: OSI Approved :: MIT License", "Programming Language :: Python", "Topic :: Internet", "Topic :: Software Development :: Libraries :: Python Modules", "Topic :: System :: Distributed Computing", "Topic :: System :: Networking", "Topic :: Software Development :: Object Brokering", ], "platforms": ["any"], "package_dir": {"": "src"}, "packages": ["foolscap", "foolscap.slicers", "foolscap.logging", "foolscap.connections", "foolscap.appserver", "foolscap.test"], "entry_points": {"console_scripts": [ "flogtool = foolscap.logging.cli:run_flogtool", "flappserver = foolscap.appserver.cli:run_flappserver", "flappclient = foolscap.appserver.client:run_flappclient", ] }, "cmdclass": commands, "install_requires": ["twisted[tls] >= 16.0.0", "pyOpenSSL"], "extras_require": { "dev": ["mock", "txsocksx", "txtorcon >= 0.16.1", "txi2p >= 0.3.2"], "socks": ["txsocksx"], "tor": ["txtorcon >= 0.16.1"], "i2p": ["txi2p >= 0.3.2"], }, } if __name__ == "__main__": setup(**setup_args) foolscap-0.13.1/src/0000755000076500000240000000000013204747603014620 5ustar warnerstaff00000000000000foolscap-0.13.1/src/foolscap/0000755000076500000240000000000013204747603016426 5ustar warnerstaff00000000000000foolscap-0.13.1/src/foolscap/__init__.py0000644000076500000240000000016113136243157020534 0ustar warnerstaff00000000000000"""Foolscap""" from ._version import get_versions __version__ = str(get_versions()['version']) del get_versions foolscap-0.13.1/src/foolscap/_version.py0000644000076500000240000000076213204747603020631 0ustar warnerstaff00000000000000 # This file was generated by 'versioneer.py' (0.18) from # revision-control system data, or from the parent directory name of an # unpacked source archive. Distribution tarballs contain a pre-generated copy # of this file. import json version_json = ''' { "date": "2017-11-20T23:01:21-0800", "dirty": false, "error": null, "full-revisionid": "a349884b483c1880269e95bd37155b1382971031", "version": "0.13.1" } ''' # END VERSION_JSON def get_versions(): return json.loads(version_json) foolscap-0.13.1/src/foolscap/api.py0000644000076500000240000000334412766553111017556 0ustar warnerstaff00000000000000 # application code should import all names from here instead of from # __init__.py . Use code like this: # # from foolscap.api import Tub # # This will make it easier to rearrange Foolscap's internals in the future. # Anything you might import from outside foolscap.api is subject to movement # in new releases. from foolscap._version import get_versions __version__ = str(get_versions()['version']) del get_versions # here is the primary entry point from foolscap.pb import Tub # names we import so that others can reach them as foolscap.api.foo from foolscap.remoteinterface import RemoteInterface from foolscap.referenceable import Referenceable, SturdyRef from foolscap.copyable import Copyable, RemoteCopy, registerRemoteCopy from foolscap.copyable import registerCopier, registerRemoteCopyFactory from foolscap.ipb import DeadReferenceError, IConnectionHintHandler from foolscap.tokens import BananaError from foolscap.schema import StringConstraint, IntegerConstraint, \ ListOf, TupleOf, SetOf, DictOf, ChoiceOf, Any from foolscap.storage import serialize, unserialize from foolscap.tokens import Violation, RemoteException from foolscap.eventual import eventually, fireEventually, flushEventualQueue from foolscap.logging import app_versions # hush pyflakes _unused = [ __version__, Tub, RemoteInterface, Referenceable, SturdyRef, Copyable, RemoteCopy, registerRemoteCopy, registerCopier, registerRemoteCopyFactory, DeadReferenceError, IConnectionHintHandler, BananaError, StringConstraint, IntegerConstraint, ListOf, TupleOf, SetOf, DictOf, ChoiceOf, Any, serialize, unserialize, Violation, RemoteException, eventually, fireEventually, flushEventualQueue, app_versions, ] del _unused foolscap-0.13.1/src/foolscap/appserver/0000755000076500000240000000000013204747603020435 5ustar warnerstaff00000000000000foolscap-0.13.1/src/foolscap/appserver/__init__.py0000644000076500000240000000000012766553111022535 0ustar warnerstaff00000000000000foolscap-0.13.1/src/foolscap/appserver/cli.py0000644000076500000240000003702213204160675021560 0ustar warnerstaff00000000000000 import os, sys, shutil, errno, time, signal from StringIO import StringIO from twisted.python import usage from twisted.internet import defer from twisted.scripts import twistd # does "flappserver start" need us to refrain from importing the reactor here? # A: probably, to allow --reactor= to work import foolscap from foolscap.api import Tub, Referenceable from foolscap.pb import generateSwissnumber from foolscap.appserver.services import build_service, BadServiceArguments from foolscap.appserver.server import AppServer, load_service_data, save_service_data # external code can rely upon the stability of add_service() and # list_services(), as well as the following properties: # * services are instantiated with (basedir,tub,type,args) # * their basedir will already exist by the time they're instantiated # * their basedir will be somewhere inside the flappserver's basedir # all other functions and classes are for foolscap's own use, and may change # in future versions def get_umask(): oldmask = os.umask(0) os.umask(oldmask) return oldmask class BaseOptions(usage.Options): opt_h = usage.Options.opt_help def getSynopsis(self): # the default usage.Options.getSynopsis prepends 'flappserver' # Options.synopsis, which looks weird return self.synopsis class CreateOptions(BaseOptions): synopsis = "Usage: flappserver create [options] BASEDIR" optFlags = [ ("quiet", "q", "Be silent upon success"), ] optParameters = [ ("port", "p", "tcp:3116", "TCP port to listen on (strports string)"), ("location", "l", None, "(required) Tub location hints to use in generated FURLs. e.g. 'example.org:3116'"), ("umask", None, None, "(octal) file creation mask to use for the server. If not provided, the current umask (%04o) is copied." % get_umask()), ] def opt_port(self, port): assert not port.startswith("ssl:") self["port"] = port def opt_umask(self, value): self["umask"] = int(value, 8) def parseArgs(self, basedir): self.basedir = basedir def postOptions(self): if self["umask"] is None: self["umask"] = get_umask() if not self["location"]: raise usage.UsageError("--location= is mandatory") FLAPPSERVER_TACFILE = """\ # -*- python -*- # we record the path when 'flappserver create' is run, in case it was run out # of a source tree. This is somewhat fragile, of course. stashed_path = [ %(path)s] import sys needed = [p for p in stashed_path if p not in sys.path] sys.path = needed + sys.path #print 'NEEDED', needed from foolscap.appserver import server from twisted.application import service appserver = server.AppServer() application = service.Application('flappserver') appserver.setServiceParent(application) """ class Create: def run(self, options): basedir = options.basedir stdout = options.stdout stderr = options.stderr if os.path.exists(basedir): print >>stderr, "Refusing to touch pre-existing directory %s" % basedir return 1 assert options["port"] assert options["location"] os.makedirs(basedir) os.makedirs(os.path.join(basedir, "services")) os.chmod(basedir, 0700) # Start the server and let it create the key. The base FURL will be # written to a file so that subsequent 'add' and 'list' can compute # FURLs without needing to run the Tub (which might already be # running). f = open(os.path.join(basedir, "port"), "w") f.write("%s\n" % options["port"]) f.close() # we'll overwrite BASEDIR/port if necessary f = open(os.path.join(basedir, "location"), "w") f.write("%s\n" % options["location"]) f.close() f = open(os.path.join(basedir, "umask"), "w") f.write("%04o\n" % options["umask"]) f.close() save_service_data(basedir, {"version": 1, "services": {}}) a = AppServer(basedir, stdout) tub = a.tub sample_furl = tub.registerReference(Referenceable()) furl_prefix = sample_furl[:sample_furl.rfind("/")+1] f = open(os.path.join(basedir, "furl_prefix"), "w") f.write(furl_prefix + "\n") f.close() f = open(os.path.join(basedir, "flappserver.tac"), "w") stashed_path = "" for p in sys.path: stashed_path += " %r,\n" % p f.write(FLAPPSERVER_TACFILE % { 'path': stashed_path }) f.close() if not options["quiet"]: print >>stdout, "Foolscap Application Server created in %s" % basedir print >>stdout, "TubID %s, listening on port %s" % (tub.getTubID(), options["port"]) print >>stdout, "Now launch the daemon with 'flappserver start %s'" % basedir return defer.succeed(0) class AddOptions(BaseOptions): synopsis = "Usage: flappserver add [--comment C] BASEDIR SERVICE-TYPE SERVICE-ARGS.." optFlags = [ ("quiet", "q", "Be silent upon success"), ] optParameters = [ ("comment", "c", None, "optional comment describing this service"), ] def parseArgs(self, basedir, service_type, *service_args): self.basedir = basedir self.service_type = service_type self.service_args = service_args def getUsage(self, width=None): t = usage.Options.getUsage(self, width) t += "\nUse 'flappserver add BASEDIR SERVICE-TYPE --help' for details." t += "\n\nSERVICE-TYPE can be one of the following:\n" from services import all_services for name in sorted(all_services.keys()): t += " %s\n" % name return t def make_swissnum(): return generateSwissnumber(Tub.NAMEBITS) def find_next_service_basedir(basedir): services_basedir = os.path.join(basedir, "services") nums = [] for dirname in os.listdir(services_basedir): try: nums.append(int(dirname)) # this might also catch old-style swissnum-named directories, if # their name contains entirely digits. The chances of that are # (6/32)^32, or 5.4e-24, so we're probably safe. except ValueError: pass # return value is relative to basedir return os.path.join("services", str(max([0]+nums)+1)) def add_service(basedir, service_type, service_args, comment, swissnum=None): if not swissnum: swissnum = make_swissnum() services_data = load_service_data(basedir) relative_service_basedir = find_next_service_basedir(basedir) service_basedir = os.path.join(basedir, relative_service_basedir) os.makedirs(service_basedir) try: # validate the service args by instantiating one s = build_service(service_basedir, None, service_type, service_args) del s except: shutil.rmtree(service_basedir) raise services_data["services"][swissnum] = { "relative_basedir": relative_service_basedir, "type": service_type, "args": service_args, "comment": comment, } save_service_data(basedir, services_data) furl_prefix = open(os.path.join(basedir, "furl_prefix")).read().strip() furl = furl_prefix + swissnum return furl, service_basedir class Add: def run(self, options): basedir = options.basedir stdout = options.stdout service_type = options.service_type service_args = options.service_args furl, service_basedir = add_service(basedir, service_type, service_args, options["comment"]) if not options["quiet"]: print >>stdout, "Service added in %s" % service_basedir print >>stdout, "FURL is %s" % furl return 0 class ListOptions(BaseOptions): synopsis = "Usage: flappserver list BASEDIR" optFlags = [ ] optParameters = [ ] def parseArgs(self, basedir): self.basedir = basedir class FlappService: pass def list_services(basedir): furl_prefix = open(os.path.join(basedir, "furl_prefix")).read().strip() services_data = load_service_data(basedir)["services"] services = [] for swissnum, data in sorted(services_data.items()): s = FlappService() s.swissnum = swissnum s.service_basedir = os.path.join(basedir, data["relative_basedir"]) s.service_type = data["type"] s.service_args = data["args"] s.comment = data["comment"] # maybe None s.furl = furl_prefix + swissnum services.append(s) return services class List: def run(self, options): basedir = options.basedir stdout = options.stdout for s in list_services(basedir): print >>stdout print >>stdout, "%s:" % s.swissnum print >>stdout, " %s %s" % (s.service_type, " ".join(s.service_args)) if s.comment: print >>stdout, " # %s" % s.comment print >>stdout, " %s" % s.furl print >>stdout, " %s" % s.service_basedir print >>stdout return 0 class StartOptions(BaseOptions): synopsis = "Usage: flappserver start BASEDIR [twistd options]" optFlags = [ ] optParameters = [ ] def parseArgs(self, basedir, *twistd_args): self.basedir = basedir self.twistd_args = twistd_args class Start: def run(self, options): basedir = options.basedir stderr = options.stderr for fn in os.listdir(basedir): if fn.endswith(".tac"): tac = fn break else: print >>stderr, "%s does not look like a node directory (no .tac file)" % basedir return 1 os.chdir(options.basedir) twistd_args = list(options.twistd_args) sys.argv[1:] = ["--no_save", "--python", tac] + twistd_args print >>stderr, "Launching Server..." twistd.run() class StopOptions(BaseOptions): synopsis = "Usage: flappserver stop BASEDIR" optFlags = [ ("quiet", "q", "Be silent when the server is not already running"), ] optParameters = [ ] def parseArgs(self, basedir): self.basedir = basedir def try_to_kill(pid, signum): # return True if we successfully sent the signal # return False if the process was already gone # might raise some other exception try: os.kill(pid, signal.SIGTERM) except OSError, e: if e.errno == errno.ESRCH: # the process disappeared before we got to it return False raise return True def try_to_remove_pidfile(pidfile): try: os.remove(pidfile) except OSError: pass class Stop: def run(self, options): basedir = options.basedir stderr = options.stderr pidfile = os.path.join(basedir, "twistd.pid") if not os.path.exists(pidfile): if not options["quiet"]: print >>stderr, "%s does not look like a running node directory (no twistd.pid)" % basedir # we define rc=2 to mean "nothing is running, but it wasn't me # who stopped it" return 2 pid = int(open(pidfile, "r").read().strip()) # kill it softly (SIGTERM), watch for it to go away, give it 15 # seconds, then kill it hard (SIGKILL) and delete the twistd.pid # file. if not try_to_kill(pid, signal.SIGTERM): try_to_remove_pidfile(pidfile) print >>stderr, "process %d wasn't running, removing twistd.pid to cleanup" % pid return 2 print >>stderr, "SIGKILL sent to process %d, waiting for shutdown" % pid counter = 30 # failsafe in case a timequake occurs timeout = time.time() + 15 while time.time() < timeout and counter > 0: counter += 1 if not try_to_kill(pid, 0): # it's gone try_to_remove_pidfile(pidfile) print >>stderr, "process %d terminated" % pid return 0 time.sleep(0.5) print >>stderr, "Process %d didn't respond to SIGTERM, sending SIGKILL." % pid try_to_kill(pid, signal.SIGKILL) try_to_remove_pidfile(pidfile) return 0 class RestartOptions(BaseOptions): synopsis = "Usage: flappserver restart BASEDIR [twistd options]" def parseArgs(self, basedir, *twistd_args): self.basedir = basedir self.twistd_args = twistd_args class Restart: def run(self, options): options["quiet"] = True rc = Stop().run(options) # ignore rc rc = Start().run(options) return rc class Options(usage.Options): synopsis = "Usage: flappserver (create|add|list|start|stop)" subCommands = [ ("create", None, CreateOptions, "create a new app server"), ("add", None, AddOptions, "add new service to an app server"), ("list", None, ListOptions, "list services in an app server"), ("start", None, StartOptions, "launch an app server"), ("stop", None, StopOptions, "shut down an app server"), ("restart", None, RestartOptions, "(first stop if necessary, then) start a server"), ] def postOptions(self): if not hasattr(self, 'subOptions'): raise usage.UsageError("must specify a command") def opt_version(self): from twisted import copyright print "Foolscap version:", foolscap.__version__ print "Twisted version:", copyright.version sys.exit(0) dispatch_table = { "create": Create, "add": Add, "list": List, "start": Start, "stop": Stop, "restart": Restart, } def dispatch(command, options): if command in dispatch_table: c = dispatch_table[command]() return c.run(options) else: print "unknown command '%s'" % command raise NotImplementedError def run_flappserver(argv=None, run_by_human=True): if argv: command_name,argv = argv[0],argv[1:] else: command_name = sys.argv[0] config = Options() try: config.parseOptions(argv) except usage.error, e: if not run_by_human: raise print "%s: %s" % (command_name, e) print c = getattr(config, 'subOptions', config) print str(c) sys.exit(1) command = config.subCommand so = config.subOptions if run_by_human: so.stdout = sys.stdout so.stderr = sys.stderr else: so.stdout = StringIO() so.stderr = StringIO() try: r = dispatch(command, so) except (usage.UsageError, BadServiceArguments), e: r = 1 print >>so.stderr, "Error:", e from twisted.internet import defer if run_by_human: if isinstance(r, defer.Deferred): # this command needs a reactor from twisted.internet import reactor stash_rc = [] def good(rc): stash_rc.append(rc) reactor.stop() def oops(f): print "Command failed:" print f stash_rc.append(-1) reactor.stop() r.addCallbacks(good, oops) if 0 == len(stash_rc): reactor.run() sys.exit(stash_rc[0]) else: sys.exit(r) else: if isinstance(r, defer.Deferred): def done(rc): return (rc, so.stdout.getvalue(), so.stderr.getvalue()) r.addCallback(done) return r else: return (r, so.stdout.getvalue(), so.stderr.getvalue()) foolscap-0.13.1/src/foolscap/appserver/client.py0000644000076500000240000002461413204746601022271 0ustar warnerstaff00000000000000 import os, sys from StringIO import StringIO from twisted.python import usage from twisted.internet import defer # does "flappserver start" need us to refrain from importing the reactor here? import foolscap from foolscap.api import Tub, Referenceable, fireEventually class BaseOptions(usage.Options): def opt_h(self): return self.opt_help() class UploadFileOptions(BaseOptions): def getSynopsis(self): return "Usage: flappclient [--furl=|--furlfile] upload-file SOURCEFILES.." def parseArgs(self, *sourcefiles): self.sourcefiles = sourcefiles longdesc = """This client sends one or more files to the upload-file service waiting at the given FURL. All files will be placed in the pre-configured target directory, using the basename of each SOURCEFILE argument.""" class Uploader(Referenceable): def run(self, rref, sourcefile, name): self.f = open(os.path.expanduser(sourcefile), "rb") return rref.callRemote("putfile", name, self) def remote_read(self, size): return self.f.read(size) class UploadFile(Referenceable): def run(self, rref, options): d = defer.succeed(None) for sf in options.sourcefiles: name = os.path.basename(sf) d.addCallback(self._upload, rref, sf, name) d.addCallback(self._done, options, name) d.addCallback(lambda _ign: 0) return d def _upload(self, _ignored, rref, sf, name): return Uploader().run(rref, sf, name) def _done(self, _ignored, options, name): print >>options.stdout, "%s: uploaded" % name class RunCommandOptions(BaseOptions): def getSynopsis(self): return "Usage: flappclient [--furl=|--furlfile] run-command" longdesc = """This client triggers a prearranged command to be executed by the run-command service waiting at the given FURL. The executable, its working directory, and all arguments are configured by the server. Unless the server has overridden the defaults, this client will emit the command's stdout and stderr as it runs, and will exit with the same result code as the remote command. If the server desires it, this client will read data from stdin and send everything (plus a close-stdin event) to the server. This client has no control over the command being run or its arguments.""" from twisted.internet.stdio import StandardIO as TwitchyStandardIO class StandardIO(TwitchyStandardIO): def childConnectionLost(self, fd, reason): # the usual StandardIO class doesn't seem to handle half-closed stdio # well, specifically when our stdout is closed, then some data is # written to our stdin. The class responds to stdout's closure by # shutting down everything. I think this is related to # ProcessWriter.doRead returning CONNECTION_LOST instead of # CONNECTION_DONE (which ProcessWriter.connectionLost sends along to # StandardIO.childConnectionLost). There is code in # StandardIO.childConnectionLost to treat CONNECTION_DONE as a # half-close, but not CONNECTION_LOST. # # so this hack is to make it look more like a half-close #print >>sys.stderr, "my StandardIO.childConnectionLost", fd, reason.value from twisted.internet import error, main from twisted.python import failure if reason.check(error.ConnectionLost) and fd == "write": #print >>sys.stderr, " fixing" reason = failure.Failure(main.CONNECTION_DONE) return TwitchyStandardIO.childConnectionLost(self, fd, reason) from twisted.internet.protocol import Protocol #from zope.interface import implements #from twisted.internet.interfaces import IHalfCloseableProtocol class RunCommand(Referenceable, Protocol): #implements(IHalfCloseableProtocol) def run(self, rref, options): self.done = False self.d = defer.Deferred() rref.notifyOnDisconnect(self._done, 3) self.stdin_writer = None self.stdio = options.stdio self.stdout = options.stdout self.stderr = options.stderr d = rref.callRemote("execute", self) d.addCallback(self._started) d.addErrback(self._err) return self.d def dataReceived(self, data): if not isinstance(data, str): raise TypeError("stdin can accept only strings of bytes, not '%s'" % (type(data),)) # this is from stdin. It shouldn't be called until after _started # sets up stdio and self.stdin_writer self.stdin_writer.callRemoteOnly("feed_stdin", data) def connectionLost(self, reason): # likewise, this won't be called unless _started wanted stdin self.stdin_writer.callRemoteOnly("close_stdin") def _started(self, stdin_writer): if stdin_writer: self.stdin_writer = stdin_writer # rref self.stdio(self) # start accepting stdin # otherwise they don't want our stdin, so leave stdin_writer=None def remote_stdout(self, data): self.stdout.write(data) self.stdout.flush() def remote_stderr(self, data): self.stderr.write(data) self.stderr.flush() def remote_done(self, signal, exitcode): if signal: self._done(127) else: self._done(exitcode) def _err(self, f): self._done(f) def _done(self, res): if not self.done: self.done = True self.d.callback(res) class ClientOptions(usage.Options): synopsis = "Usage: flappclient [--furl=|--furlfile=] COMMAND" optParameters = [ ("furl", None, None, "FURL of the service to contact"), ("furlfile", "f", None, "file containing the FURL of the service"), ] longdesc = """This client invokes a remote service that is running as part of a 'flappserver'. Each service lives at a specific secret FURL, which starts with 'pb://'. This FURL can be passed on the command line with --furl=FURL, or it can be stored in a file (along with comment lines that start with '#') and passed with --furlfile=FILE. Each service has a specific COMMAND type, and the client invocation must match the service. For more details on a specific command, run 'flappclient COMMAND --help', e.g. 'flappclient upload-file --help'. """ subCommands = [ ("upload-file", None, UploadFileOptions, "upload a file"), ("run-command", None, RunCommandOptions, "cause a command to be run"), ] def read_furlfile(self): ff = os.path.expanduser(self["furlfile"]) for line in open(ff).readlines(): line = line.strip() if line.startswith("pb://"): return line return None def postOptions(self): self.furl = self["furl"] if self["furlfile"]: self.furl = self.read_furlfile() if not self.furl: raise usage.UsageError("must provide --furl or --furlfile") if not hasattr(self, 'subOptions'): raise usage.UsageError("must specify a command") def opt_help(self): print >>self.stdout, self.synopsis sys.exit(0) def opt_version(self): from twisted import copyright print >>self.stdout, "Foolscap version:", foolscap.__version__ print >>self.stdout, "Twisted version:", copyright.version sys.exit(0) dispatch_table = { "upload-file": UploadFile, "run-command": RunCommand, } def parse_options(command_name, argv, stdio, stdout, stderr): try: config = ClientOptions() config.stdout = stdout config.stderr = stderr config.parseOptions(argv) config.subOptions.stdio = stdio # for streaming input config.subOptions.stdout = stdout config.subOptions.stderr = stderr except usage.error, e: print >>stderr, "%s: %s" % (command_name, e) print >>stderr c = getattr(config, 'subOptions', config) print >>stderr, str(c) sys.exit(1) return config def run_command(config): c = dispatch_table[config.subCommand]() tub = Tub() try: from twisted.internet import reactor from twisted.internet.endpoints import clientFromString from foolscap.connections import tor CONTROL = os.environ.get("FOOLSCAP_TOR_CONTROL_PORT", "") SOCKS = os.environ.get("FOOLSCAP_TOR_SOCKS_PORT", "") if CONTROL: h = tor.control_endpoint(clientFromString(reactor, CONTROL)) tub.addConnectionHintHandler("tor", h) elif SOCKS: h = tor.socks_endpoint(clientFromString(reactor, SOCKS)) tub.addConnectionHintHandler("tor", h) #else: # h = tor.default_socks() # tub.addConnectionHintHandler("tor", h) except ImportError: pass d = defer.succeed(None) d.addCallback(lambda _ign: tub.startService()) d.addCallback(lambda _ign: tub.getReference(config.furl)) d.addCallback(c.run, config.subOptions) # might provide tub here d.addBoth(lambda res: tub.stopService().addCallback(lambda _ign: res)) return d def run_flappclient(argv=None, run_by_human=True, stdio=StandardIO): if run_by_human: stdout = sys.stdout stderr = sys.stderr else: stdout = StringIO() stderr = StringIO() if argv: command_name,argv = argv[0],argv[1:] else: command_name = sys.argv[0] d = fireEventually() d.addCallback(lambda _ign: parse_options(command_name, argv, stdio, stdout, stderr)) d.addCallback(run_command) if run_by_human: # we need to spin up our own reactor from twisted.internet import reactor stash_rc = [] def good(rc): stash_rc.append(rc) reactor.stop() def oops(f): if f.check(SystemExit): stash_rc.append(f.value.args[0]) else: print >>stderr, "flappclient command failed:" print >>stderr, f stash_rc.append(-1) reactor.stop() d.addCallbacks(good, oops) reactor.run() sys.exit(stash_rc[0]) else: def _convert_system_exit(f): f.trap(SystemExit) return f.value.args[0] d.addErrback(_convert_system_exit) def done(rc): return (rc, stdout.getvalue(), stderr.getvalue()) d.addCallback(done) return d foolscap-0.13.1/src/foolscap/appserver/server.py0000644000076500000240000001116212766553111022317 0ustar warnerstaff00000000000000 import os, sys, json, ast from twisted.application import service from foolscap.api import Tub from foolscap.appserver.services import build_service from foolscap.util import move_into_place class UnknownVersion(Exception): pass def load_service_data(basedir): services_file = os.path.join(basedir, "services.json") if os.path.exists(services_file): data = json.load(open(services_file, "rb")) if data["version"] != 1: raise UnknownVersion("unable to handle version %d" % data["version"]) else: # otherwise look for the old-style separate files services = {} services_basedir = os.path.join(basedir, "services") for (service_basedir, dirnames, filenames) in os.walk(services_basedir): if "service_type" not in filenames: continue assert service_basedir.startswith(services_basedir) swissnum = service_basedir[len(services_basedir):].lstrip(os.sep) s = services[swissnum] = {} s["relative_basedir"] = os.path.join("services", swissnum) service_type_f = os.path.join(service_basedir, "service_type") s["type"] = open(service_type_f).read().strip() # old-style service_args was written with repr(), before the days # of JSON. It was always a tuple, though. It's safe to load this # with ast.literal_eval() . Note that json.loads() wouldn't work # here because repr() emits single-quotes (\x27) and JSON # requires double-quotes (\x22). service_args_f = os.path.join(service_basedir, "service_args") f = open(service_args_f, "rb") args_s = f.read().decode("utf-8") f.close() args = ast.literal_eval(args_s) if isinstance(args, tuple): args = list(args) # make it more like the JSON equivalent s["args"] = args comment_f = os.path.join(service_basedir, "comment") s["comment"] = None if os.path.exists(comment_f): s["comment"] = open(comment_f).read().strip() data = {"version": 1, "services": services} return data # has ["version"]=1 and ["services"] def save_service_data(basedir, data): assert data["version"] == 1 services_file = os.path.join(basedir, "services.json") tmpfile = services_file+".tmp" f = open(tmpfile, "wb") json.dump(data, f, indent=2) f.close() move_into_place(tmpfile, services_file) class AppServer(service.MultiService): def __init__(self, basedir=".", stdout=sys.stdout): service.MultiService.__init__(self) self.basedir = os.path.abspath(basedir) try: umask = open(os.path.join(basedir, "umask")).read().strip() self.umask = int(umask, 8) # octal string like 0022 except EnvironmentError: self.umask = None self.port = open(os.path.join(basedir, "port")).read().strip() self.tub = Tub(certFile=os.path.join(basedir, "tub.pem")) self.tub.listenOn(self.port) self.tub.setServiceParent(self) self.tub.registerNameLookupHandler(self.lookup) self.setMyLocation() print >>stdout, "Server Running" def startService(self): if self.umask is not None: os.umask(self.umask) service.MultiService.startService(self) def setMyLocation(self): location_fn = os.path.join(self.basedir, "location") location = open(location_fn).read().strip() if not location: raise ValueError("This flappserver was created without " "'--location=', and Foolscap no longer uses " "IP-address autodetection. Please edit '%s' " "to contain e.g. 'example.org:12345', with a " "hostname and port number that match this " "server (we're listening on %s)" % (location_fn, self.port)) self.tub.setLocation(location) def lookup(self, name): # walk through our configured services, see if we know about this one services = load_service_data(self.basedir)["services"] s = services.get(name) if not s: return None service_basedir = os.path.join(self.basedir, s["relative_basedir"].encode("utf-8")) service_type = s["type"] service_args = [arg.encode("utf-8") for arg in s["args"]] s = build_service(service_basedir, self.tub, service_type, service_args) s.setServiceParent(self) return s foolscap-0.13.1/src/foolscap/appserver/services.py0000644000076500000240000002441412766553111022640 0ustar warnerstaff00000000000000 import os from twisted.python import usage, runtime, filepath, log from twisted.application import service from twisted.internet import defer, reactor, protocol from foolscap.api import Referenceable class BadServiceArguments(Exception): pass class UnknownServiceType(Exception): pass class BaseOptions(usage.Options): details = None def getUsage(self, width=None): t = usage.Options.getUsage(self, width) if self.details: t += self.details return t class FileUploaderOptions(BaseOptions): synopsis = "Usage: flappserver add BASEDIR upload-file [options] TARGETDIR" details = """ This service allows clients to upload files to a specific directory. """ optFlags = [ ("allow-subdirectories", None, "allow client to write to subdirectories"), ] optParameters = [ ("mode", None, 0644, "(octal) mode to set uploaded files to, use 0644 for world-readable") ] def opt_mode(self, mode): if mode.startswith("0"): self["mode"] = int(mode, 8) else: self["mode"] = int(mode) def parseArgs(self, targetdir): self.targetdir = os.path.abspath(targetdir) if self["allow-subdirectories"]: raise BadServiceArguments("--allow-subdirectories is not yet implemented") if not os.path.exists(self.targetdir): raise BadServiceArguments("targetdir '%s' must already exist" % self.targetdir) if not os.access(self.targetdir, os.W_OK): raise BadServiceArguments("targetdir '%s' must be writeable" % self.targetdir) class FileUploaderReader(Referenceable): BLOCKSIZE = 1024*1024 def __init__(self, f, source): self.f = f self.source = source self.d = defer.Deferred() def read_file(self): self.read_block() return self.d def read_block(self): d = self.source.callRemote("read", self.BLOCKSIZE) d.addCallback(self._got_data) d.addErrback(self._got_error) def _got_data(self, data): if data: self.f.write(data) self.read_block() else: # no more data: we're done self.d.callback(None) def _got_error(self, f): self.d.errback(f) class BadFilenameError(Exception): pass class FileUploader(service.MultiService, Referenceable): def __init__(self, basedir, tub, options): # tub might be None. No network activity should be done until # startService. Validate all options in the constructor. Do not use # the Service/MultiService ".name" attribute (which would prevent # having multiple instances of a single service type in the same # server). service.MultiService.__init__(self) self.basedir = basedir self.tub = tub self.options = options self.targetdir = filepath.FilePath(options.targetdir) def remote_putfile(self, name, source): #if "/" in name or name == "..": # raise BadFilenameError() #targetfile = os.path.join(self.options.targetdir, name) # I think that .child() will reject attempts to follow symlinks out # of the target directory. It will also reject the use of # subdirectories: 'name' must not contain any slashes. To implement # allow-subdirectories, we should pass a list of dirnames and handle # it specially. targetfile = self.targetdir.child(name) #tmpfile = targetfile.temporarySibling() # # temporarySibling() creates a tempfile with the same extension as # the targetfile, which is useless for our purposes: one goal of # file-uploader is to let you send .deb packages to an APT # repository, and we need to hide the .deb from the package-index # building scripts until the whole file is present, so we want an # atomic rename from foo.deb.partial to foo.deb tmpfile = targetfile.siblingExtension(".partial") # TODO: use os.open and set the file mode earlier #f = open(tmpfile, "w") f = tmpfile.open("w") reader = FileUploaderReader(f, source) d = reader.read_file() def _done(res): f.close() if runtime.platform.isWindows() and targetfile.exists(): os.unlink(targetfile.path) tmpfile.moveTo(targetfile) #targetfile.chmod(self.options["mode"]) # older Twisteds do not have FilePath.chmod os.chmod(targetfile.path, self.options["mode"]) return None def _err(fail): f.close() os.unlink(tmpfile.path) return fail d.addCallbacks(_done, _err) return d class CommandRunnerOptions(BaseOptions): synopsis = "Usage: flappserver add BASEDIR run-command [options] TARGETDIR COMMAND.." details = """ This service allows clients to execute a pre-configured command and receive the exit code, optionally providing stdin and receiving stdout/stderr. """ optFlags = [ ("accept-stdin", None, "allow client to write to COMMAND stdin"), ("no-stdin", None, "do not write to COMMAND stdin [default]"), ("log-stdin", None, "log incoming stdin (to twistd.log)"), ("no-log-stdin", None, "do not log incoming stdin [default]"), ("send-stdout", None, "send COMMAND stdout to client [default]"), ("no-stdout", None, "do not send COMMAND stdout to client"), ("log-stdout", None, "log outbound stdout (to twistd.log)"), ("no-log-stdout", None, "do not log oubound stdout [default]"), ("send-stderr", None, "send COMMAND stderr to client [default]"), ("no-stderr", None, "do not send COMMAND stderr to client"), ("log-stderr", None, "log outbound stderr (to twistd.log) [default]"), ("no-log-stderr", None, "do not log outbound stderr"), ] optParameters = [ ] accept_stdin = False def opt_accept_stdin(self): self.accept_stdin = True def opt_no_stdin(self): self.accept_stdin = False send_stdout = True def opt_send_stdout(self): self.send_stdout = True def opt_no_stdout(self): self.send_stdout = False send_stderr = True def opt_send_stderr(self): self.send_stderr = True def opt_no_stderr(self): self.send_stderr = False log_stdin = False def opt_log_stdin(self): self.log_stdin = True def opt_no_log_stdin(self): self.log_stdin = False log_stdout = False def opt_log_stdout(self): self.log_stdout = True def opt_no_log_stdout(self): self.log_stdout = False log_stderr = True def opt_log_stderr(self): self.log_stderr = True def opt_no_log_stderr(self): self.log_stderr = False def parseArgs(self, targetdir, *command_argv): self.targetdir = targetdir self.command_argv = command_argv class CommandPP(protocol.ProcessProtocol): def __init__(self, outpipe, errpipe, watcher, log_stdout, log_stderr): self.outpipe = outpipe self.errpipe = errpipe self.watcher = watcher self.log_stdout = log_stdout self.log_stderr = log_stderr def outReceived(self, data): if self.outpipe: self.outpipe.callRemoteOnly("stdout", data) if self.log_stdout: sent = {True:"sent", False:"not sent"}[bool(self.outpipe)] log.msg("stdout (%s): %r" % (sent, data)) def errReceived(self, data): if self.errpipe: self.errpipe.callRemoteOnly("stderr", data) if self.log_stderr: sent = {True:"sent", False:"not sent"}[bool(self.errpipe)] log.msg("stderr (%s): %r" % (sent, data)) def processEnded(self, reason): e = reason.value code = e.exitCode log.msg("process ended (signal=%s, rc=%s)" % (e.signal, code)) self.watcher.callRemoteOnly("done", e.signal, code) class Command(Referenceable): def __init__(self, process, log_stdin): self.process = process self.log_stdin = log_stdin self.closed = False def remote_feed_stdin(self, data): if not isinstance(data, str): raise TypeError("stdin can accept only strings of bytes, not '%s'" % (type(data),)) if self.log_stdin: log.msg("stdin: %r" % data) self.process.write(data) def remote_close_stdin(self): if not self.closed: self.closed = True if self.log_stdin: log.msg("stdin closed") self.process.closeStdin() class CommandRunner(service.MultiService, Referenceable): def __init__(self, basedir, tub, options): service.MultiService.__init__(self) self.basedir = basedir self.tub = tub self.options = options def remote_execute(self, watcher): o = self.options outpipe = None if o.send_stdout: outpipe = watcher errpipe = None if o.send_stderr: errpipe = watcher pp = CommandPP(outpipe, errpipe, watcher, o.log_stdout, o.log_stderr) # spawnProcess uses os.execvpe, which will search your $PATH executable = o.command_argv[0] log.msg("command started in dir %s: %s" % (o.targetdir, o.command_argv)) p = reactor.spawnProcess(pp, executable, o.command_argv, os.environ, o.targetdir) if o.accept_stdin: c = Command(p, o.log_stdin) watcher.notifyOnDisconnect(c.remote_close_stdin) return c return None all_services = { "upload-file": (FileUploaderOptions, FileUploader), "run-command": (CommandRunnerOptions, CommandRunner), } def build_service(basedir, tub, service_type, service_args): # this will be replaced by a plugin system. For now it's pretty static. if service_type in all_services: (optclass, svcclass) = all_services[service_type] options = optclass() options.parseOptions(service_args) service = svcclass(basedir, tub, options) return service else: raise UnknownServiceType(service_type) foolscap-0.13.1/src/foolscap/banana.py0000644000076500000240000013502112766553111020223 0ustar warnerstaff00000000000000 import struct, time from twisted.internet import protocol, defer, reactor from twisted.python.failure import Failure from twisted.python import log # make sure to import allslicers, so they all get registered. Even if the # need for RootSlicer/etc goes away, do the import here anyway. from foolscap.slicers.allslicers import RootSlicer, RootUnslicer from foolscap.slicers.allslicers import ReplaceVocabSlicer, AddVocabSlicer import stringchain import tokens from tokens import SIZE_LIMIT, STRING, LIST, INT, NEG, \ LONGINT, LONGNEG, VOCAB, FLOAT, OPEN, CLOSE, ABORT, ERROR, \ PING, PONG, \ BananaError, BananaFailure, Violation EPSILON = 0.1 def int2b128(integer, stream): if integer == 0: stream(chr(0)) return assert integer > 0, "can only encode positive integers" while integer: stream(chr(integer & 0x7f)) integer = integer >> 7 def b1282int(st): # NOTE that this is little-endian oneHundredAndTwentyEight = 128 i = 0 place = 0 for char in st: num = ord(char) i = i + (num * (oneHundredAndTwentyEight ** place)) place = place + 1 return i # long_to_bytes and bytes_to_long taken from PyCrypto: Crypto/Util/number.py def long_to_bytes(n, blocksize=0): """long_to_bytes(n:long, blocksize:int) : string Convert a long integer to a byte string. If optional blocksize is given and greater than zero, pad the front of the byte string with binary zeros so that the length is a multiple of blocksize. """ # after much testing, this algorithm was deemed to be the fastest s = '' n = long(n) pack = struct.pack while n > 0: s = pack('>I', n & 0xffffffffL) + s n = n >> 32 # strip off leading zeros for i in range(len(s)): if s[i] != '\000': break else: # only happens when n == 0 s = '\000' i = 0 s = s[i:] # add back some pad bytes. this could be done more efficiently w.r.t. the # de-padding being done above, but sigh... if blocksize > 0 and len(s) % blocksize: s = (blocksize - len(s) % blocksize) * '\000' + s return s def bytes_to_long(s): """bytes_to_long(string) : long Convert a byte string to a long integer. This is (essentially) the inverse of long_to_bytes(). """ acc = 0L unpack = struct.unpack length = len(s) if length % 4: extra = (4 - length % 4) s = '\000' * extra + s length = length + extra for i in range(0, length, 4): acc = (acc << 32) + unpack('>I', s[i:i+4])[0] return acc HIGH_BIT_SET = chr(0x80) # Banana is a big class. It is split up into three sections: sending, # receiving, and connection setup. These used to be separate classes, but # the __init__ functions got too weird. class Banana(protocol.Protocol): def __init__(self, features={}): """ @param features: a dictionary of negotiated connection features """ self.initSend() self.initReceive() def populateVocabTable(self, vocabStrings): """ I expect a list of strings. I will populate my initial vocab table (both inbound and outbound) with this list. It is not safe to use this method once anything has been serialized onto the wire. This method can only be used to set up the initial vocab table based upon a negotiated set of common words. The 'initial-vocab-table-index' parameter is used to decide upon the contents of this table. """ out_vocabDict = dict(zip(vocabStrings, range(len(vocabStrings)))) self.outgoingVocabTableWasReplaced(out_vocabDict) in_vocabDict = dict(zip(range(len(vocabStrings)), vocabStrings)) self.replaceIncomingVocabulary(in_vocabDict) ### connection setup def connectionMade(self): if self.debugSend: print "Banana.connectionMade" self.initSlicer() self.initUnslicer() if self.keepaliveTimeout is not None: self.dataLastReceivedAt = time.time() t = reactor.callLater(self.keepaliveTimeout + EPSILON, self.keepaliveTimerFired) self.keepaliveTimer = t self.useKeepalives = True if self.disconnectTimeout is not None: self.dataLastReceivedAt = time.time() t = reactor.callLater(self.disconnectTimeout + EPSILON, self.disconnectTimerFired) self.disconnectTimer = t self.useKeepalives = True # prime the pump self.produce() def connectionLost(self, why): if self.disconnectTimer: self.disconnectTimer.cancel() self.disconnectTimer = None if self.keepaliveTimer: self.keepaliveTimer.cancel() self.keepaliveTimer = None protocol.Protocol.connectionLost(self, why) ### SendBanana # called by .send() # calls transport.write() and transport.loseConnection() slicerClass = RootSlicer # this is used in connectionMade() paused = False streamable = True # this is checked at connectionMade() time debugSend = False def initSend(self): self.openCount = 0 self.outgoingVocabulary = {} self.nextAvailableOutgoingVocabularyIndex = 0 self.pendingVocabAdditions = set() def initSlicer(self): self.rootSlicer = self.slicerClass(self) self.rootSlicer.allowStreaming(self.streamable) assert tokens.ISlicer.providedBy(self.rootSlicer) assert tokens.IRootSlicer.providedBy(self.rootSlicer) itr = self.rootSlicer.slice() next = iter(itr).next top = (self.rootSlicer, next, None) self.slicerStack = [top] def send(self, obj): if self.debugSend: print "Banana.send(%s)" % obj return self.rootSlicer.send(obj) def _slice_error(self, f, s): log.msg("Error in Deferred returned by slicer %s: %s" % (s, f)) self.sendFailed(f) def produce(self, dummy=None): # optimize: cache 'next' because we get many more tokens than stack # pushes/pops while self.slicerStack and not self.paused: if self.debugSend: print "produce.loop" try: slicer, next, openID = self.slicerStack[-1] obj = next() if self.debugSend: print " produce.obj=%s" % (obj,) if isinstance(obj, defer.Deferred): for s,n,o in self.slicerStack: if not s.streamable: raise Violation("parent not streamable") obj.addCallback(self.produce) obj.addErrback(self._slice_error, s) # this is the primary exit point break elif type(obj) in (int, long, float, str): # sendToken raises a BananaError for weird tokens self.sendToken(obj) else: # newSlicerFor raises a Violation for unsendable types # pushSlicer calls .slice, which can raise Violation try: slicer = self.newSlicerFor(obj) self.pushSlicer(slicer, obj) except Violation, v: # pushSlicer is arranged such that the pushing of # the Slicer and the sending of the OPEN happen # together: either both occur or neither occur. In # addition, there is nothing past the OPEN/push # which can cause an exception. # Therefore, if an exception was raised, we know # that the OPEN has not been sent (so we don't have # to send an ABORT), and that the new Unslicer has # not been pushed (so we don't have to pop one from # the stack) f = BananaFailure() if self.debugSend: print " violation in newSlicerFor:", f self.handleSendViolation(f, doPop=False, sendAbort=False) except StopIteration: if self.debugSend: print "StopIteration" self.popSlicer() except Violation, v: # Violations that occur because of Constraints are caught # before the Slicer is pushed. A Violation that is caught # here was raised inside .next(), or .streamable wasn't # obeyed. The Slicer should now be abandoned. if self.debugSend: print " violation in .next:", v f = BananaFailure() self.handleSendViolation(f, doPop=True, sendAbort=True) except: print "exception in produce" log.msg("exception in produce") self.sendFailed(Failure()) # there is no point to raising this again. The Deferreds are # all errbacked in sendFailed(). This function was called # inside a Deferred which errbacks to sendFailed(), and # we've already called that once. The connection will be # dropped by sendFailed(), and the error is logged, so there # is nothing left to do. return assert self.slicerStack # should never be empty def handleSendViolation(self, f, doPop, sendAbort): f.value.setLocation(self.describeSend()) while True: top = self.slicerStack[-1][0] if self.debugSend: print " handleSendViolation.loop, top=%s" % top # should we send an ABORT? Only if an OPEN has been sent, which # happens in pushSlicer (if at all). if sendAbort: lastOpenID = self.slicerStack[-1][2] if lastOpenID is not None: if self.debugSend: print " sending ABORT(%s)" % lastOpenID self.sendAbort(lastOpenID) # should we pop the Slicer? yes if doPop: if self.debugSend: print " popping %s" % top self.popSlicer() if not self.slicerStack: if self.debugSend: print "RootSlicer died!" raise BananaError("Hey! You killed the RootSlicer!") top = self.slicerStack[-1][0] # now inform the parent. If they also give up, we will # loop, popping more Slicers off the stack until the # RootSlicer ignores the error if self.debugSend: print " notifying parent", top f = top.childAborted(f) if f: doPop = True sendAbort = True continue else: break # the parent wants to forge ahead def newSlicerFor(self, obj): if tokens.ISlicer.providedBy(obj): return obj topSlicer = self.slicerStack[-1][0] # slicerForObject could raise a Violation, for unserializeable types return topSlicer.slicerForObject(obj) def pushSlicer(self, slicer, obj): if self.debugSend: print "push", slicer assert len(self.slicerStack) < 10000 # failsafe # if this method raises a Violation, it means that .slice failed, # and neither the OPEN nor the stack-push has occurred topSlicer = self.slicerStack[-1][0] slicer.parent = topSlicer # we start the Slicer (by getting its iterator) first, so that if it # fails we can refrain from sending the OPEN (hence we do not have # to send an ABORT and CLOSE, which simplifies the send logic # considerably). slicer.slice is the only place where a Violation # can be raised: it is caught and passed cleanly to the parent. If # it happens anywhere else, or if any other exception is raised, the # connection will be dropped. # the downside to this approach is that .slice happens before # .registerRefID, so any late-validation being done in .slice will # not be able to detect the fact that this object has already begun # serialization. Validation performed in .next is ok. # also note that if .slice is a generator, any exception it raises # will not occur until .next is called, which happens *after* the # slicer has been pushed. This check is only useful for .slice # methods which are *not* generators. itr = slicer.slice(topSlicer.streamable, self) next = iter(itr).next # we are now committed to sending the OPEN token, meaning that # failures after this point will cause an ABORT/CLOSE to be sent openID = None if slicer.sendOpen: openID = self.sendOpen() if slicer.trackReferences: topSlicer.registerRefID(openID, obj) # note that the only reason to hold on to the openID here is for # the debug/optional copy in the CLOSE token. Consider ripping # this code out if we decide to stop sending that copy. slicertuple = (slicer, next, openID) self.slicerStack.append(slicertuple) def popSlicer(self): slicer, next, openID = self.slicerStack.pop() if openID is not None: self.sendClose(openID) if self.debugSend: print "pop", slicer def describeSend(self): where = [] for i in self.slicerStack: try: piece = i[0].describe() except: log.msg("Banana.describeSend") log.err() piece = "???" where.append(piece) return ".".join(where) def setOutgoingVocabulary(self, vocabStrings): """Schedule a replacement of the outbound VOCAB table. Higher-level code may call this at any time with a list of strings. Immediately after the replacement has occured, the outbound VOCAB table will contain all of the strings in vocabStrings and nothing else. This table tells the token-sending code which strings to abbreviate with short integers in a VOCAB token. This function can be called at any time (even while the protocol is in the middle of serializing and transmitting some other object) because it merely schedules a replacement to occur at some point in the future. A special marker (the ReplaceVocabSlicer) is placed in the outbound queue, and the table replacement will only happend after all the items ahead of that marker have been serialized. At the same time the table is replaced, a (set-vocab..) sequence will be serialized towards the far end. This insures that we set our outbound table at the same 'time' as the far end starts using it. """ # build a VOCAB message, send it, then set our outgoingVocabulary # dictionary to start using the new table assert isinstance(vocabStrings, (list, tuple)) for s in vocabStrings: assert isinstance(s, str) vocabDict = dict(zip(vocabStrings, range(len(vocabStrings)))) s = ReplaceVocabSlicer(vocabDict) # the ReplaceVocabSlicer does some magic to insure the VOCAB message # does not use vocab tokens itself. This would be legal (sort of a # differential compression), but confusing. It accomplishes this by # clearing our self.outgoingVocabulary dict when it begins to be # serialized. self.send(s) # likewise, when it finishes, the ReplaceVocabSlicer replaces our # self.outgoingVocabulary dict when it has finished sending the # strings. It is important that this occur in the serialization code, # or somewhen very close to it, because otherwise there could be a # race condition that could result in some strings being vocabized # with the wrong keys. def addToOutgoingVocabulary(self, value): """Schedule 'value' for addition to the outbound VOCAB table. This may be called at any time. If the string is already scheduled for addition, or if it is already in the VOCAB table, it will be ignored. (TODO: does this introduce an annoying-but-not-fatal race condition?) The string will not actually be added to the table until the outbound serialization queue has been serviced. """ assert isinstance(value, str) if value in self.outgoingVocabulary: return if value in self.pendingVocabAdditions: return self.pendingVocabAdditions.add(str) s = AddVocabSlicer(value) self.send(s) def outgoingVocabTableWasReplaced(self, newTable): # this is called by the ReplaceVocabSlicer to manipulate our table. # It must certainly *not* be called by higher-level user code. self.outgoingVocabulary = newTable if newTable: maxIndex = max(newTable.values()) + 1 self.nextAvailableOutgoingVocabularyIndex = maxIndex else: self.nextAvailableOutgoingVocabularyIndex = 0 def allocateEntryInOutgoingVocabTable(self, string): assert string not in self.outgoingVocabulary # TODO: a softer failure more for this assert is to re-send the # existing key. To make sure that really happens, though, we have to # remove it from the vocab table, otherwise we'll tokenize the # string. If we can insure that, then this failure mode would waste # time and network but would otherwise be harmless. # # return self.outgoingVocabulary[string] self.pendingVocabAdditions.remove(self.value) index = self.nextAvailableOutgoingVocabularyIndex self.nextAvailableOutgoingVocabularyIndex = index + 1 return index def outgoingVocabTableWasAmended(self, index, string): self.outgoingVocabulary[string] = index # these methods define how we emit low-level tokens def sendPING(self, number=0): if number: int2b128(number, self.transport.write) self.transport.write(PING) def sendPONG(self, number): if number: int2b128(number, self.transport.write) self.transport.write(PONG) def sendOpen(self): openID = self.openCount self.openCount += 1 int2b128(openID, self.transport.write) self.transport.write(OPEN) return openID def sendToken(self, obj): write = self.transport.write if isinstance(obj, (int, long)): if obj >= 2**31: s = long_to_bytes(obj) int2b128(len(s), write) write(LONGINT) write(s) elif obj >= 0: int2b128(obj, write) write(INT) elif -obj > 2**31: # NEG is [-2**31, 0) s = long_to_bytes(-obj) int2b128(len(s), write) write(LONGNEG) write(s) else: int2b128(-obj, write) write(NEG) elif isinstance(obj, float): write(FLOAT) write(struct.pack("!d", obj)) elif isinstance(obj, str): if self.outgoingVocabulary.has_key(obj): symbolID = self.outgoingVocabulary[obj] int2b128(symbolID, write) write(VOCAB) else: self.maybeVocabizeString(obj) int2b128(len(obj), write) write(STRING) write(obj) else: raise BananaError, "could not send object: %s" % repr(obj) def maybeVocabizeString(self, string): # TODO: keep track of the last 30 strings we've send in full. If this # string appears more than 3 times on that list, create a vocab item # for it. Make sure we don't start using the vocab number until the # ADDVOCAB token has been queued. if False: self.addToOutgoingVocabulary(string) def sendClose(self, openID): int2b128(openID, self.transport.write) self.transport.write(CLOSE) def sendAbort(self, count=0): int2b128(count, self.transport.write) self.transport.write(ABORT) def sendError(self, msg): if not self.transport: return if len(msg) > SIZE_LIMIT: msg = msg[:SIZE_LIMIT-10] + "..." int2b128(len(msg), self.transport.write) self.transport.write(ERROR) self.transport.write(msg) # now you should drop the connection self.transport.loseConnection() def sendFailed(self, f): # call this if an exception is raised in transmission. The Failure # will be logged and the connection will be dropped. This is # suitable for use as an errback handler. print "SendBanana.sendFailed:", f log.msg("Sendfailed.sendfailed") log.err(f) try: if self.transport: self.transport.loseConnection() except: print "exception during transport.loseConnection" log.err() try: self.rootSlicer.connectionLost(f) except: print "exception during rootSlicer.connectionLost" log.err() ### ReceiveBanana # called with dataReceived() # calls self.receivedObject() unslicerClass = RootUnslicer debugReceive = False logViolations = False logReceiveErrors = True useKeepalives = False keepaliveTimeout = None keepaliveTimer = None disconnectTimeout = None disconnectTimer = None def initReceive(self): self.inOpen = False # set during the Index Phase of an OPEN sequence self.opentype = [] # accumulates Index Tokens # to pre-negotiate, set the negotiation parameters and set # self.negotiated to True. It might instead make sense to fill # self.buffer with the inbound negotiation block. self.negotiated = False self.connectionAbandoned = False self.buffer = stringchain.StringChain() self.incomingVocabulary = {} self.skipBytes = 0 # used to discard a single long token self.discardCount = 0 # used to discard non-primitive objects self.exploded = None # last-ditch error catcher def initUnslicer(self): self.rootUnslicer = self.unslicerClass(self) self.receiveStack = [self.rootUnslicer] self.objectCounter = 0 self.objects = {} def printStack(self, verbose=0): print "STACK:" for s in self.receiveStack: if verbose: d = s.__dict__.copy() del d['protocol'] print " %s: %s" % (s, d) else: print " %s" % s def setObject(self, count, obj): for i in range(len(self.receiveStack)-1, -1, -1): self.receiveStack[i].setObject(count, obj) def getObject(self, count): for i in range(len(self.receiveStack)-1, -1, -1): obj = self.receiveStack[i].getObject(count) if obj is not None: return obj raise ValueError, "dangling reference '%d'" % count def replaceIncomingVocabulary(self, vocabDict): # maps small integer to string, should be called in response to a # OPEN(set-vocab) sequence. self.incomingVocabulary = vocabDict def addIncomingVocabulary(self, key, value): # called in response to an OPEN(add-vocab) sequence self.incomingVocabulary[key] = value def dataReceived(self, chunk): if self.connectionAbandoned: return if self.useKeepalives: self.dataLastReceivedAt = time.time() try: self.handleData(chunk) except Exception, e: if isinstance(e, BananaError): # only reveal the reason if it is a protocol error e.where = self.describeReceive() msg = str(e) # send them the text of the error else: msg = ("exception while processing data, more " "information in the logfiles") if not self.logReceiveErrors: msg += ", except that self.logReceiveErrors=False" msg += ", sucks to be you" self.sendError(msg) self.connectionAbandoned = True self.reportReceiveError(Failure()) def keepaliveTimerFired(self): self.keepaliveTimer = None age = time.time() - self.dataLastReceivedAt if age > self.keepaliveTimeout: # the connection looks idle, so let's provoke a response self.sendPING() # we restart the timer in either case t = reactor.callLater(self.keepaliveTimeout + EPSILON, self.keepaliveTimerFired) self.keepaliveTimer = t def disconnectTimerFired(self): self.disconnectTimer = None age = time.time() - self.dataLastReceivedAt if age > self.disconnectTimeout: # the connection looks dead, so drop it log.msg("disconnectTimeout, no data for %d seconds" % age) self.connectionTimedOut() # we assume that connectionTimedOut() will actually drop the # connection, so we don't restart the timer. TODO: this might not # be the right thing to do, perhaps we should restart it # unconditionally. else: # we're still ok, so restart the timer t = reactor.callLater(self.disconnectTimeout + EPSILON, self.disconnectTimerFired) self.disconnectTimer = t def getDataLastReceivedAt(self): """If keepalives are enabled, this returns the seconds-since-epoch when the most recent data was received on this connection. If keepalives are disabled (which is the detault), it returns None.""" if self.useKeepalives: return self.dataLastReceivedAt return None def connectionTimedOut(self): # this is to be implemented by higher-level code. It ought to log a # suitable message and then drop the connection. pass def reportReceiveError(self, f): # tests can override this to stash the failure somewhere else. Tests # which intentionally cause an error set self.logReceiveErrors=False # so that the log.err doesn't flunk the test. log.msg("Banana.reportReceiveError: an error occured during receive") if self.logReceiveErrors: log.err(f) if self.debugReceive: # trial watches log.err and treats it as a failure, so log the # exception in a way that doesn't make trial flunk the test log.msg(f.getBriefTraceback()) def handleData(self, chunk): # buffer, assemble into tokens # call self.receiveToken(token) with each if self.skipBytes: if len(chunk) <= self.skipBytes: # skip the whole chunk self.skipBytes -= len(chunk) return # skip part of the chunk, and stop skipping chunk = chunk[self.skipBytes:] self.skipBytes = 0 self.buffer.append(chunk) # Loop through the available input data, extracting one token per # pass. while len(self.buffer): first65 = self.buffer.popleft(65) pos = 0 for ch in first65: if ch >= HIGH_BIT_SET: break pos = pos + 1 if pos > 64: # drop the connection. We log more of the buffer, but not # all of it, to make it harder for someone to spam our # logs. s = first65 + self.buffer.popleft(200) raise BananaError("token prefix is limited to 64 bytes: " "but got %r" % s) else: # we've run out of buffer without seeing the high bit, which # means we're still waiting for header to finish self.buffer.appendleft(first65) return assert pos <= 64 # At this point, the header and type byte have been received. # The body may or may not be complete. typebyte = first65[pos] if pos: header = b1282int(first65[:pos]) else: header = 0 # rejected is set as soon as a violation is detected. It # indicates that this single token will be rejected. rejected = False if self.discardCount: rejected = True wasInOpen = self.inOpen if typebyte == OPEN: self.inboundObjectCount = self.objectCounter self.objectCounter += 1 if self.inOpen: raise BananaError("OPEN token followed by OPEN") self.inOpen = True # the inOpen flag is set as soon as the OPEN token is # witnessed (even it it gets rejected later), because it # means that there is a new sequence starting that must be # handled somehow (either discarded or given to a new # Unslicer). # The inOpen flag is cleared when the Index Phase ends. There # are two possibilities: 1) a new Unslicer is pushed, and # tokens are delivered to it normally. 2) a Violation was # raised, and the tokens must be discarded # (self.discardCount++). *any* rejection-caused True->False # transition of self.inOpen must be accompanied by exactly # one increment of self.discardCount # determine if this token will be accepted, and if so, how large # it is allowed to be (for STRING and LONGINT/LONGNEG) if ((not rejected) and (typebyte not in (PING, PONG, ABORT, CLOSE, ERROR))): # PING, PONG, ABORT, CLOSE, and ERROR are always legal. All # others (including OPEN) can be rejected by the schema: for # example, a list of integers would reject STRING, VOCAB, and # OPEN because none of those will produce integers. If the # unslicer's .checkToken rejects the tokentype, its # .receiveChild will immediately get an Failure try: # the purpose here is to limit the memory consumed by # the body of a STRING, OPEN, LONGINT, or LONGNEG token # (i.e., the size of a primitive type). If the sender # wants to feed us more data than we want to accept, the # checkToken() method should raise a Violation. This # will never be called with ABORT or CLOSE types. top = self.receiveStack[-1] if wasInOpen: top.openerCheckToken(typebyte, header, self.opentype) else: top.checkToken(typebyte, header) except Violation: rejected = True f = BananaFailure() if wasInOpen: methname = "openerCheckToken" else: methname = "checkToken" self.handleViolation(f, methname, inOpen=self.inOpen) self.inOpen = False if typebyte == ERROR and header > SIZE_LIMIT: # someone is trying to spam us with an ERROR token. Drop # them with extreme prejudice. raise BananaError("oversized ERROR token") self.buffer.appendleft(first65[pos+1:]) # determine what kind of token it is. Each clause finishes in # one of four ways: # # raise BananaError: the protocol was violated so badly there is # nothing to do for it but hang up abruptly # # return: if the token is not yet complete (need more data) # # continue: if the token is complete but no object (for # handleToken) was produced, e.g. OPEN, CLOSE, ABORT # # obj=foo: the token is complete and an object was produced # # note that if rejected==True, the object is dropped instead of # being passed up to the current Unslicer if typebyte == OPEN: self.inboundOpenCount = header if rejected: if self.debugReceive: print "DROP (OPEN)" if self.inOpen: # we are discarding everything at the old level, so # discard everything in the new level too self.discardCount += 1 if self.debugReceive: print "++discardCount (OPEN), now %d" \ % self.discardCount self.inOpen = False else: # the checkToken handleViolation has already started # discarding this new sequence, we don't have to pass else: self.inOpen = True self.opentype = [] continue elif typebyte == CLOSE: count = header if self.discardCount: self.discardCount -= 1 if self.debugReceive: print "--discardCount (CLOSE), now %d" \ % self.discardCount else: self.handleClose(count) continue elif typebyte == ABORT: count = header # TODO: this isn't really a Violation, but we need something # to describe it. It does behave identically to what happens # when receiveChild raises a Violation. The .handleViolation # will pop the now-useless Unslicer and start discarding # tokens just as if the Unslicer had made the decision. if rejected: if self.debugReceive: print "DROP (ABORT)" # I'm ignoring you, LALALALALA. # # In particular, do not deliver a second Violation # because of the ABORT that we're supposed to be # ignoring because of a first Violation that happened # earlier. continue try: # slightly silly way to do it, but nice and uniform raise Violation("ABORT received") except Violation: f = BananaFailure() self.handleViolation(f, "receive-abort") continue elif typebyte == ERROR: strlen = header if len(self.buffer) >= strlen: # the whole string is available obj = self.buffer.popleft(strlen) # handleError must drop the connection self.handleError(obj) return else: self.buffer.appendleft(first65[:pos+1]) return # there is more to come elif typebyte == LIST: raise BananaError("oldbanana peer detected, " + "compatibility code not yet written") #listStack.append((header, [])) elif typebyte == STRING: strlen = header if len(self.buffer) >= strlen: # the whole string is available obj = self.buffer.popleft(strlen) # although it might be rejected else: # there is more to come if rejected: # drop all we have and note how much more should be # dropped if self.debugReceive: print "DROPPED some string bits" self.skipBytes = strlen - len(self.buffer) self.buffer.clear() else: self.buffer.appendleft(first65[:pos+1]) return elif typebyte == INT: obj = int(header) elif typebyte == NEG: # -2**31 is too large for a positive int, so go through # LongType first obj = int(-long(header)) elif typebyte == LONGINT or typebyte == LONGNEG: strlen = header if len(self.buffer) >= strlen: # the whole number is available obj = bytes_to_long(self.buffer.popleft(strlen)) if typebyte == LONGNEG: obj = -obj # although it might be rejected else: # there is more to come if rejected: # drop all we have and note how much more should be # dropped self.skipBytes = strlen - len(self.buffer) self.buffer.clear() else: self.buffer.appendleft(first65[:pos+1]) return elif typebyte == VOCAB: obj = self.incomingVocabulary[header] # TODO: bail if expanded string is too big # this actually means doing self.checkToken(VOCAB, len(obj)) # but we have to make sure we handle the rejection properly elif typebyte == FLOAT: if len(self.buffer) >= 8: obj = struct.unpack("!d", self.buffer.popleft(8))[0] else: # this case is easier than STRING, because it is only 8 # bytes. We don't bother skipping anything. self.buffer.appendleft(first65[:pos+1]) return elif typebyte == PING: self.sendPONG(header) continue # otherwise ignored elif typebyte == PONG: continue # otherwise ignored else: raise BananaError("Invalid Type Byte 0x%x" % ord(typebyte)) if not rejected: if self.inOpen: self.handleOpen(self.inboundOpenCount, self.inboundObjectCount, obj) # handleOpen might push a new unslicer and clear # .inOpen, or leave .inOpen true and append the object # to .indexOpen else: self.handleToken(obj) else: if self.debugReceive: print "DROP", type(obj), obj pass # drop the object # while loop ends here # note: this is redundant, as there are no 'break' statements in that # loop, and the loop exit condition is 'while len(self.buffer)' self.buffer.clear() def handleOpen(self, openCount, objectCount, indexToken): self.opentype.append(indexToken) opentype = tuple(self.opentype) if self.debugReceive: print "handleOpen(%d,%d,%s)" % (openCount, objectCount, indexToken) top = self.receiveStack[-1] try: # obtain a new Unslicer to handle the object child = top.doOpen(opentype) if not child: if self.debugReceive: print " doOpen wants more index tokens" return # they want more index tokens, leave .inOpen=True if self.debugReceive: print " opened[%d] with %s" % (openCount, child) except Violation: # must discard the rest of the child object. There is no new # unslicer pushed yet, so we don't use abandonUnslicer self.inOpen = False f = BananaFailure() self.handleViolation(f, "doOpen", inOpen=True) return assert tokens.IUnslicer.providedBy(child), "child is %s" % child self.inOpen = False child.protocol = self child.openCount = openCount child.parent = top self.receiveStack.append(child) try: child.start(objectCount) except Violation: # the child is now on top, so use abandonUnslicer to discard the # rest of the child f = BananaFailure() # notifies the new child self.handleViolation(f, "start") def handleToken(self, token, ready_deferred=None): top = self.receiveStack[-1] if self.debugReceive: print "handleToken(%s)" % (token,) if ready_deferred: assert isinstance(ready_deferred, defer.Deferred) try: top.receiveChild(token, ready_deferred) except Violation: # this is how the child says "I've been contaminated". We don't # pop them automatically: if they want that, they should return # back the failure in their reportViolation method. f = BananaFailure() self.handleViolation(f, "receiveChild") def handleClose(self, closeCount): if self.debugReceive: print "handleClose(%d)" % closeCount if self.receiveStack[-1].openCount != closeCount: raise BananaError("lost sync, got CLOSE(%d) but expecting %s" \ % (closeCount, self.receiveStack[-1].openCount)) child = self.receiveStack[-1] # don't pop yet: describe() needs it try: obj, ready_deferred = child.receiveClose() except Violation: # the child is contaminated. However, they're finished, so we # don't have to discard anything. Just give an Failure to the # parent instead of the object they would have returned. f = BananaFailure() self.handleViolation(f, "receiveClose", inClose=True) return if self.debugReceive: print "receiveClose returned", obj try: child.finish() except Violation: # .finish could raise a Violation if an object that references # the child is just now deciding that they don't like it # (perhaps their TupleConstraint couldn't be asserted until the # tuple was complete and referenceable). In this case, the child # has produced a valid object, but an earlier (incomplete) # object is not valid. So we treat this as if this child itself # raised the Violation. The .where attribute will point to this # child, which is the node that caused somebody problems, but # will be marked , which indicates that it wasn't the # child itself which raised the Violation. TODO: not true # # TODO: it would be more useful if the UF could also point to # the completing object (the one which raised Violation). f = BananaFailure() self.handleViolation(f, "finish", inClose=True) return self.receiveStack.pop() # now deliver the object to the parent self.handleToken(obj, ready_deferred) def handleViolation(self, f, methname, inOpen=False, inClose=False): """An Unslicer has decided to give up, or we have given up on it (because we received an ABORT token). """ where = self.describeReceive() f.value.setLocation(where) if self.debugReceive: print " handleViolation-%s (inOpen=%s, inClose=%s): %s" \ % (methname, inOpen, inClose, f) assert isinstance(f, BananaFailure) if self.logViolations: log.msg("Violation in %s at %s" % (methname, where)) log.err(f) if inOpen: self.discardCount += 1 if self.debugReceive: print " ++discardCount (inOpen), now %d" % self.discardCount while True: # tell the parent that their child is dead. This is useful for # things like PB, which may want to errback the current request. if self.debugReceive: print " reportViolation to %s" % self.receiveStack[-1] f = self.receiveStack[-1].reportViolation(f) if not f: # they absorbed the failure if self.debugReceive: print " buck stopped, error absorbed" break # the old top wants to propagate it upwards if self.debugReceive: print " popping %s" % self.receiveStack[-1] if not inClose: self.discardCount += 1 if self.debugReceive: print " ++discardCount (pop, not inClose), now %d" \ % self.discardCount inClose = False old = self.receiveStack.pop() try: # TODO: if handleClose encountered a Violation in .finish, # we will end up calling it a second time old.finish() # ?? except Violation: pass # they've already failed once if not self.receiveStack: # now there's nobody left to create new Unslicers, so we # must drop the connection why = "Oh my god, you killed the RootUnslicer! " + \ "You bastard!!" raise BananaError(why) # now we loop until someone absorbs the failure def handleError(self, msg): log.msg("got banana ERROR from remote side: %s" % msg) self.transport.loseConnection() def describeReceive(self): where = [] for i in self.receiveStack: try: piece = i.describe() except: piece = "???" #raise where.append(piece) return ".".join(where) def receivedObject(self, obj): """Decoded objects are delivered here, unless you use a RootUnslicer variant which does something else in its .childFinished method. """ raise NotImplementedError def reportViolation(self, why): return why foolscap-0.13.1/src/foolscap/base32.py0000644000076500000240000000161512766553111020063 0ustar warnerstaff00000000000000 # copied from the waterken.org Web-Calculus python implementation def encode(bytes): chars = "" buffer = 0; n = 0; for b in bytes: buffer = buffer << 8 buffer = buffer | ord(b) n = n + 8 while n >= 5: chars = chars + _encode((buffer >> (n - 5)) & 0x1F) n = n - 5; buffer = buffer & 0x1F # To quiet any warning from << operator if n > 0: buffer = buffer << (5 - n) chars = chars + _encode(buffer & 0x1F) return chars def _encode(v): if v < 26: return chr(ord('a') + v) else: return chr(ord('2') + (v - 26)) # we use the rfc4648 base32 alphabet, in lowercase BASE32_ALPHABET = "".join([_encode(i) for i in range(0x20)]) # 'abcdefghijklmnopqrstuvwxyz234567' def is_base32(s): for c in s.lower(): if c not in BASE32_ALPHABET: return False return True foolscap-0.13.1/src/foolscap/broker.py0000644000076500000240000007413313204160675020272 0ustar warnerstaff00000000000000 # This module is responsible for the per-connection Broker object import types, time from itertools import count from zope.interface import implements from twisted.python import failure from twisted.internet import defer, error from twisted.internet import interfaces as twinterfaces from twisted.internet.protocol import connectionDone from foolscap import banana, tokens, ipb, vocab from foolscap import call, slicer, referenceable, copyable, remoteinterface from foolscap.constraint import Any from foolscap.tokens import Violation, BananaError from foolscap.ipb import DeadReferenceError, IBroker from foolscap.slicers.root import RootSlicer, RootUnslicer, ScopedRootSlicer from foolscap.eventual import eventually from foolscap.logging import log LOST_CONNECTION_ERRORS = [error.ConnectionLost, error.ConnectionDone] try: from OpenSSL import SSL LOST_CONNECTION_ERRORS.append(SSL.Error) except ImportError: pass PBTopRegistry = { ("call",): call.CallUnslicer, ("answer",): call.AnswerUnslicer, ("error",): call.ErrorUnslicer, } PBOpenRegistry = { ('arguments',): call.ArgumentUnslicer, ('my-reference',): referenceable.ReferenceUnslicer, ('your-reference',): referenceable.YourReferenceUnslicer, ('their-reference',): referenceable.TheirReferenceUnslicer, # ('copyable', classname) is handled inline, through the CopyableRegistry } class PBRootUnslicer(RootUnslicer): # topRegistries defines what objects are allowed at the top-level topRegistries = [PBTopRegistry] # openRegistries defines what objects are allowed at the second level and # below openRegistries = [slicer.UnslicerRegistry, PBOpenRegistry] logViolations = False def checkToken(self, typebyte, size): if typebyte != tokens.OPEN: raise BananaError("top-level must be OPEN") def openerCheckToken(self, typebyte, size, opentype): if typebyte == tokens.STRING: if len(opentype) == 0: if size > self.maxIndexLength: why = "first opentype STRING token is too long, %d>%d" % \ (size, self.maxIndexLength) raise Violation(why) if opentype == ("copyable",): # TODO: this is silly, of course (should pre-compute maxlen) maxlen = reduce(max, [len(cname) \ for cname in copyable.CopyableRegistry.keys()] ) if size > maxlen: why = "copyable-classname token is too long, %d>%d" % \ (size, maxlen) raise Violation(why) elif typebyte == tokens.VOCAB: return else: # TODO: hack for testing raise Violation("index token 0x%02x not STRING or VOCAB" % \ ord(typebyte)) raise BananaError("index token 0x%02x not STRING or VOCAB" % \ ord(typebyte)) def open(self, opentype): # used for lower-level objects, delegated up from childunslicer.open child = RootUnslicer.open(self, opentype) if child: child.broker = self.broker return child def doOpen(self, opentype): child = RootUnslicer.doOpen(self, opentype) if child: child.broker = self.broker return child def reportViolation(self, f): if self.logViolations: print "hey, something failed:", f return None # absorb the failure def receiveChild(self, token, ready_deferred): if isinstance(token, call.InboundDelivery): self.broker.scheduleCall(token, ready_deferred) class PBRootSlicer(RootSlicer): slicerTable = {types.MethodType: referenceable.CallableSlicer, types.FunctionType: referenceable.CallableSlicer, } def registerRefID(self, refid, obj): # references are never Broker-scoped: they're always scoped more # narrowly, by the CallSlicer or the AnswerSlicer. assert 0 class RIBroker(remoteinterface.RemoteInterface): def getReferenceByName(name=str): """If I have published an object by that name, return a reference to it.""" # return Remote(interface=any) return Any() def decref(clid=int, count=int): """Release some references to my-reference 'clid'. I will return an ack when the operation has completed.""" return None def decgift(giftID=int, count=int): """Release some reference to a their-reference 'giftID' that was sent earlier.""" return None class Broker(banana.Banana, referenceable.Referenceable): """I manage a connection to a remote Broker. @ivar tub: the L{Tub} which contains us @ivar yourReferenceByCLID: maps your CLID to a RemoteReferenceData #@ivar yourReferenceByName: maps a per-Tub name to a RemoteReferenceData @ivar yourReferenceByURL: maps a global URL to a RemoteReferenceData """ implements(RIBroker, IBroker) slicerClass = PBRootSlicer unslicerClass = PBRootUnslicer unsafeTracebacks = True requireSchema = False disconnected = False factory = None tub = None remote_broker = None startingTLS = False startedTLS = False use_remote_broker = True def __init__(self, remote_tubref, params={}, keepaliveTimeout=None, disconnectTimeout=None, connectionInfo=None): banana.Banana.__init__(self, params) self._expose_remote_exception_types = True self.remote_tubref = remote_tubref self.keepaliveTimeout = keepaliveTimeout self.disconnectTimeout = disconnectTimeout self._banana_decision_version = params.get("banana-decision-version") vocab_table_index = params.get('initial-vocab-table-index') if vocab_table_index: table = vocab.INITIAL_VOCAB_TABLES[vocab_table_index] self.populateVocabTable(table) self.initBroker() self.current_slave_IR = params.get('current-slave-IR') self.current_seqnum = params.get('current-seqnum') self.creation_timestamp = time.time() self._connectionInfo = connectionInfo def initBroker(self): # tracking Referenceables # sending side uses these self.nextCLID = count(1).next # 0 is for the broker self.myReferenceByPUID = {} # maps ref.processUniqueID to a tracker self.myReferenceByCLID = {} # maps CLID to a tracker # receiving side uses these self.yourReferenceByCLID = {} self.yourReferenceByURL = {} # tracking Gifts self.nextGiftID = count(1).next self.myGifts = {} # maps (broker,clid) to (rref, giftID, count) self.myGiftsByGiftID = {} # maps giftID to (broker,clid) # remote calls # sending side uses these self.nextReqID = count(1).next # 0 means "we don't want a response" self.waitingForAnswers = {} # we wait for the other side to answer self.disconnectWatchers = [] # Callables waiting to hear about connectionLost. self._connectionLostWatchers = [] # receiving side uses these self.inboundDeliveryQueue = [] self._waiting_for_call_to_be_ready = False self.activeLocalCalls = {} # the other side wants an answer from us def setTub(self, tub): assert ipb.ITub.providedBy(tub) self.tub = tub self.unsafeTracebacks = tub.unsafeTracebacks self._expose_remote_exception_types = tub._expose_remote_exception_types if tub.debugBanana: self.debugSend = True self.debugReceive = True def connectionMade(self): banana.Banana.connectionMade(self) self.rootSlicer.broker = self self.rootUnslicer.broker = self if self.use_remote_broker: self._create_remote_broker() def _create_remote_broker(self): # create the remote_broker object. We don't use the usual # reference-counting mechanism here, because this is a synthetic # object that lives forever. tracker = referenceable.RemoteReferenceTracker(self, 0, None, "RIBroker") self.remote_broker = referenceable.RemoteReference(tracker) # connectionTimedOut is called in response to the Banana layer detecting # the lack of connection activity def connectionTimedOut(self): err = error.ConnectionLost("banana timeout: connection dropped") why = failure.Failure(err) self.shutdown(why) def shutdown(self, why, fireDisconnectWatchers=True): """Stop using this connection. If fireDisconnectWatchers is False, all disconnect watchers are removed before shutdown, so they will not be called (this is appropriate when the Broker is shutting down because the whole Tub is being shut down). We terminate the connection quickly, rather than waiting for the transmit queue to drain. """ assert isinstance(why, failure.Failure) if not fireDisconnectWatchers: self.disconnectWatchers = [] self.finish(why) # loseConnection eventually provokes connectionLost() self.transport.loseConnection() def connectionLost(self, why): tubid = "?" if self.remote_tubref: tubid = self.remote_tubref.getShortTubID() log.msg("connection to %s lost" % tubid, facility="foolscap.connection") banana.Banana.connectionLost(self, why) self.finish(why) self._notifyConnectionLostWatchers() def _notifyConnectionLostWatchers(self): """ Call all functions waiting to learn about the loss of the connection of this broker. """ watchers = self._connectionLostWatchers self._connectionLostWatchers = None for w in watchers: eventually(w) def finish(self, why): if self.disconnected: return assert isinstance(why, failure.Failure), why self.disconnected = True self.remote_broker = None self.abandonAllRequests(why) # TODO: why reset all the tables to something useable? There may be # outstanding RemoteReferences that point to us, but I don't see why # that requires all these empty dictionaries. self.myReferenceByPUID = {} self.myReferenceByCLID = {} self.yourReferenceByCLID = {} self.yourReferenceByURL = {} self.myGifts = {} self.myGiftsByGiftID = {} for (cb,args,kwargs) in self.disconnectWatchers: eventually(cb, *args, **kwargs) self.disconnectWatchers = [] if self.tub: # TODO: remove the conditional. It is only here to accomodate # some tests: test_pb.TestCall.testDisconnect[123] self.tub.brokerDetached(self, why) def _notifyOnConnectionLost(self, callback): """ Arrange to have C{callback} called when this broker loses its connection. """ self._connectionLostWatchers.append(callback) def notifyOnDisconnect(self, callback, *args, **kwargs): marker = (callback, args, kwargs) if self.disconnected: eventually(callback, *args, **kwargs) else: self.disconnectWatchers.append(marker) return marker def dontNotifyOnDisconnect(self, marker): if self.disconnected: return # be tolerant of attempts to unregister a callback that has already # fired. I think it is hard to write safe code without this # tolerance. # TODO: on the other hand, I'm not sure this is the best policy, # since you lose the feedback that tells you about # unregistering-the-wrong-thing bugs. We need to look at the way that # register/unregister gets used and see if there is a way to retain # the typechecking that results from insisting that you can only # remove something that was stil in the list. if marker in self.disconnectWatchers: self.disconnectWatchers.remove(marker) def getConnectionInfo(self): return self._connectionInfo # methods to send my Referenceables to the other side def getTrackerForMyReference(self, puid, obj): tracker = self.myReferenceByPUID.get(puid) if not tracker: # need to add one clid = self.nextCLID() tracker = referenceable.ReferenceableTracker(self.tub, obj, puid, clid) self.myReferenceByPUID[puid] = tracker self.myReferenceByCLID[clid] = tracker return tracker def getTrackerForMyCall(self, puid, obj): # just like getTrackerForMyReference, but with a negative clid tracker = self.myReferenceByPUID.get(puid) if not tracker: # need to add one clid = self.nextCLID() clid = -clid tracker = referenceable.ReferenceableTracker(self.tub, obj, puid, clid) self.myReferenceByPUID[puid] = tracker self.myReferenceByCLID[clid] = tracker return tracker # methods to handle inbound 'my-reference' sequences def getTrackerForYourReference(self, clid, interfaceName=None, url=None): """The far end holds a Referenceable and has just sent us a reference to it (expressed as a small integer). If this is a new reference, they will give us an interface name too, and possibly a global URL for it. Obtain a RemoteReference object (creating it if necessary) to give to the local recipient. The sender remembers that we hold a reference to their object. When our RemoteReference goes away, we send a decref message to them, so they can possibly free their object. """ assert type(interfaceName) is str or interfaceName is None if url is not None: assert type(url) is str tracker = self.yourReferenceByCLID.get(clid) if not tracker: # TODO: translate interfaceNames to RemoteInterfaces if clid >= 0: trackerclass = referenceable.RemoteReferenceTracker else: trackerclass = referenceable.RemoteMethodReferenceTracker tracker = trackerclass(self, clid, url, interfaceName) self.yourReferenceByCLID[clid] = tracker if url: self.yourReferenceByURL[url] = tracker return tracker def freeYourReference(self, tracker, count): # this is called when the RemoteReference is deleted if not self.remote_broker: # tests do not set this up self.freeYourReferenceTracker(None, tracker) return try: rb = self.remote_broker # TODO: do we want callRemoteOnly here? is there a way we can # avoid wanting to know when the decref has completed? Only if we # send the interface list and URL on every occurrence of the # my-reference sequence. Either A) we use callRemote("decref") # and wait until the ack to free the tracker, or B) we use # callRemoteOnly("decref") and free the tracker right away. In # case B, the far end has no way to know that we've just freed # the tracker and will therefore forget about everything they # told us (including the interface list), so they cannot # accurately do anything special on the "first" send of this # reference. Which means that if we do B, we must either send # that extra information on every my-reference sequence, or do # without it, or make it optional, or retrieve it separately, or # something. # rb.callRemoteOnly("decref", clid=tracker.clid, count=count) # self.freeYourReferenceTracker('bogus', tracker) # return d = rb.callRemote("decref", clid=tracker.clid, count=count) # if the connection was lost before we can get an ack, we're # tearing this down anyway def _ignore_loss(f): f.trap(DeadReferenceError, *LOST_CONNECTION_ERRORS) return None d.addErrback(_ignore_loss) # once the ack comes back, or if we know we'll never get one, # release the tracker d.addCallback(self.freeYourReferenceTracker, tracker) except: f = failure.Failure() log.msg("failure during freeRemoteReference", facility="foolscap", level=log.UNUSUAL, failure=f) def freeYourReferenceTracker(self, res, tracker): if tracker.received_count != 0: return if self.yourReferenceByCLID.has_key(tracker.clid): del self.yourReferenceByCLID[tracker.clid] if tracker.url and self.yourReferenceByURL.has_key(tracker.url): del self.yourReferenceByURL[tracker.url] # methods to handle inbound 'your-reference' sequences def getMyReferenceByCLID(self, clid): """clid is the connection-local ID of the Referenceable the other end is trying to invoke or point to. If it is a number, they want an implicitly-created per-connection object that we sent to them at some point in the past. If it is a string, they want an object that was registered with our Factory. """ assert isinstance(clid, (int, long)) if clid == 0: return self return self.myReferenceByCLID[clid].obj # obj = IReferenceable(obj) # assert isinstance(obj, pb.Referenceable) # obj needs .getMethodSchema, which needs .getArgConstraint def remote_decref(self, clid, count): # invoked when the other side sends us a decref message assert isinstance(clid, (int, long)) assert clid != 0 tracker = self.myReferenceByCLID.get(clid, None) if not tracker: return # already gone, probably because we're shutting down done = tracker.decref(count) if done: del self.myReferenceByPUID[tracker.puid] del self.myReferenceByCLID[clid] # methods to send RemoteReference 'gifts' to third-parties def makeGift(self, rref): # return the giftid broker, clid = rref.tracker.broker, rref.tracker.clid i = (broker, clid) old = self.myGifts.get(i) if old: rref, giftID, count = old self.myGifts[i] = (rref, giftID, count+1) else: giftID = self.nextGiftID() self.myGiftsByGiftID[giftID] = i self.myGifts[i] = (rref, giftID, 1) return giftID def remote_decgift(self, giftID, count): broker, clid = self.myGiftsByGiftID[giftID] rref, giftID, gift_count = self.myGifts[(broker, clid)] gift_count -= count if gift_count == 0: del self.myGiftsByGiftID[giftID] del self.myGifts[(broker, clid)] else: self.myGifts[(broker, clid)] = (rref, giftID, gift_count) # methods to deal with URLs def getYourReferenceByName(self, name): d = self.remote_broker.callRemote("getReferenceByName", name=name) return d def remote_getReferenceByName(self, name): return self.tub.getReferenceForName(name) # remote-method-invocation methods, calling side, invoked by # RemoteReference.callRemote and CallSlicer def newRequestID(self): if self.disconnected: raise DeadReferenceError("Calling Stale Broker") return self.nextReqID() def addRequest(self, req): req.broker = self self.waitingForAnswers[req.reqID] = req def removeRequest(self, req): del self.waitingForAnswers[req.reqID] def getRequest(self, reqID): # invoked by AnswerUnslicer and ErrorUnslicer try: return self.waitingForAnswers[reqID] except KeyError: raise Violation("non-existent reqID '%d'" % reqID) def abandonAllRequests(self, why): for req in self.waitingForAnswers.values(): if why.check(*LOST_CONNECTION_ERRORS): # map all connection-lost errors to DeadReferenceError, so # application code only needs to check for one exception type tubid = None # since we're creating a new exception object for each call, # let's add more information to it if self.remote_tubref: tubid = self.remote_tubref.getShortTubID() e = DeadReferenceError("Connection was lost", tubid, req) why = failure.Failure(e) eventually(req.fail, why) # target-side, invoked by CallUnslicer def getRemoteInterfaceByName(self, riname): # this lives in the broker because it ought to be per-connection return remoteinterface.RemoteInterfaceRegistry[riname] def getSchemaForMethod(self, rifaces, methodname): # this lives in the Broker so it can override the resolution order, # not that overlapping RemoteInterfaces should be allowed to happen # all that often for ri in rifaces: m = ri.get(methodname) if m: return m return None def scheduleCall(self, delivery, ready_deferred): self.inboundDeliveryQueue.append( (delivery,ready_deferred) ) eventually(self.doNextCall) def doNextCall(self): if self.disconnected: return if self._waiting_for_call_to_be_ready: return if not self.inboundDeliveryQueue: return delivery, ready_deferred = self.inboundDeliveryQueue.pop(0) self._waiting_for_call_to_be_ready = True if not ready_deferred: ready_deferred = defer.succeed(None) d = ready_deferred def _ready(res): self._waiting_for_call_to_be_ready = False eventually(self.doNextCall) return res d.addBoth(_ready) # at this point, the Deferred chain for this one delivery runs # independently of any other, and methods which take a long time to # complete will not hold up other methods. We must call _doCall and # let the remote_ method get control before we process any other # message, but the eventually() above insures we'll have a chance to # do that before we give up control. d.addCallback(lambda res: self._doCall(delivery)) d.addCallback(self._callFinished, delivery) d.addErrback(self.callFailed, delivery.reqID, delivery) d.addErrback(log.err) return None def _doCall(self, delivery): # our ordering rules require that the order in which each # remote_foo() method gets control is exactly the same as the order # in which the original caller invoked callRemote(). To insure this, # _startCall() is not allowed to insert additional delays before it # runs doRemoteCall() on the target object. obj = delivery.obj args = delivery.allargs.args kwargs = delivery.allargs.kwargs for i in args + kwargs.values(): assert not isinstance(i, defer.Deferred) if delivery.methodSchema: # we asked about each argument on the way in, but ask again so # they can look for missing arguments. TODO: see if we can remove # the redundant per-argument checks. delivery.methodSchema.checkAllArgs(args, kwargs, True) # interesting case: if the method completes successfully, but # our schema prohibits us from sending the result (perhaps the # method returned an int but the schema insists upon a string). # TODO: move the return-value schema check into # Referenceable.doRemoteCall, so the exception's traceback will be # attached to the object that caused it if delivery.methodname is None: assert callable(obj) return obj(*args, **kwargs) else: obj = ipb.IRemotelyCallable(obj) return obj.doRemoteCall(delivery.methodname, args, kwargs) def _callFinished(self, res, delivery): reqID = delivery.reqID if reqID == 0: return methodSchema = delivery.methodSchema assert self.activeLocalCalls[reqID] methodName = None if methodSchema: methodName = methodSchema.name try: methodSchema.checkResults(res, False) # may raise Violation except Violation, v: v.prependLocation("in return value of %s.%s" % (delivery.obj, methodSchema.name)) raise answer = call.AnswerSlicer(reqID, res, methodName) # once the answer has started transmitting, any exceptions must be # logged and dropped, and not turned into an Error to be sent. try: self.send(answer) # TODO: .send should return a Deferred that fires when the last # byte has been queued, and we should delete the local note then except: f = failure.Failure() log.msg("Broker._callfinished unable to send", facility="foolscap", level=log.UNUSUAL, failure=f) del self.activeLocalCalls[reqID] def callFailed(self, f, reqID, delivery=None): # this may be called either when an inbound schema is violated, or # when the method is run and raises an exception. If a Violation is # raised after we receive the reqID but before we've actually invoked # the method, we are called by CallUnslicer.reportViolation and don't # get a delivery= argument. if delivery: if (self.tub and self.tub.logLocalFailures) or not self.tub: # the 'not self.tub' case is for unit tests delivery.logFailure(f) if reqID != 0: assert self.activeLocalCalls[reqID] self.send(call.ErrorSlicer(reqID, f)) del self.activeLocalCalls[reqID] class StorageBrokerRootSlicer(ScopedRootSlicer): # each StorageBroker is a single serialization domain, so we inherit from # ScopedRootSlicer slicerTable = {types.MethodType: referenceable.CallableSlicer, types.FunctionType: referenceable.CallableSlicer, } PBStorageOpenRegistry = { ('their-reference',): referenceable.TheirReferenceUnslicer, } class StorageBrokerRootUnslicer(PBRootUnslicer): # we want all the behavior of PBRootUnslicer, plus the scopedness of a # ScopedRootUnslicer. TODO: find some way to refactor all of this, # probably by making the scopedness a mixin. openRegistries = [slicer.UnslicerRegistry, PBStorageOpenRegistry] topRegistries = openRegistries def __init__(self, protocol): PBRootUnslicer.__init__(self, protocol) self.references = {} def setObject(self, counter, obj): self.references[counter] = obj def getObject(self, counter): obj = self.references.get(counter) return obj def receiveChild(self, obj, ready_deferred): self.protocol.receiveChild(obj, ready_deferred) def reportViolation(self, why): # unlike PBRootUnslicer, we do *not* absorb the failure. Any error # during deserialization is fatal to the process. We give it to the # StorageBroker, which will eventually fire the unserialization # Deferred. self.protocol.reportViolation(why) class StorageBroker(Broker): # like Broker, but used to serialize data for storage rather than for # transmission over a specific connection. slicerClass = StorageBrokerRootSlicer unslicerClass = StorageBrokerRootUnslicer object = None violation = None disconnectReason = None use_remote_broker = False def prepare(self): self.d = defer.Deferred() return self.d def receiveChild(self, obj, ready_deferred): if ready_deferred: ready_deferred.addBoth(self.d.callback) self.d.addCallback(lambda res: obj) else: self.d.callback(obj) del self.d def reportViolation(self, why): self.violation = why eventually(self.d.callback, None) return None def reportReceiveError(self, f): self.disconnectReason = f f.raiseException() # this loopback stuff is based upon twisted.protocols.loopback, except that # we use it for real, not just for testing. The IConsumer stuff hasn't been # tested at all. class LoopbackAddress(object): implements(twinterfaces.IAddress) class LoopbackTransport(object): # we always create these in pairs, with .peer pointing at each other implements(twinterfaces.ITransport, twinterfaces.IConsumer) producer = None def __init__(self): self.connected = True def setPeer(self, peer): self.peer = peer def write(self, bytes): eventually(self.peer.dataReceived, bytes) def writeSequence(self, iovec): self.write(''.join(iovec)) def dataReceived(self, data): if self.connected: self.protocol.dataReceived(data) def loseConnection(self, _connDone=connectionDone): if not self.connected: return self.connected = False eventually(self.peer.connectionLost, _connDone) eventually(self.protocol.connectionLost, _connDone) def connectionLost(self, reason): if not self.connected: return self.connected = False self.protocol.connectionLost(reason) def getPeer(self): return LoopbackAddress() def getHost(self): return LoopbackAddress() # IConsumer def registerProducer(self, producer, streaming): assert self.producer is None self.producer = producer self.streamingProducer = streaming self._pollProducer() def unregisterProducer(self): assert self.producer is not None self.producer = None def _pollProducer(self): if self.producer is not None and not self.streamingProducer: self.producer.resumeProducing() foolscap-0.13.1/src/foolscap/call.py0000644000076500000240000010725212766553111017723 0ustar warnerstaff00000000000000 from twisted.python import failure, reflect, log as twlog from twisted.internet import defer from foolscap import copyable, slicer, tokens from foolscap.copyable import AttributeDictConstraint from foolscap.constraint import ByteStringConstraint from foolscap.slicers.list import ListConstraint from tokens import BananaError, Violation from foolscap.util import AsyncAND from foolscap.logging import log def wrap_remote_failure(f): return failure.Failure(tokens.RemoteException(f)) class FailureConstraint(AttributeDictConstraint): opentypes = [("copyable", "twisted.python.failure.Failure")] name = "FailureConstraint" klass = failure.Failure def __init__(self): attrs = [('type', ByteStringConstraint(200)), ('value', ByteStringConstraint(1000)), ('traceback', ByteStringConstraint(2000)), ('parents', ListConstraint(ByteStringConstraint(200))), ] AttributeDictConstraint.__init__(self, *attrs) def checkObject(self, obj, inbound): if not isinstance(obj, self.klass): raise Violation("is not an instance of %s" % self.klass) class PendingRequest(object): # this object is a local representation of a message we have sent to # someone else, that will be executed on their end. active = True def __init__(self, reqID, rref, interface_name, method_name): self.reqID = reqID self.rref = rref # keep it alive self.broker = None # if set, the broker knows about us self.deferred = defer.Deferred() self.constraint = None # this constrains the results self.failure = None self.interface_name = interface_name # for error messages self.method_name = method_name # same def setConstraint(self, constraint): self.constraint = constraint def getMethodNameInfo(self): return (self.interface_name, self.method_name) def complete(self, res): if self.broker: self.broker.removeRequest(self) if self.active: self.active = False self.deferred.callback(res) else: log.msg("PendingRequest.complete called on an inactive request") def fail(self, why): if self.active: if self.broker: self.broker.removeRequest(self) self.active = False self.failure = why if (self.broker and self.broker.tub and self.broker.tub.logRemoteFailures): my_short_tubid = "??" if self.broker.tub: # for tests my_short_tubid = self.broker.tub.getShortTubID() their_short_tubid = self.broker.remote_tubref.getShortTubID() lp = log.msg("an outbound callRemote (that we [%s] sent to " "someone else [%s]) failed on the far end" % (my_short_tubid, their_short_tubid), level=log.UNUSUAL) methname = ".".join([self.interfaceName or "?", self.methodName or "?"]) log.msg(" reqID=%d, rref=%s, methname=%s" % (self.reqID, self.rref, methname), level=log.NOISY, parent=lp) #stack = why.getTraceback() # TODO: include the first few letters of the remote tubID in # this REMOTE tag #stack = "REMOTE: " + stack.replace("\n", "\nREMOTE: ") log.msg(" the REMOTE failure was:", failure=why, level=log.NOISY, parent=lp) #log.msg(stack, level=log.NOISY, parent=lp) self.deferred.errback(why) else: log.msg("WEIRD: fail() on an inactive request", traceback=True) if self.failure: log.msg("multiple failures") log.msg("first one was:", self.failure) log.msg("this one was:", why) log.err("multiple failures indicate a problem") class ArgumentSlicer(slicer.ScopedSlicer): opentype = ('arguments',) def __init__(self, args, kwargs, methodname="?"): slicer.ScopedSlicer.__init__(self, None) self.args = args self.kwargs = kwargs self.which = "" self.methodname = methodname def sliceBody(self, streamable, banana): yield len(self.args) for i,arg in enumerate(self.args): self.which = "arg[%d]-of-%s" % (i, self.methodname) yield arg keys = self.kwargs.keys() keys.sort() for argname in keys: self.which = "arg[%s]-of-%s" % (argname, self.methodname) yield argname yield self.kwargs[argname] def describe(self): return "<%s>" % self.which class CallSlicer(slicer.ScopedSlicer): opentype = ('call',) def __init__(self, reqID, clid, methodname, args, kwargs): slicer.ScopedSlicer.__init__(self, None) self.reqID = reqID self.clid = clid self.methodname = methodname self.args = args self.kwargs = kwargs def sliceBody(self, streamable, banana): yield self.reqID yield self.clid yield self.methodname yield ArgumentSlicer(self.args, self.kwargs, self.methodname) def describe(self): return "" % (self.reqID, self.clid, self.methodname) class InboundDelivery(object): """An inbound message that has not yet been delivered. This is created when a 'call' sequence has finished being received. The Broker will add it to a queue. The delivery at the head of the queue is serviced when all of its arguments have been resolved. The only way that the arguments might not all be available is if one of the Unslicers which created them has provided a 'ready_deferred' along with the prospective object. The only standard Unslicer which does this is the TheirReferenceUnslicer, which handles introductions. (custom Unslicers might also provide a ready_deferred, for example a URL slicer/unslicer pair for which the receiving end fetches the target of the URL as its value, or a UnixFD slicer/unslicer that had to wait for a side-channel unix-domain socket to finish transferring control over the FD to the recipient before being ready). Most Unslicers refuse to accept unready objects as their children (most implementations of receiveChild() do 'assert ready_deferred is None'). The CallUnslicer is fairly unique in not rejecting such objects. We do require, however, that all of the arguments be at least referenceable. This is not generally a problem: the only time an unslicer's receiveChild() can get a non-referenceable object (represented by a Deferred) is if that unslicer is participating in a reference cycle that has not yet completed, and CallUnslicers only live at the top level, above any cycles. """ def __init__(self, broker, reqID, obj, interface, methodname, methodSchema, allargs): self.broker = broker self.reqID = reqID self.obj = obj self.interface = interface self.methodname = methodname self.methodSchema = methodSchema self.allargs = allargs def logFailure(self, f): # called if tub.logLocalFailures is True my_short_tubid = "??" if self.broker.tub: # for tests my_short_tubid = self.broker.tub.getShortTubID() their_short_tubid = "" if self.broker.remote_tubref: their_short_tubid = self.broker.remote_tubref.getShortTubID() lp = log.msg("an inbound callRemote that we [%s] executed (on behalf " "of someone else, TubID %s) failed" % (my_short_tubid, their_short_tubid), level=log.UNUSUAL) if self.interface: methname = self.interface.getName() + "." + self.methodname else: methname = self.methodname log.msg(" reqID=%d, rref=%s, methname=%s" % (self.reqID, self.obj, methname), level=log.NOISY, parent=lp) log.msg(" args=%s" % (self.allargs.args,), level=log.NOISY, parent=lp) log.msg(" kwargs=%s" % (self.allargs.kwargs,), level=log.NOISY, parent=lp) #if isinstance(f.type, str): # stack = "getTraceback() not available for string exceptions\n" #else: # stack = f.getTraceback() # TODO: trim stack to everything below Broker._doCall #stack = "LOCAL: " + stack.replace("\n", "\nLOCAL: ") log.msg(" the LOCAL failure was:", failure=f, level=log.NOISY, parent=lp) #log.msg(stack, level=log.NOISY, parent=lp) class ArgumentUnslicer(slicer.ScopedUnslicer): methodSchema = None debug = False def setConstraint(self, methodSchema): self.methodSchema = methodSchema def start(self, count): if self.debug: log.msg("%s.start: %s" % (self, count)) self.numargs = None self.args = [] self.kwargs = {} self.argname = None self.argConstraint = None self.num_unreferenceable_children = 0 self._all_children_are_referenceable_d = None self._ready_deferreds = [] self.closed = False def checkToken(self, typebyte, size): if self.numargs is None: # waiting for positional-arg count if typebyte != tokens.INT: raise BananaError("posarg count must be an INT") return if len(self.args) < self.numargs: # waiting for a positional arg if self.argConstraint: self.argConstraint.checkToken(typebyte, size) return if self.argname is None: # waiting for the name of a keyword arg if typebyte not in (tokens.STRING, tokens.VOCAB): raise BananaError("kwarg name must be a STRING") # TODO: limit to longest argument name of the method? return # waiting for the value of a kwarg if self.argConstraint: self.argConstraint.checkToken(typebyte, size) def doOpen(self, opentype): if self.argConstraint: self.argConstraint.checkOpentype(opentype) unslicer = self.open(opentype) if unslicer: if self.argConstraint: unslicer.setConstraint(self.argConstraint) return unslicer def receiveChild(self, token, ready_deferred=None): if self.debug: log.msg("%s.receiveChild: %s %s %s %s %s args=%s kwargs=%s" % (self, self.closed, self.num_unreferenceable_children, len(self._ready_deferreds), token, ready_deferred, self.args, self.kwargs)) if self.numargs is None: # this token is the number of positional arguments assert isinstance(token, int) assert ready_deferred is None self.numargs = token if self.numargs: ms = self.methodSchema if ms: accept, self.argConstraint = \ ms.getPositionalArgConstraint(0) assert accept return if len(self.args) < self.numargs: # this token is a positional argument argvalue = token argpos = len(self.args) self.args.append(argvalue) if isinstance(argvalue, defer.Deferred): # this may occur if the child is a gift which has not # resolved yet. self.num_unreferenceable_children += 1 argvalue.addCallback(self.updateChild, argpos) if ready_deferred: if self.debug: log.msg("%s.receiveChild got an unready posarg" % self) self._ready_deferreds.append(ready_deferred) if len(self.args) < self.numargs: # more to come ms = self.methodSchema if ms: nextargnum = len(self.args) accept, self.argConstraint = \ ms.getPositionalArgConstraint(nextargnum) assert accept return if self.argname is None: # this token is the name of a keyword argument assert ready_deferred is None self.argname = token # if the argname is invalid, this may raise Violation ms = self.methodSchema if ms: accept, self.argConstraint = \ ms.getKeywordArgConstraint(self.argname, self.numargs, self.kwargs.keys()) assert accept return # this token is the value of a keyword argument argvalue = token self.kwargs[self.argname] = argvalue if isinstance(argvalue, defer.Deferred): self.num_unreferenceable_children += 1 argvalue.addCallback(self.updateChild, self.argname) if ready_deferred: if self.debug: log.msg("%s.receiveChild got an unready kwarg" % self) self._ready_deferreds.append(ready_deferred) self.argname = None return def updateChild(self, obj, which): # one of our arguments has just now become referenceable. Normal # types can't trigger this (since the arguments to a method form a # top-level serialization domain), but special Unslicers might. For # example, the Gift unslicer will eventually provide us with a # RemoteReference, but for now all we get is a Deferred as a # placeholder. if self.debug: log.msg("%s.updateChild, [%s] became referenceable: %s" % (self, which, obj)) if isinstance(which, int): self.args[which] = obj else: self.kwargs[which] = obj self.num_unreferenceable_children -= 1 if self.num_unreferenceable_children == 0: if self._all_children_are_referenceable_d: self._all_children_are_referenceable_d.callback(None) return obj def receiveClose(self): if self.debug: log.msg("%s.receiveClose: %s %s %s" % (self, self.closed, self.num_unreferenceable_children, len(self._ready_deferreds))) if (self.numargs is None or len(self.args) < self.numargs or self.argname is not None): raise BananaError("'arguments' sequence ended too early") self.closed = True dl = [] if self.num_unreferenceable_children: d = self._all_children_are_referenceable_d = defer.Deferred() dl.append(d) dl.extend(self._ready_deferreds) ready_deferred = None if dl: ready_deferred = AsyncAND(dl) return self, ready_deferred def describe(self): s = " 0: self.broker.callFailed(f, self.reqID) return f # give up our sequence def receiveChild(self, token, ready_deferred=None): assert not isinstance(token, defer.Deferred) if self.debug: log.msg("%s.receiveChild [s%d]: %s" % (self, self.stage, repr(token))) if self.stage == 0: # reqID # we don't yet know which reqID to send any failure to assert ready_deferred is None self.reqID = token self.stage = 1 if self.reqID != 0: assert self.reqID not in self.broker.activeLocalCalls self.broker.activeLocalCalls[self.reqID] = self return if self.stage == 1: # objID # this might raise an exception if objID is invalid assert ready_deferred is None self.objID = token try: self.obj = self.broker.getMyReferenceByCLID(token) except KeyError: raise Violation("unknown CLID %d" % (token,)) #iface = self.broker.getRemoteInterfaceByName(token) if self.objID < 0: self.interface = None else: self.interface = self.obj.getInterface() self.stage = 2 return if self.stage == 2: # methodname # validate the methodname, get the schema. This may raise an # exception for unknown methods # must find the schema, using the interfaces # TODO: getSchema should probably be in an adapter instead of in # a pb.Referenceable base class. Old-style (unconstrained) # flavors.Referenceable should be adapted to something which # always returns None # TODO: make this faster. A likely optimization is to take a # tuple of components.getInterfaces(obj) and use it as a cache # key. It would be even faster to use obj.__class__, but that # would probably violate the expectation that instances can # define their own __implements__ (independently from their # class). If this expectation were to go away, a quick # obj.__class__ -> RemoteReferenceSchema cache could be built. assert ready_deferred is None self.stage = 3 if self.objID < 0: # the target is a bound method, ignore the methodname self.methodSchema = getattr(self.obj, "methodSchema", None) self.methodname = None # TODO: give it something useful if self.broker.requireSchema and not self.methodSchema: why = "This broker does not accept unconstrained " + \ "method calls" raise Violation(why) return self.methodname = token if self.interface: # they are calling an interface+method pair ms = self.interface.get(self.methodname) if not ms: why = "method '%s' not defined in %s" % \ (self.methodname, self.interface.__remote_name__) raise Violation(why) self.methodSchema = ms return if self.stage == 3: # arguments assert isinstance(token, ArgumentUnslicer) self.allargs = token # queue the message. It will not be executed until all the # arguments are ready. The .args list and .kwargs dict may change # before then. if ready_deferred: self._ready_deferreds.append(ready_deferred) self.stage = 4 return def receiveClose(self): if self.stage != 4: raise BananaError("'call' sequence ended too early") # time to create the InboundDelivery object so we can queue it delivery = InboundDelivery(self.broker, self.reqID, self.obj, self.interface, self.methodname, self.methodSchema, self.allargs) ready_deferred = None if self._ready_deferreds: ready_deferred = AsyncAND(self._ready_deferreds) return delivery, ready_deferred def describe(self): s = "= 1: s += " reqID=%d" % self.reqID if self.stage >= 2: s += " obj=%s" % (self.obj,) ifacename = "[none]" if self.interface: ifacename = self.interface.__remote_name__ s += " iface=%s" % ifacename if self.stage >= 3: s += " methodname=%s" % self.methodname s += ">" return s class AnswerSlicer(slicer.ScopedSlicer): opentype = ('answer',) def __init__(self, reqID, results, methodname="?"): assert reqID != 0 slicer.ScopedSlicer.__init__(self, None) self.reqID = reqID self.results = results self.methodname = methodname def sliceBody(self, streamable, banana): yield self.reqID yield self.results def describe(self): return "" % (self.reqID, self.methodname) class AnswerUnslicer(slicer.ScopedUnslicer): request = None resultConstraint = None haveResults = False def start(self, count): slicer.ScopedUnslicer.start(self, count) self._ready_deferreds = [] self._child_deferred = None def checkToken(self, typebyte, size): if self.request is None: if typebyte != tokens.INT: raise BananaError("request ID must be an INT") elif not self.haveResults: if self.resultConstraint: try: self.resultConstraint.checkToken(typebyte, size) except Violation, v: # improve the error message if v.args: # this += gives me a TypeError "object doesn't # support item assignment", which confuses me #v.args[0] += " in inbound method results" why = v.args[0] + " in inbound method results" v.args = why, else: v.args = ("in inbound method results",) raise # this will errback the request else: raise BananaError("stop sending me stuff!") def doOpen(self, opentype): if self.resultConstraint: self.resultConstraint.checkOpentype(opentype) # TODO: improve the error message unslicer = self.open(opentype) if unslicer: if self.resultConstraint: unslicer.setConstraint(self.resultConstraint) return unslicer def receiveChild(self, token, ready_deferred=None): if self.request == None: assert not isinstance(token, defer.Deferred) assert ready_deferred is None reqID = token # may raise Violation for bad reqIDs self.request = self.broker.getRequest(reqID) self.resultConstraint = self.request.constraint else: if isinstance(token, defer.Deferred): self._child_deferred = token else: self._child_deferred = defer.succeed(token) if ready_deferred: self._ready_deferreds.append(ready_deferred) self.haveResults = True def reportViolation(self, f): # if the Violation was received after we got the reqID, we can tell # the broker it was an error if self.request != None: self.request.fail(f) # local violation return f # give up our sequence def receiveClose(self): # three things must happen before our request is complete: # receiveClose has occurred # the receiveChild object deferred (if any) has fired # ready_deferred has finished # If ready_deferred errbacks, provide its failure object to the # request. If not, provide the request with whatever receiveChild # got. if not self._child_deferred: raise BananaError("Answer didn't include an answer") if self._ready_deferreds: d = AsyncAND(self._ready_deferreds) else: d = defer.succeed(None) def _ready(res): return self._child_deferred d.addCallback(_ready) def _done(res): self.request.complete(res) def _fail(f): # we hit here if any of the _ready_deferreds fail (i.e a Gift # failed to resolve), or if the _child_deferred fails (not sure # how this could happen). I think it's ok to return a local # exception (instead of a RemoteException) for both. self.request.fail(f) d.addCallbacks(_done, _fail) return None, None def describe(self): if self.request: return "Answer(req=%s)" % self.request.reqID return "Answer(req=?)" class ErrorSlicer(slicer.ScopedSlicer): opentype = ('error',) def __init__(self, reqID, f): slicer.ScopedSlicer.__init__(self, None) assert isinstance(f, failure.Failure) self.reqID = reqID self.f = f def sliceBody(self, streamable, banana): yield self.reqID yield self.f def describe(self): return "" % self.reqID class ErrorUnslicer(slicer.ScopedUnslicer): request = None fConstraint = FailureConstraint() gotFailure = False def checkToken(self, typebyte, size): if self.request == None: if typebyte != tokens.INT: raise BananaError("request ID must be an INT") elif not self.gotFailure: self.fConstraint.checkToken(typebyte, size) else: raise BananaError("stop sending me stuff!") def doOpen(self, opentype): self.fConstraint.checkOpentype(opentype) unslicer = self.open(opentype) if unslicer: unslicer.setConstraint(self.fConstraint) return unslicer def reportViolation(self, f): # a failure while receiving the failure. A bit daft, really. if self.request != None: self.request.fail(f) return f # give up our sequence def receiveChild(self, token, ready_deferred=None): assert not isinstance(token, defer.Deferred) assert ready_deferred is None if self.request == None: reqID = token # may raise BananaError for bad reqIDs self.request = self.broker.getRequest(reqID) else: self.failure = token self.gotFailure = True def receiveClose(self): f = self.failure if not self.broker._expose_remote_exception_types: f = wrap_remote_failure(f) self.request.fail(f) return None, None def describe(self): if self.request is None: return "" return "" % self.request.reqID def truncate(s, limit): assert limit > 3 if s and len(s) > limit: s = s[:limit-3] + ".." return s # failures are sent as Copyables class FailureSlicer(slicer.BaseSlicer): slices = failure.Failure classname = "twisted.python.failure.Failure" def slice(self, streamable, banana): self.streamable = streamable yield 'copyable' yield self.classname state = self.getStateToCopy(self.obj, banana) for k,v in state.iteritems(): yield k yield v def describe(self): return "<%s>" % self.classname def getStateToCopy(self, obj, broker): #state = obj.__dict__.copy() #state['tb'] = None #state['frames'] = [] #state['stack'] = [] state = {} # string exceptions show up as obj.value == None and # isinstance(obj.type, str). Normal exceptions show up as obj.value # == text and obj.type == exception class. We need to make sure we # can handle both. if isinstance(obj.value, failure.Failure): # TODO: how can this happen? I got rid of failure2Copyable, so # if this case is possible, something needs to replace it raise RuntimeError("not implemented yet") #state['value'] = failure2Copyable(obj.value, banana.unsafeTracebacks) elif isinstance(obj.type, str): state['value'] = str(obj.value) state['type'] = obj.type # a string else: state['value'] = str(obj.value) # Exception instance state['type'] = reflect.qual(obj.type) # Exception class # TODO: I suspect that f.value may be getting a copy of the # traceback, because I've seen it be 1819 bytes at one point. I had # assumed that it was just the exception name plus args: whatever # Exception.__repr__ returns. state['value'] = truncate(state['value'], 1000) state['type'] = truncate(state['type'], 200) if broker.unsafeTracebacks: if isinstance(obj.type, str): stack = "getTraceback() not available for string exceptions\n" else: stack = obj.getTraceback() state['traceback'] = stack # TODO: provide something with globals and locals and HTML and # all that cool stuff else: state['traceback'] = 'Traceback unavailable\n' # The last few lines are often the most interesting. If we need to # truncate this, grab the first few lines and then as much of the # tail as we can get. if len(state['traceback']) > 1900: state['traceback'] = (state['traceback'][:700] + "\n\n-- TRACEBACK ELIDED --\n\n" + state['traceback'][-1200:]) parents = obj.parents[:] if parents: for i,value in enumerate(parents): parents[i] = truncate(value, 200) state['parents'] = parents return state class CopiedFailure(failure.Failure, copyable.RemoteCopyOldStyle): # this is a RemoteCopyOldStyle because you can't raise new-style # instances as exceptions. """I am a shadow of some remote Failure instance. I contain less information than the original did. You can still extract a (brief) printable traceback from me. My .parents attribute is a list of strings describing the class of the exception that I contain, just like the real Failure had, so my trap() and check() methods work fine. My .type and .value attributes are string representations of the original exception class and exception instance, respectively. The most significant effect is that you cannot access f.value.args, and should instead just use f.value . My .frames and .stack attributes are empty, although this may change in the future (and with the cooperation of the sender). """ nonCyclic = True stateSchema = FailureConstraint() def __init__(self): copyable.RemoteCopyOldStyle.__init__(self) def __getstate__(self): s = failure.Failure.__getstate__(self) # the ExceptionLikeString we use in self.type is not pickleable, so # replace it with the same sort of string that we use in the wire # protocol. if not isinstance(self.type, str): s['type'] = reflect.qual(self.type) return s def __setstate__(self, state): self.setCopyableState(state) def setCopyableState(self, state): #self.__dict__.update(state) self.__dict__ = state # state includes: type, value, traceback, parents #self.type = state['type'] #self.value = state['value'] #self.traceback = state['traceback'] #self.parents = state['parents'] self.tb = None self.frames = [] self.stack = [] # MAYBE: for native exception types, be willing to wire up a # reference to the real exception class. For other exception types, # our .type attribute will be a string, which (from a Failure's point # of view) looks as if someone raised an old-style string exception. # This is here so that trial will properly render a CopiedFailure # that comes out of a test case (since it unconditionally does # reflect.qual(f.type) # ACTUALLY: replace self.type with a class that looks a lot like the # original exception class (meaning that reflect.qual() will return # the same string for this as for the original). If someone calls our # .trap method, resulting in a new Failure with contents copied from # this one, then the new Failure.printTraceback will attempt to use # reflect.qual() on our self.type, so it needs to be a class instead # of a string. assert isinstance(self.type, str) typepieces = self.type.split(".") class ExceptionLikeString: pass self.type = ExceptionLikeString self.type.__module__ = ".".join(typepieces[:-1]) self.type.__name__ = typepieces[-1] def __str__(self): return "[CopiedFailure instance: %s]" % self.getBriefTraceback() pickled = 1 def printTraceback(self, file=None, elideFrameworkCode=0, detail='default'): if file is None: file = twlog.logerr file.write("Traceback from remote host -- ") file.write(self.traceback) copyable.registerRemoteCopy(FailureSlicer.classname, CopiedFailure) class CopiedFailureSlicer(FailureSlicer): # A calls B. B calls C. C fails and sends a Failure to B. B gets a # CopiedFailure and sends it to A. A should get a CopiedFailure too. This # class lives on B and slices the CopiedFailure as it is sent to A. slices = CopiedFailure def getStateToCopy(self, obj, broker): state = {} for k in ('value', 'type', 'parents'): state[k] = getattr(obj, k) if broker.unsafeTracebacks: state['traceback'] = obj.traceback else: state['traceback'] = "Traceback unavailable\n" if not isinstance(state['type'], str): state['type'] = reflect.qual(state['type']) # Exception class return state foolscap-0.13.1/src/foolscap/connection.py0000644000076500000240000003744413204160675021151 0ustar warnerstaff00000000000000import time from twisted.python.failure import Failure from twisted.internet import protocol, reactor, error, defer from foolscap.tokens import (NoLocationHintsError, NegotiationError, RemoteNegotiationError) from foolscap.info import ConnectionInfo from foolscap.logging import log from foolscap.logging.log import CURIOUS, UNUSUAL, OPERATIONAL from foolscap.util import isSubstring from foolscap.ipb import InvalidHintError from foolscap.connections.tcp import convert_legacy_hint class TubConnectorFactory(protocol.Factory, object): # this is for internal use only. Application code should use # Tub.getReference(url) noisy = False def __init__(self, tc, host, location, logparent): self.tc = tc # the TubConnector self.host = host self.location = location self._logparent = logparent def __repr__(self): # make it clear which remote Tub we're trying to connect to base = object.__repr__(self) at = base.find(" at ") if at == -1: # our annotation isn't really important, so don't fail just # because we guessed the default __repr__ incorrectly return base assert self.tc.tub.tubID origin = self.tc.tub.tubID[:8] assert self.tc.target.getTubID() target = self.tc.target.getTubID()[:8] return base[:at] + " [from %s]" % origin + " [to %s]" % target + base[at:] def buildProtocol(self, addr): nc = self.tc.tub.negotiationClass # this is usually Negotiation proto = nc(self._logparent) proto.initClient(self.tc, self.host, self.tc._connectionInfo) proto.factory = self return proto def describe_handler(h): try: return h.describe() except AttributeError: return repr(h) def get_endpoint(location, connectionPlugins, connectionInfo): def _update_status(status): connectionInfo._set_connection_status(location, status) def _try(): hint = convert_legacy_hint(location) if ":" not in hint: raise InvalidHintError("no colon") hint_type = hint.split(":", 1)[0] plugin = connectionPlugins.get(hint_type) if not plugin: connectionInfo._describe_connection_handler(location, None) raise InvalidHintError("no handler registered") connectionInfo._describe_connection_handler(location, describe_handler(plugin)) _update_status("resolving hint") return plugin.hint_to_endpoint(hint, reactor, _update_status) return defer.maybeDeferred(_try) class TubConnector(object): """I am used to make an outbound connection. I am given a target TubID and a list of locationHints, and I try all of them until I establish a Broker connected to the target. I will consider redirections returned along the way. The first hint that yields a connected Broker will stop the search. This is a single-use object. The connection attempt begins as soon as my connect() method is called. I live until all but one of the TCP connections I initiated have finished closing down. This means that connection establishment attempts in progress are cancelled, and established connections (the ones which did *not* complete negotiation before the winning connection) have called their connectionLost() methods. """ failureReason = None CONNECTION_TIMEOUT = 60 timer = None def __init__(self, parent, tubref, connectionPlugins): self._logparent = log.msg(format="TubConnector created from " "%(fromtubid)s to %(totubid)s", fromtubid=parent.tubID, totubid=tubref.getTubID(), level=OPERATIONAL, facility="foolscap.connection", umid="pH4QDA") self.tub = parent self.target = tubref self.connectionPlugins = connectionPlugins self._connectionInfo = ConnectionInfo() self.remainingLocations = list(self.target.getLocations()) # attemptedLocations keeps track of where we've already tried to # connect, so we don't try them twice, even if they appear in the # hints multiple times. this isn't too clever: slight variations of # the same hint will fool it, but it should be enough to avoid # infinite redirection loops. self.attemptedLocations = [] # validHints tracks which hints were successfully turned into # endpoints. If we don't recognize any hint type in a FURL, # validHints will be empty when we're done, and we'll signal # NoLocationHintsError self.validHints = [] # pendingConnections contains a Deferred for each endpoint.connect() # that has started (but not yet established) a connection. We keep # track of these so we can shut them down (using d.cancel()) when we # stop connecting (either because one of the other connections # succeeded, or because someone told us to give up). self.pendingConnections = set() # self.pendingNegotiations maps Negotiation instances (connected but # not finished negotiation) to the hint that got us the connection. # We track these so we can abandon the negotiation. self.pendingNegotiations = {} def __repr__(self): s = object.__repr__(self) s = s[:-1] s += " from %s to %s>" % (self.tub.tubID, self.target.getTubID()) return s def log(self, *args, **kwargs): kwargs['parent'] = kwargs.get('parent') or self._logparent kwargs['facility'] = kwargs.get('facility') or "foolscap.connection" return log.msg(*args, **kwargs) def getConnectionInfo(self): return self._connectionInfo def connect(self): """Begin the connection process. This should only be called once. This will either result in the successful Negotiation object invoking the parent Tub's brokerAttached() method, or us calling the Tub's connectionFailed() method.""" self.tub.connectorStarted(self) timeout = self.tub._test_options.get('connect_timeout', self.CONNECTION_TIMEOUT) self.timer = reactor.callLater(timeout, self.connectionTimedOut) self.active = True self.connectToAll() def stopConnectionTimer(self): if self.timer: self.timer.cancel() del self.timer def shutdown(self): self.active = False self.remainingLocations = [] self.stopConnectionTimer() self.cancelRemainingConnections() def cancelRemainingConnections(self): for d in list(self.pendingConnections): d.cancel() # this will trigger self._connectionFailed(), via the errback, # with a ConnectingCancelledError for n in self.pendingNegotiations.keys(): n.transport.loseConnection() # triggers n.connectionLost(), then self.connectorNegotiationFailed() def connectToAll(self): while self.remainingLocations: location = self.remainingLocations.pop() if location in self.attemptedLocations: continue self.attemptedLocations.append(location) lp = self.log("considering hint: %s" % (location,)) d = get_endpoint(location, self.connectionPlugins, self._connectionInfo) # no handler for this hint?: InvalidHintError thrown here def _good_hint(res, location=location): self._connectionInfo._set_connection_status(location, "connecting") self.validHints.append(location) (ep, host) = res self.log("connecting to hint: %s" % (location,), parent=lp, umid="9iX0eg") return ep.connect(TubConnectorFactory(self, host, location, lp)) d.addCallback(_good_hint) self.pendingConnections.add(d) def _remove(res, d=d): self.pendingConnections.remove(d) return res d.addBoth(_remove) d.addCallback(self._connectionSuccess, location, lp) d.addErrback(self._connectionFailed, location, lp) if self.tub._test_options.get("debug_stall_second_connection"): # for unit tests, hold off on making the second connection # for a moment. This allows the first connection to get to a # known state. reactor.callLater(0.1, self.connectToAll) return self.checkForFailure() def connectionTimedOut(self): # this timer is for the overall connection attempt, not each # individual endpoint/TCP connector self.timer = None why = "no connection established within client timeout" self.failureReason = Failure(NegotiationError(why)) self.shutdown() self.failed() def _connectionFailed(self, reason, hint, lp): # this is called if some individual TCP connection cannot be # established if reason.check(error.ConnectionRefusedError): description = "connection refused" self.log("connection refused for %s" % hint, level=OPERATIONAL, parent=lp, umid="rSrUxQ") elif reason.check(error.ConnectingCancelledError, defer.CancelledError): description = "abandoned" self.log("abandoned attempt to %s" % hint, level=OPERATIONAL, parent=lp, umid="CC8vwg") elif reason.check(InvalidHintError): description = "bad hint: %s" % str(reason.value) self.log("unable to use hint: %s: %s" % (hint, reason.value), level=UNUSUAL, parent=lp, umid="z62ctA") else: # some errors, like txsocksx.errors.ServerFailure, extend # Exception without defining a __str__, so when one is # constructed without arguments, their str() is empty, which is # not very useful. Their repr() at least includes the exception # name. In general, str() is better than repr(), since it lets # the exception designer build a human-meaningful string, so # we'll prefer str() unless it's empty. why = str(reason.value) or repr(reason.value) description = "failed to connect: %s" % why log.err(reason, "failed to connect to %s" % hint, level=CURIOUS, parent=lp, facility="foolscap.connection", umid="2PEowg") suffix = getattr(reason.value, "foolscap_connection_handler_error_suffix", None) if suffix: description += suffix self._connectionInfo._set_connection_status(hint, description) if not self.failureReason: self.failureReason = reason self.checkForFailure() self.checkForIdle() def _connectionSuccess(self, p, hint, lp): # fires with the Negotiation protocol instance, after # p.makeConnection(transport) returns, which is after # p.connectionMade() returns self.log("connected to %s, beginning negotiation" % hint, level=OPERATIONAL, parent=lp, umid="VN0XGQ") self.pendingNegotiations[p] = hint self._connectionInfo._set_connection_status(hint, "negotiating") def redirectReceived(self, newLocation): # the redirected connection will disconnect soon, which will trigger # connectorNegotiationFailed(), so we don't have to do a self.remainingLocations.append(newLocation) self.connectToAll() def connectorNegotiationFailed(self, n, location, reason): assert isinstance(n, self.tub.negotiationClass) # this is called if protocol negotiation cannot be established, or if # the connection is closed for any reason prior to switching to the # Banana protocol # abandoned connections will not have hit _connectionSuccess, so they # won't have been added to pendingNegotiations self.pendingNegotiations.pop(n, None) description = "negotiation failed: %s" % str(reason.value) self._connectionInfo._set_connection_status(location, description) assert isinstance(reason, Failure), \ "Hey, %s isn't a Failure" % (reason,) if (not self.failureReason or isinstance(reason, NegotiationError)): # don't let mundane things like ConnectionFailed override the # actually significant ones like NegotiationError self.failureReason = reason self.checkForFailure() self.checkForIdle() def connectorNegotiationComplete(self, n, location): assert isinstance(n, self.tub.negotiationClass) # 'factory' has just completed negotiation, so abandon all the other # connection attempts self.log("connectorNegotiationComplete, %s won" % n) self.pendingNegotiations.pop(n, None) # this one succeeded self._connectionInfo._set_connection_status(location, "successful") self._connectionInfo._set_winning_hint(location) self._connectionInfo._set_established_at(time.time()) self.active = False if self.timer: self.timer.cancel() self.timer = None self.cancelRemainingConnections() # abandon the others self.checkForIdle() def checkForFailure(self): if not self.active: return if (self.remainingLocations or self.pendingConnections or self.pendingNegotiations): return if not self.validHints: self.failureReason = Failure(NoLocationHintsError()) # we have no more options, so the connection attempt will fail. The # getBrokerForTubRef may have succeeded, however, if the other side # tried to connect to us at exactly the same time, they were the # master, they established their connection first (but the final # decision is still in flight), and they hung up on our connection # because they felt it was a duplicate. So, if self.failureReason # indicates a duplicate connection, do not signal a failure here. We # leave the connection timer in place in case they lied about having # a duplicate connection ready to go. if (self.failureReason.check(RemoteNegotiationError) and isSubstring(self.failureReason.value.args[0], "Duplicate connection")): self.log("TubConnector.checkForFailure: connection attempt " "failed because the other end decided ours was a " "duplicate connection, so we won't signal the " "failure here") return self.failed() def failed(self): self.stopConnectionTimer() self.active = False if self.failureReason: self.failureReason._connectionInfo = self._connectionInfo self.tub.connectionFailed(self.target, self.failureReason) self.tub.connectorFinished(self) def checkForIdle(self): # When one connection finishes negotiation, the others are cancelled # to hurry them along their way towards disconnection. The last one # to resolve finally causes us to notify our parent Tub. if (self.remainingLocations or self.pendingConnections or self.pendingNegotiations): return # we have no more outstanding connections (either in progress or in # negotiation), so this connector is finished. self.log("connectorFinished (%s)" % self) self.tub.connectorFinished(self) foolscap-0.13.1/src/foolscap/connections/0000755000076500000240000000000013204747603020750 5ustar warnerstaff00000000000000foolscap-0.13.1/src/foolscap/connections/__init__.py0000644000076500000240000000000012766553111023050 0ustar warnerstaff00000000000000foolscap-0.13.1/src/foolscap/connections/i2p.py0000644000076500000240000000357413204160675022023 0ustar warnerstaff00000000000000import re from twisted.internet.endpoints import clientFromString from twisted.internet.interfaces import IStreamClientEndpoint from txi2p.sam import SAMI2PStreamClientEndpoint from zope.interface import implementer from foolscap.ipb import IConnectionHintHandler, InvalidHintError HINT_RE=re.compile(r"^i2p:([A-Za-z.0-9\-]+)(:(\d+){1,5})?$") @implementer(IConnectionHintHandler) class _RunningI2P: def __init__(self, sam_endpoint, **kwargs): assert IStreamClientEndpoint.providedBy(sam_endpoint) self._sam_endpoint = sam_endpoint self._kwargs = kwargs def hint_to_endpoint(self, hint, reactor, update_status): # Return (endpoint, hostname), where "hostname" is what we pass to the # HTTP "Host:" header so a dumb HTTP server can be used to redirect us. mo = HINT_RE.search(hint) if not mo: raise InvalidHintError("unrecognized I2P hint") host, portnum = mo.group(1), int(mo.group(3)) if mo.group(3) else None kwargs = self._kwargs.copy() if not portnum and 'port' in kwargs: portnum = kwargs.pop('port') ep = SAMI2PStreamClientEndpoint.new(self._sam_endpoint, host, portnum, **kwargs) return ep, host def describe(self): return "i2p" def default(reactor, **kwargs): """Return a handler which connects to a pre-existing I2P process on the default SAM port. """ return _RunningI2P(clientFromString(reactor, 'tcp:127.0.0.1:7656'), **kwargs) def sam_endpoint(sam_port_endpoint, **kwargs): """Return a handler which connects to a pre-existing I2P process on the given SAM port. - sam_endpoint: a ClientEndpoint which points at the SAM API """ return _RunningI2P(sam_port_endpoint, **kwargs) def local_i2p(i2p_configdir=None): raise NotImplementedError def launch(i2p_configdir=None, i2p_binary=None): raise NotImplementedError foolscap-0.13.1/src/foolscap/connections/socks.py0000644000076500000240000000211113204160675022435 0ustar warnerstaff00000000000000import re from zope.interface import implementer from foolscap.ipb import IConnectionHintHandler, InvalidHintError from .tcp import DOTTED_QUAD_RESTR, DNS_NAME_RESTR from txsocksx.client import SOCKS5ClientEndpoint HINT_RE = re.compile(r"^[^:]*:(%s|%s):(\d+){1,5}$" % (DOTTED_QUAD_RESTR, DNS_NAME_RESTR)) @implementer(IConnectionHintHandler) class _SOCKS: """This can connect to tcp: or tor: hints through a SOCKS5 proxy.""" def __init__(self, proxy_endpoint): self._proxy_endpoint = proxy_endpoint def hint_to_endpoint(self, hint, reactor, update_status): mo = HINT_RE.search(hint) if not mo: raise InvalidHintError("unrecognized hint, wanted TYPE:HOST:PORT") host, port = mo.group(1), int(mo.group(2)) # note: txsockx does not expose a way to provide the reactor ep = SOCKS5ClientEndpoint(host, port, self._proxy_endpoint) return ep, host def describe(self): return "socks" def socks_endpoint(proxy_endpoint): return _SOCKS(proxy_endpoint) foolscap-0.13.1/src/foolscap/connections/tcp.py0000644000076500000240000000620113204160675022105 0ustar warnerstaff00000000000000import re from zope.interface import implementer from twisted.internet.endpoints import HostnameEndpoint from foolscap.ipb import IConnectionHintHandler, InvalidHintError DOTTED_QUAD_RESTR=r"\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}" # In addition to the usual colon-hex IPv6 addresses, accept "::FFFF:1.2.3.4" # (IPv4-mapped), and "FE8::1%en0" (local-scope/site-scope with a zone-id) COLON_HEX_RESTR=(r"\[[A-Fa-f0-9:]+" + r"(?:" + DOTTED_QUAD_RESTR + r"|%[a-zA-Z0-9.]+)?\]") DNS_NAME_RESTR=r"[A-Za-z.0-9\-]+" # This matches just (hostname or IPv4 address) and port number OLD_STYLE_HINT_RE=re.compile(r"^(%s|%s):(\d+){1,5}$" % (DOTTED_QUAD_RESTR, DNS_NAME_RESTR)) # This matches "tcp:" prefix plus (hostname or IPv4 address or []-wrapped # IPv6 address) plus port number NEW_STYLE_HINT_RE=re.compile(r"^tcp:(%s|%s|%s):(\d+){1,5}$" % (DOTTED_QUAD_RESTR, COLON_HEX_RESTR, DNS_NAME_RESTR)) # Each location hint must start with "TYPE:" (where TYPE is alphanumeric) and # then can contain any characters except "," and "/". These are generally # expected to contain ":"-separated fields (e.g. "TYPE:stuff:morestuff" or # "TYPE:key=value:key=value"). # For compatibility with current and older Foolscap releases, we also accept # old-syle implicit TCP hints ("host:port"). These are caught and converted # into new-style "tcp:HOST:PORT" hints in convert_legacy_hint() before we # look up the handler. In this case, HOST can either be a DNS name or a # dotted-quad IPv4 address. # To avoid being interpreted as an old-style hint, the part after TYPE: may # not consist of only 1-5 digits (so "type:123" will be treated as type="tcp" # and hostname="type"). Creators of new hint types are advised to either use # multiple colons (e.g. tor:HOST:PORT), or use key=value in the right-hand # portion (e.g. systemd:fd=3). # Future versions of foolscap may put hints in their FURLs which we do not # understand. We will ignore such hints. This version understands two types # of hints: # # HOST:PORT (old-style implicit tcp) # tcp:HOST:PORT (endpoint syntax for TCP connections) # For new-style hints, HOST can be a DNS name, a dotted-quad IPv4 address, or # a square-bracked-wrapped colon-hex IPv6 address. def convert_legacy_hint(location): mo = OLD_STYLE_HINT_RE.search(location) if mo: host, port = mo.group(1), int(mo.group(2)) return "tcp:%s:%d" % (host, port) return location @implementer(IConnectionHintHandler) class DefaultTCP: def hint_to_endpoint(self, hint, reactor, update_status): # Return (endpoint, hostname), where "hostname" is what we pass to the # HTTP "Host:" header so a dumb HTTP server can be used to redirect us. mo = NEW_STYLE_HINT_RE.search(hint) if not mo: raise InvalidHintError("unrecognized TCP hint") host, port = mo.group(1), int(mo.group(2)) host = host.lstrip("[").rstrip("]") return HostnameEndpoint(reactor, host, port), host def describe(self): return "tcp" def default(): return DefaultTCP() foolscap-0.13.1/src/foolscap/connections/tor.py0000644000076500000240000002232013204160675022123 0ustar warnerstaff00000000000000import os, re from twisted.internet.interfaces import IStreamClientEndpoint from twisted.internet.defer import inlineCallbacks, returnValue, succeed from twisted.internet.endpoints import clientFromString import ipaddress from .. import observer from zope.interface import implementer from ..ipb import IConnectionHintHandler, InvalidHintError from ..util import allocate_tcp_port import txtorcon from .tcp import DOTTED_QUAD_RESTR, DNS_NAME_RESTR def is_non_public_numeric_address(host): # for numeric hostnames, skip RFC1918 addresses, since no Tor exit # node will be able to reach those. Likewise ignore IPv6 addresses. try: a = ipaddress.ip_address(host.decode("ascii")) # wants unicode except ValueError: return False # non-numeric, let Tor try it if a.version != 4: return True # IPv6 gets ignored if (a.is_loopback or a.is_multicast or a.is_private or a.is_reserved or a.is_unspecified): return True # too weird, don't connect return False HINT_RE = re.compile(r"^[^:]*:(%s|%s):(\d+){1,5}$" % (DOTTED_QUAD_RESTR, DNS_NAME_RESTR)) class add_context(object): def __init__(self, update_status, context): self.update_status = update_status self.context = context self.suffix = " (while %s)" % context def __enter__(self): self.update_status(self.context) def __exit__(self, type, value, traceback): if value is not None: if not hasattr(value, "foolscap_connection_handler_error_suffix"): value.foolscap_connection_handler_error_suffix = self.suffix @implementer(IConnectionHintHandler) class _Common: # subclasses must define self._connect(reactor), which fires with the # socks Endpoint that TorClientEndpoint can use def __init__(self): self._connected = False self._when_connected = observer.OneShotObserverList() def _maybe_connect(self, reactor, update_status): if not self._connected: self._connected = True d = self._connect(reactor, update_status) d.addBoth(self._when_connected.fire) return self._when_connected.whenFired() @inlineCallbacks def hint_to_endpoint(self, hint, reactor, update_status): # Return (endpoint, hostname), where "hostname" is what we pass to the # HTTP "Host:" header so a dumb HTTP server can be used to redirect us. mo = HINT_RE.search(hint) if not mo: raise InvalidHintError("unrecognized TCP/Tor hint") host, portnum = mo.group(1), int(mo.group(2)) if is_non_public_numeric_address(host): raise InvalidHintError("ignoring non-Tor-able ipaddr %s" % host) with add_context(update_status, "connecting to a Tor"): socks_endpoint = yield self._maybe_connect(reactor, update_status) # txsocksx doesn't like unicode: it concatenates some binary protocol # bytes with the hostname when talking to the SOCKS server, so the # py2 automatic unicode promotion blows up host = host.encode("ascii") ep = txtorcon.TorClientEndpoint(host, portnum, socks_endpoint=socks_endpoint) returnValue( (ep, host) ) def describe(self): return "tor" # note: TorClientEndpoint imports 'reactor' itself, doesn't provide override. # This will be fixed in txtorcon 1.0 class _SocksTor(_Common): def __init__(self, socks_endpoint=None): _Common.__init__(self) self._socks_endpoint = socks_endpoint # socks_endpoint=None means to use defaults: TCP to 127.0.0.1 with # 9050, then 9150 def _connect(self, reactor, update_status): return succeed(self._socks_endpoint) def default_socks(): return _SocksTor() def socks_endpoint(tor_socks_endpoint): assert IStreamClientEndpoint.providedBy(tor_socks_endpoint) return _SocksTor(tor_socks_endpoint) class _LaunchedTor(_Common): def __init__(self, data_directory=None, tor_binary=None): _Common.__init__(self) self._data_directory = data_directory self._tor_binary = tor_binary @inlineCallbacks def _connect(self, reactor, update_status): # create a new Tor config = self.config = txtorcon.TorConfig() if self._data_directory: # The default is for launch_tor to create a tempdir itself, and # delete it when done. We only need to set a DataDirectory if we # want it to be persistent. This saves some startup time, because # we cache the descriptors from last time. On one of my hosts, # this reduces connect from 20s to 15s. if not os.path.exists(self._data_directory): # tor will mkdir this, but txtorcon wants to chdir to it # before spawning the tor process, so (for now) we need to # mkdir it ourselves. TODO: txtorcon should take # responsibility for this. os.mkdir(self._data_directory) config.DataDirectory = self._data_directory #config.ControlPort = allocate_tcp_port() # defaults to 9052 config.SocksPort = allocate_tcp_port() socks_desc = "tcp:127.0.0.1:%d" % config.SocksPort self._socks_desc = socks_desc # stash for tests socks_endpoint = clientFromString(reactor, socks_desc) with add_context(update_status, "launching Tor"): tpp = yield txtorcon.launch_tor(config, reactor, tor_binary=self._tor_binary) #print "launched" # gives a TorProcessProtocol with .tor_protocol self._tor_protocol = tpp.tor_protocol returnValue(socks_endpoint) def launch(data_directory=None, tor_binary=None): """Return a handler which launches a new Tor process (once). - data_directory: a persistent directory where Tor can cache its descriptors. This allows subsequent invocations to start faster. If None, the process will use an ephemeral tempdir, deleting it when Tor exits. - tor_binary: the path to the Tor executable we should use. If None, search $PATH. """ return _LaunchedTor(data_directory, tor_binary) @implementer(IConnectionHintHandler) class _ConnectedTor(_Common): def __init__(self, tor_control_endpoint_maker): _Common.__init__(self) assert callable(tor_control_endpoint_maker), tor_control_endpoint_maker self._tor_control_endpoint_maker = tor_control_endpoint_maker @inlineCallbacks def _connect(self, reactor, update_status): maker = self._tor_control_endpoint_maker with add_context(update_status, "making Tor control endpoint"): tor_control_endpoint = yield maker(reactor, update_status) assert IStreamClientEndpoint.providedBy(tor_control_endpoint) with add_context(update_status, "connecting to Tor"): tproto = yield txtorcon.build_tor_connection(tor_control_endpoint, build_state=False) with add_context(update_status, "waiting for Tor bootstrap"): config = yield txtorcon.TorConfig.from_protocol(tproto) ports = list(config.SocksPort) # I've seen "9050", and "unix:/var/run/tor/socks WorldWritable" for port in ports: pieces = port.split() p = pieces[0] if p == txtorcon.DEFAULT_VALUE: p = "9050" try: portnum = int(p) socks_desc = "tcp:127.0.0.1:%d" % portnum self._socks_desc = socks_desc # stash for tests socks_endpoint = clientFromString(reactor, socks_desc) returnValue(socks_endpoint) except ValueError: pass raise ValueError("could not use config.SocksPort: %r" % (ports,)) def control_endpoint_maker(tor_control_endpoint_maker, takes_status=False): """Return a handler which connects to a pre-existing Tor process on a control port provided by the maker function. - tor_control_endpoint_maker: a callable, which will be invoked once (with a single argument: 'reactor'). It can return immediately or return a Deferred, returning/yielding a ClientEndpoint which points at the Tor control port. - If takes_status=True, the callable will be invoked with two arguments: 'reactor' and 'update_status'. This is the preferred API, and allows the maker function to make status updates as it launches or locates the Tor control port. """ assert callable(tor_control_endpoint_maker), tor_control_endpoint_maker if takes_status: return _ConnectedTor(tor_control_endpoint_maker) else: def does_not_take_status(reactor, update_status): return tor_control_endpoint_maker(reactor) return _ConnectedTor(does_not_take_status) def control_endpoint(tor_control_endpoint): """Return a handler which connects to a pre-existing Tor process on the given control port. - tor_control_endpoint: a ClientEndpoint which points at the Tor control port """ assert IStreamClientEndpoint.providedBy(tor_control_endpoint) return _ConnectedTor(lambda reactor, update_status: tor_control_endpoint) foolscap-0.13.1/src/foolscap/constraint.py0000644000076500000240000002471512766553111021176 0ustar warnerstaff00000000000000 # This provides a base for the various Constraint subclasses to use. Those # Constraint subclasses live next to the slicers. It also contains # Constraints for primitive types (int, str). # This imports foolscap.tokens, but no other Foolscap modules. import re from zope.interface import implements, Interface from foolscap.tokens import Violation, BananaError, SIZE_LIMIT, \ STRING, LIST, INT, NEG, LONGINT, LONGNEG, VOCAB, FLOAT, OPEN, \ tokenNames everythingTaster = { # he likes everything STRING: None, LIST: None, INT: None, NEG: None, LONGINT: SIZE_LIMIT, # this limits numbers to about 2**8000, probably ok LONGNEG: SIZE_LIMIT, VOCAB: None, FLOAT: None, OPEN: None, } openTaster = { OPEN: None, } nothingTaster = {} class IConstraint(Interface): pass class IRemoteMethodConstraint(IConstraint): def getPositionalArgConstraint(argnum): """Return the constraint for posargs[argnum]. This is called on inbound methods when receiving positional arguments. This returns a tuple of (accept, constraint), where accept=False means the argument should be rejected immediately, regardless of what type it might be.""" def getKeywordArgConstraint(argname, num_posargs=0, previous_kwargs=[]): """Return the constraint for kwargs[argname]. The other arguments are used to handle mixed positional and keyword arguments. Returns a tuple of (accept, constraint).""" def checkAllArgs(args, kwargs, inbound): """Submit all argument values for checking. When inbound=True, this is called after the arguments have been deserialized, but before the method is invoked. When inbound=False, this is called just inside callRemote(), as soon as the target object (and hence the remote method constraint) is located. This should either raise Violation or return None.""" pass def getResponseConstraint(): """Return an IConstraint-providing object to enforce the response constraint. This is called on outbound method calls so that when the response starts to come back, we can start enforcing the appropriate constraint right away.""" def checkResults(results, inbound): """Inspect the results of invoking a method call. inbound=False is used on the side that hosts the Referenceable, just after the target method has provided a value. inbound=True is used on the RemoteReference side, just after it has finished deserializing the response. This should either raise Violation or return None.""" class Constraint(object): """ Each __schema__ attribute is turned into an instance of this class, and is eventually given to the unserializer (the 'Unslicer') to enforce as the tokens are arriving off the wire. """ implements(IConstraint) taster = everythingTaster """the Taster is a dict that specifies which basic token types are accepted. The keys are typebytes like INT and STRING, while the values are size limits: the body portion of the token must not be longer than LIMIT bytes. """ strictTaster = False """If strictTaster is True, taste violations are raised as BananaErrors (indicating a protocol error) rather than a mere Violation. """ opentypes = None """opentypes is a list of currently acceptable OPEN token types. None indicates that all types are accepted. An empty list indicates that no OPEN tokens are accepted. """ name = None """Used to describe the Constraint in a Violation error message""" def checkToken(self, typebyte, size): """Check the token type. Raise an exception if it is not accepted right now, or if the body-length limit is exceeded.""" limit = self.taster.get(typebyte, "not in list") if limit == "not in list": if self.strictTaster: raise BananaError("invalid token type: %s" % tokenNames[typebyte]) else: raise Violation("%s token rejected by %s" % (tokenNames[typebyte], self.name)) if limit and size > limit: raise Violation("%s token too large: %d>%d" % (tokenNames[typebyte], size, limit)) def setNumberTaster(self, maxValue): self.taster = {INT: None, NEG: None, LONGINT: None, # TODO LONGNEG: None, FLOAT: None, } def checkOpentype(self, opentype): """Check the OPEN type (the tuple of Index Tokens). Raise an exception if it is not accepted. """ if self.opentypes == None: return # shared references are always accepted. checkOpentype() is a defense # against resource-exhaustion attacks, and references don't consume # any more resources than any other token. For inbound method # arguments, the CallUnslicer will perform a final check on all # arguments (after these shared references have been resolved), and # that will get to verify that they have resolved to the correct # type. #if opentype == ReferenceSlicer.opentype: if opentype == ('reference',): return for o in self.opentypes: if len(o) == len(opentype): if o == opentype: return if len(o) > len(opentype): # we might have a partial match: they haven't flunked yet if opentype == o[:len(opentype)]: return # still in the running raise Violation("unacceptable OPEN type: %s not in my list %s" % (opentype, self.opentypes)) def checkObject(self, obj, inbound): """Validate an existing object. Usually objects are validated as their tokens come off the wire, but pre-existing objects may be added to containers if a REFERENCE token arrives which points to them. The older objects were were validated as they arrived (by a different schema), but now they must be re-validated by the new schema. A more naive form of validation would just accept the entire object tree into memory and then run checkObject() on the result. This validation is too late: it is vulnerable to both DoS and made-you-run-code attacks. If inbound=True, this object is arriving over the wire. If inbound=False, this is being called to validate an existing object before it is sent over the wire. This is done as a courtesy to the remote end, and to improve debuggability. Most constraints can use the same checker for both inbound and outbound objects. """ # this default form passes everything return COUNTERBYTES = 64 # max size of opencount def OPENBYTES(self, dummy): # an OPEN,type,CLOSE sequence could consume: # 64 (header) # 1 (OPEN) # 64 (header) # 1 (STRING) # 1000 (value) # or # 64 (header) # 1 (VOCAB) # 64 (header) # 1 (CLOSE) # for a total of 65+1065+65 = 1195 return self.COUNTERBYTES+1 + 64+1+1000 + self.COUNTERBYTES+1 class OpenerConstraint(Constraint): taster = openTaster class Any(Constraint): pass # accept everything # constraints which describe individual banana tokens class ByteStringConstraint(Constraint): opentypes = [] # redundant, as taster doesn't accept OPEN name = "ByteStringConstraint" def __init__(self, maxLength=None, minLength=0, regexp=None): self.maxLength = maxLength self.minLength = minLength # regexp can either be a string or a compiled SRE_Match object.. # re.compile appears to notice SRE_Match objects and pass them # through unchanged. self.regexp = None if regexp: self.regexp = re.compile(regexp) self.taster = {STRING: self.maxLength, VOCAB: None} def checkObject(self, obj, inbound): if not isinstance(obj, str): raise Violation("'%r' is not a bytestring" % (obj,)) if self.maxLength != None and len(obj) > self.maxLength: raise Violation("string too long (%d > %d)" % (len(obj), self.maxLength)) if len(obj) < self.minLength: raise Violation("string too short (%d < %d)" % (len(obj), self.minLength)) if self.regexp: if not self.regexp.search(obj): raise Violation("regexp failed to match") class IntegerConstraint(Constraint): opentypes = [] # redundant # taster set in __init__ name = "IntegerConstraint" def __init__(self, maxBytes=-1): # -1 means s_int32_t: INT/NEG instead of INT/NEG/LONGINT/LONGNEG # None means unlimited assert maxBytes == -1 or maxBytes == None or maxBytes >= 4 self.maxBytes = maxBytes self.taster = {INT: None, NEG: None} if maxBytes != -1: self.taster[LONGINT] = maxBytes self.taster[LONGNEG] = maxBytes def checkObject(self, obj, inbound): if not isinstance(obj, (int, long)): raise Violation("'%r' is not a number" % (obj,)) if self.maxBytes == -1: if obj >= 2**31 or obj < -2**31: raise Violation("number too large") elif self.maxBytes != None: if abs(obj) >= 2**(8*self.maxBytes): raise Violation("number too large") class NumberConstraint(IntegerConstraint): """I accept floats, ints, and longs.""" name = "NumberConstraint" def __init__(self, maxBytes=1024): assert maxBytes != -1 # not valid here IntegerConstraint.__init__(self, maxBytes) self.taster[FLOAT] = None def checkObject(self, obj, inbound): if isinstance(obj, float): return IntegerConstraint.checkObject(self, obj, inbound) #TODO class Shared(Constraint): name = "Shared" def __init__(self, constraint, refLimit=None): self.constraint = IConstraint(constraint) self.refLimit = refLimit #TODO: might be better implemented with a .optional flag class Optional(Constraint): name = "Optional" def __init__(self, constraint, default): self.constraint = IConstraint(constraint) self.default = default foolscap-0.13.1/src/foolscap/copyable.py0000644000076500000240000003726112766553111020610 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test.test_copyable -*- # this module is responsible for all copy-by-value objects from zope.interface import interface, implements from twisted.python import reflect, log from twisted.python.components import registerAdapter from twisted.internet import defer import slicer, tokens from tokens import BananaError, Violation from foolscap.constraint import OpenerConstraint, IConstraint, Optional Interface = interface.Interface ############################################################ # the first half of this file is sending/serialization class ICopyable(Interface): """I represent an object which is passed-by-value across PB connections. """ def getTypeToCopy(): """Return a string which names the class. This string must match the one that gets registered at the receiving end. This is typically a URL of some sort, in a namespace which you control.""" def getStateToCopy(): """Return a state dictionary (with plain-string keys) which will be serialized and sent to the remote end. This state object will be given to the receiving object's setCopyableState method.""" class Copyable(object): implements(ICopyable) # you *must* set 'typeToCopy' def getTypeToCopy(self): try: copytype = self.typeToCopy except AttributeError: raise RuntimeError("Copyable subclasses must specify 'typeToCopy'") return copytype def getStateToCopy(self): return self.__dict__ class CopyableSlicer(slicer.BaseSlicer): """I handle ICopyable objects (things which are copied by value).""" def slice(self, streamable, banana): self.streamable = streamable yield 'copyable' copytype = self.obj.getTypeToCopy() assert isinstance(copytype, str) yield copytype state = self.obj.getStateToCopy() for k,v in state.iteritems(): yield k yield v def describe(self): return "<%s>" % self.obj.getTypeToCopy() registerAdapter(CopyableSlicer, ICopyable, tokens.ISlicer) class Copyable2(slicer.BaseSlicer): # I am my own Slicer. This has more methods than you'd usually want in a # base class, but if you can't register an Adapter for a whole class # hierarchy then you may have to use it. def getTypeToCopy(self): return reflect.qual(self.__class__) def getStateToCopy(self): return self.__dict__ def slice(self, streamable, banana): self.streamable = streamable yield 'instance' yield self.getTypeToCopy() yield self.getStateToCopy() def describe(self): return "<%s>" % self.getTypeToCopy() #registerRemoteCopy(typename, factory) #registerUnslicer(typename, factory) def registerCopier(klass, copier): """This is a shortcut for arranging to serialize third-party clases. 'copier' must be a callable which accepts an instance of the class you want to serialize, and returns a tuple of (typename, state_dictionary). If it returns a typename of None, the original class's fully-qualified classname is used. """ klassname = reflect.qual(klass) class _CopierAdapter: implements(ICopyable) def __init__(self, original): self.nameToCopy, self.state = copier(original) if self.nameToCopy is None: self.nameToCopy = klassname def getTypeToCopy(self): return self.nameToCopy def getStateToCopy(self): return self.state registerAdapter(_CopierAdapter, klass, ICopyable) ############################################################ # beyond here is the receiving/deserialization side class RemoteCopyUnslicer(slicer.BaseUnslicer): attrname = None attrConstraint = None def __init__(self, factory, stateSchema): self.factory = factory self.schema = stateSchema def start(self, count): self.d = {} self.count = count self.deferred = defer.Deferred() self.protocol.setObject(count, self.deferred) def checkToken(self, typebyte, size): if self.attrname == None: if typebyte not in (tokens.STRING, tokens.VOCAB): raise BananaError("RemoteCopyUnslicer keys must be STRINGs") else: if self.attrConstraint: self.attrConstraint.checkToken(typebyte, size) def doOpen(self, opentype): if self.attrConstraint: self.attrConstraint.checkOpentype(opentype) unslicer = self.open(opentype) if unslicer: if self.attrConstraint: unslicer.setConstraint(self.attrConstraint) return unslicer def receiveChild(self, obj, ready_deferred=None): assert not isinstance(obj, defer.Deferred) assert ready_deferred is None if self.attrname == None: attrname = obj if self.d.has_key(attrname): raise BananaError("duplicate attribute name '%s'" % attrname) s = self.schema if s: accept, self.attrConstraint = s.getAttrConstraint(attrname) assert accept self.attrname = attrname else: if isinstance(obj, defer.Deferred): # TODO: this is an artificial restriction, and it might # be possible to remove it, but I need to think through # it carefully first raise BananaError("unreferenceable object in attribute") self.setAttribute(self.attrname, obj) self.attrname = None self.attrConstraint = None def setAttribute(self, name, value): self.d[name] = value def receiveClose(self): try: obj = self.factory(self.d) except: log.msg("%s.receiveClose: problem in factory %s" % (self.__class__.__name__, self.factory)) log.err() raise self.protocol.setObject(self.count, obj) self.deferred.callback(obj) return obj, None def describe(self): if self.classname == None: return "" me = "<%s>" % self.classname if self.attrname is None: return "%s.attrname??" % me else: return "%s.%s" % (me, self.attrname) class NonCyclicRemoteCopyUnslicer(RemoteCopyUnslicer): # The Deferred used in RemoteCopyUnslicer (used in case the RemoteCopy # is participating in a reference cycle, say 'obj.foo = obj') makes it # unsuitable for holding Failures (which cannot be passed through # Deferred.callback). Use this class for Failures. It cannot handle # reference cycles (they will cause a KeyError when the reference is # followed). def start(self, count): self.d = {} self.count = count self.gettingAttrname = True def receiveClose(self): obj = self.factory(self.d) return obj, None class IRemoteCopy(Interface): """This interface defines what a RemoteCopy class must do. RemoteCopy subclasses are used as factories to create objects that correspond to Copyables sent over the wire. Note that the constructor of an IRemoteCopy class will be called without any arguments. """ def setCopyableState(statedict): """I accept an attribute dictionary name/value pairs and use it to set my internal state. Some of the values may be Deferreds, which are placeholders for the as-yet-unreferenceable object which will eventually go there. If you receive a Deferred, you are responsible for adding a callback to update the attribute when it fires. [note: RemoteCopyUnslicer.receiveChild currently has a restriction which prevents this from happening, but that may go away in the future] Some of the objects referenced by the attribute values may have Deferreds in them (e.g. containers which reference recursive tuples). Such containers are responsible for updating their own state when those Deferreds fire, but until that point their state is still subject to change. Therefore you must be careful about how much state inspection you perform within this method.""" stateSchema = interface.Attribute("""I return an AttributeDictConstraint object which places restrictions on incoming attribute values. These restrictions are enforced as the tokens are received, before the state is passed to setCopyableState.""") # This maps typename to an Unslicer factory CopyableRegistry = {} def registerRemoteCopyUnslicerFactory(typename, unslicerfactory, registry=None): """Tell PB that unslicerfactory can be used to handle Copyable objects that provide a getTypeToCopy name of 'typename'. 'unslicerfactory' must be a callable which takes no arguments and returns an object which provides IUnslicer. """ assert callable(unslicerfactory) # in addition, it must produce a tokens.IUnslicer . This is safe to do # because Unslicers don't do anything significant when they are created. test_unslicer = unslicerfactory() assert tokens.IUnslicer.providedBy(test_unslicer) assert type(typename) is str if registry == None: registry = CopyableRegistry assert not registry.has_key(typename) registry[typename] = unslicerfactory # this keeps track of everything submitted to registerRemoteCopyFactory debug_CopyableFactories = {} def registerRemoteCopyFactory(typename, factory, stateSchema=None, cyclic=True, registry=None): """Tell PB that 'factory' can be used to handle Copyable objects that provide a getTypeToCopy name of 'typename'. 'factory' must be a callable which accepts a state dictionary and returns a fully-formed instance. 'cyclic' is a boolean, which should be set to False to avoid using a Deferred to provide the resulting RemoteCopy instance. This is needed to deserialize Failures (or instances which inherit from one, like CopiedFailure). In exchange for this, it cannot handle reference cycles. """ assert callable(factory) debug_CopyableFactories[typename] = (factory, stateSchema, cyclic) if cyclic: def _RemoteCopyUnslicerFactory(): return RemoteCopyUnslicer(factory, stateSchema) registerRemoteCopyUnslicerFactory(typename, _RemoteCopyUnslicerFactory, registry) else: def _RemoteCopyUnslicerFactoryNonCyclic(): return NonCyclicRemoteCopyUnslicer(factory, stateSchema) registerRemoteCopyUnslicerFactory(typename, _RemoteCopyUnslicerFactoryNonCyclic, registry) # this keeps track of everything submitted to registerRemoteCopy, which may # be useful when you're wondering what's been auto-registered by the # RemoteCopy metaclass magic debug_RemoteCopyClasses = {} def registerRemoteCopy(typename, remote_copy_class, registry=None): """Tell PB that remote_copy_class is the appropriate RemoteCopy class to use when deserializing a Copyable sequence that is tagged with 'typename'. 'remote_copy_class' should be a RemoteCopy subclass or implement the same interface, which means its constructor takes no arguments and it has a setCopyableState(state) method to actually set the instance's state after initialization. It must also have a nonCyclic attribute. """ assert IRemoteCopy.implementedBy(remote_copy_class) assert type(typename) is str debug_RemoteCopyClasses[typename] = remote_copy_class def _RemoteCopyFactory(state): obj = remote_copy_class() obj.setCopyableState(state) return obj registerRemoteCopyFactory(typename, _RemoteCopyFactory, remote_copy_class.stateSchema, not remote_copy_class.nonCyclic, registry) class RemoteCopyClass(type): # auto-register RemoteCopy classes def __init__(self, name, bases, dict): type.__init__(self, name, bases, dict) # don't try to register RemoteCopy itself if name == "RemoteCopy" and _RemoteCopyBase in bases: #print "not auto-registering %s %s" % (name, bases) return if "copytype" not in dict: # TODO: provide a file/line-number for the class raise RuntimeError("RemoteCopy subclass %s must specify 'copytype'" % name) copytype = dict['copytype'] if copytype: registry = dict.get('copyableRegistry', None) registerRemoteCopy(copytype, self, registry) class _RemoteCopyBase: implements(IRemoteCopy) stateSchema = None # always a class attribute nonCyclic = False def __init__(self): # the constructor will always be called without arguments pass def setCopyableState(self, state): self.__dict__ = state class RemoteCopyOldStyle(_RemoteCopyBase): # note that these will not auto-register for you, because old-style # classes do not do metaclass magic copytype = None class RemoteCopy(_RemoteCopyBase, object): # Set 'copytype' to a unique string that is shared between the # sender-side Copyable and the receiver-side RemoteCopy. This RemoteCopy # subclass will be auto-registered using the 'copytype' name. Set # copytype to None to disable auto-registration. __metaclass__ = RemoteCopyClass pass class AttributeDictConstraint(OpenerConstraint): """This is a constraint for dictionaries that are used for attributes. All keys are short strings, and each value has a separate constraint. It could be used to describe instance state, but could also be used to constraint arbitrary dictionaries with string keys. Some special constraints are legal here: Optional. """ opentypes = [("attrdict",)] name = "AttributeDictConstraint" def __init__(self, *attrTuples, **kwargs): self.ignoreUnknown = kwargs.get('ignoreUnknown', False) self.acceptUnknown = kwargs.get('acceptUnknown', False) self.keys = {} for name, constraint in (list(attrTuples) + kwargs.get('attributes', {}).items()): assert name not in self.keys.keys() self.keys[name] = IConstraint(constraint) def getAttrConstraint(self, attrname): c = self.keys.get(attrname) if c: if isinstance(c, Optional): c = c.constraint return (True, c) # unknown attribute if self.ignoreUnknown: return (False, None) if self.acceptUnknown: return (True, None) raise Violation("unknown attribute '%s'" % attrname) def checkObject(self, obj, inbound): if type(obj) != type({}): raise Violation, "'%s' (%s) is not a Dictionary" % (obj, type(obj)) allkeys = self.keys.keys() for k in obj.keys(): try: constraint = self.keys[k] allkeys.remove(k) except KeyError: if not self.ignoreUnknown: raise Violation, "key '%s' not in schema" % k else: # hmm. kind of a soft violation. allow it for now. pass else: constraint.checkObject(obj[k], inbound) for k in allkeys[:]: if isinstance(self.keys[k], Optional): allkeys.remove(k) if allkeys: raise Violation("object is missing required keys: %s" % \ ",".join(allkeys)) foolscap-0.13.1/src/foolscap/crypto.py0000644000076500000240000000663012766553111020326 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test.test_crypto -*- from OpenSSL import SSL from twisted.internet.ssl import CertificateOptions, DistinguishedName, \ KeyPair, Certificate, PrivateCertificate from foolscap import base32 peerFromTransport = Certificate.peerFromTransport def alwaysValidate(conn, cert, errno, depth, preverify_ok): # This function is called to validate the certificate received by # the other end. OpenSSL calls it multiple times, each time it # see something funny, to ask if it should proceed. # We do not care about certificate authorities or revocation # lists, we just want to know that the certificate has a valid # signature and follow the chain back to one which is # self-signed. The TubID will be the digest of one of these # certificates. We need to protect against forged signatures, but # not the usual SSL concerns about invalid CAs or revoked # certificates. # these constants are from openssl-0.9.7g/crypto/x509/x509_vfy.h # and do not appear to be exposed by pyopenssl. Ick. TODO. We # could just always return '1' here (ignoring all errors), but I # think that would ignore forged signatures too, which would # obviously be a security hole. things_are_ok = (0, # X509_V_OK 9, # X509_V_ERR_CERT_NOT_YET_VALID 10, # X509_V_ERR_CERT_HAS_EXPIRED 18, # X509_V_ERR_DEPTH_ZERO_SELF_SIGNED_CERT 19, # X509_V_ERR_SELF_SIGNED_CERT_IN_CHAIN ) if errno in things_are_ok: return 1 # TODO: log the details of the error, because otherwise they get # lost in the PyOpenSSL exception that will eventually be raised # (possibly OpenSSL.SSL.Error: certificate verify failed) # I think that X509_V_ERR_CERT_SIGNATURE_FAILURE is the most # obvious sign of hostile attack. return 0 class FoolscapContextFactory(CertificateOptions): def getContext(self): ctx = CertificateOptions.getContext(self) # VERIFY_PEER means we ask the the other end for their certificate. # not adding VERIFY_FAIL_IF_NO_PEER_CERT means it's ok if they don't # give us one (i.e. if an anonymous client connects to an # authenticated server). I don't know what VERIFY_CLIENT_ONCE does. ctx.set_verify(SSL.VERIFY_PEER | #SSL.VERIFY_FAIL_IF_NO_PEER_CERT | SSL.VERIFY_CLIENT_ONCE, alwaysValidate) return ctx def digest32(colondigest): digest = "".join([chr(int(c,16)) for c in colondigest.split(":")]) digest = base32.encode(digest) return digest def createCertificate(): # this is copied from test_sslverify.py dn = DistinguishedName(commonName="newpb_thingy") keypair = KeyPair.generate(size=2048) req = keypair.certificateRequest(dn, digestAlgorithm="sha256") certData = keypair.signCertificateRequest(dn, req, lambda dn: True, 1, # serial number digestAlgorithm="sha256", ) cert = keypair.newCertificate(certData) #opts = cert.options() # 'opts' can be given to reactor.listenSSL, or to transport.startTLS return cert def loadCertificate(certData): cert = PrivateCertificate.loadPEM(certData) return cert foolscap-0.13.1/src/foolscap/eventual.py0000644000076500000240000000516112766553111020627 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test.test_eventual -*- from twisted.internet import reactor, defer from twisted.python import log class _SimpleCallQueue(object): # XXX TODO: merge epsilon.cooperator in, and make this more complete. def __init__(self): self._events = [] self._flushObservers = [] self._timer = None def append(self, cb, args, kwargs): self._events.append((cb, args, kwargs)) if not self._timer: self._timer = reactor.callLater(0, self._turn) def _turn(self): self._timer = None # flush all the messages that are currently in the queue. If anything # gets added to the queue while we're doing this, those events will # be put off until the next turn. events, self._events = self._events, [] for cb, args, kwargs in events: try: cb(*args, **kwargs) except: log.err() if not self._events: observers, self._flushObservers = self._flushObservers, [] for o in observers: o.callback(None) def flush(self): """Return a Deferred that will fire (with None) when the call queue is completely empty.""" if not self._events: return defer.succeed(None) d = defer.Deferred() self._flushObservers.append(d) return d _theSimpleQueue = _SimpleCallQueue() def eventually(cb, *args, **kwargs): """This is the eventual-send operation, used as a plan-coordination primitive. The callable will be invoked (with args and kwargs) in a later reactor turn. Doing 'eventually(a); eventually(b)' guarantees that a will be called before b. Any exceptions that occur in the callable will be logged with log.err(). If you really want to ignore them, be sure to provide a callable that catches those exceptions. This function returns None. If you care to know when the callable was run, be sure to provide a callable that notifies somebody. """ _theSimpleQueue.append(cb, args, kwargs) def fireEventually(value=None): """This returns a Deferred which will fire in a later reactor turn, after the current call stack has been completed, and after all other deferreds previously scheduled with callEventually(). """ d = defer.Deferred() eventually(d.callback, value) return d def flushEventualQueue(_ignored=None): """This returns a Deferred which fires when the eventual-send queue is finally empty. This is useful to wait upon as the last step of a Trial test method. """ return _theSimpleQueue.flush() foolscap-0.13.1/src/foolscap/furl.py0000644000076500000240000000276412766553111017762 0ustar warnerstaff00000000000000import re from foolscap import base32 class BadFURLError(Exception): pass AUTH_STURDYREF_RE = re.compile(r"pb://([^@]+)@([^/]*)/(.+)$") def decode_furl(furl): """Returns (tubID, location_hints, name)""" # pb://key@{ip:port,host:port,[ipv6]:port}[/unix]/swissnumber # i.e. pb://tubID@{locationHints..}/name # # it can live at any one of a (TODO) variety of network-accessible # locations, or (TODO) at a single UNIX-domain socket. mo_auth_furl = AUTH_STURDYREF_RE.search(furl) if mo_auth_furl: # we only pay attention to the first 32 base32 characters # of the tubid string. Everything else is left for future # extensions. tubID_s = mo_auth_furl.group(1) tubID = tubID_s[:32] if not base32.is_base32(tubID): raise BadFURLError("'%s' is not a valid tubid" % (tubID,)) hints = mo_auth_furl.group(2) location_hints = hints.split(",") if location_hints == [""]: location_hints = [] if "" in location_hints: raise BadFURLError("no connection hint may be empty") # it is legal to have no hints at all: an empty string turns into an # empty list name = mo_auth_furl.group(3) else: raise ValueError("unknown FURL prefix in %r" % (furl,)) return (tubID, location_hints, name) def encode_furl(tubID, location_hints, name): location_hints_s = ",".join(location_hints) return "pb://" + tubID + "@" + location_hints_s + "/" + name foolscap-0.13.1/src/foolscap/info.py0000644000076500000240000000201613204160675017730 0ustar warnerstaff00000000000000 class ConnectionInfo: def __init__(self): self.connected = False self.connectorStatuses = {} self.connectionHandlers = {} self.listenerStatus = (None, None) self.winningHint = None self.establishedAt = None self.lostAt = None def _set_connected(self, connected): self.connected = connected def _set_connection_status(self, location, status): self.connectorStatuses[location] = status def _describe_connection_handler(self, location, description): self.connectionHandlers[location] = description def _set_established_at(self, when): self.establishedAt = when def _set_winning_hint(self, location): self.winningHint = location def _set_listener_description(self, description): self.listenerStatus = (description, self.listenerStatus[1]) def _set_listener_status(self, status): self.listenerStatus = (self.listenerStatus[0], status) def _set_lost_at(self, when): self.lostAt = when foolscap-0.13.1/src/foolscap/ipb.py0000644000076500000240000001334313204160675017554 0ustar warnerstaff00000000000000 from zope.interface import interface Interface = interface.Interface # TODO: move these here from foolscap.tokens import ISlicer, IRootSlicer, IUnslicer _ignored = [ISlicer, IRootSlicer, IUnslicer] # hush pyflakes class InvalidHintError(Exception): """The hint was malformed and could not be used.""" class IConnectionHintHandler(Interface): def hint_to_endpoint(hint, reactor, update_status): """Return (endpoint, hostname), or a Deferred that fires with the same, where endpoint is an IStreamClientEndpoint object, and hostname is a string (for use in the HTTP headers during negotiation). The endpoint, once connected, must be capable of handling .startTLS(). Hints are strings which always start with 'TYPE:', and handlers are registered for specific types (and will not be called with hints of other types). update_status() can be called (with a string) to report progress, and should typically be set just before waiting for some connections step (e.g. connecting to a Tor daemon). Raise InvalidHintError (or return a Deferred that errbacks with one) if the hint could not be parsed or otherwise turned into an Endpoint. Set an attribute named 'foolscap_connection_handler_error' on the exception object to have `ConnectionInfo.connectorStatuses()` report that string instead of an exception-class -based status message.""" def describe(): """Return a short string describing this handler, like 'tcp' or 'tor'. If this method is not implemented, the handler's repr will be used.""" class DeadReferenceError(Exception): """The RemoteReference is dead, Jim.""" def __init__(self, why=None, remote_tubid=None, request=None): self.why = why self.remote_tubid = remote_tubid self.request = request def __str__(self): args = [] if self.why: args.append(self.why) if self.remote_tubid: args.append("(to tubid=%s)" % self.remote_tubid) if self.request: iname, mname = self.request.getMethodNameInfo() args.append("(during method=%s:%s)" % (iname, mname)) return " ".join([str(a) for a in args]) class IReferenceable(Interface): """This object is remotely referenceable. This means it is represented to remote systems as an opaque identifier, and that round-trips preserve identity. """ def processUniqueID(): """Return a unique identifier (scoped to the process containing the Referenceable). Most objects can just use C{id(self)}, but objects which should be indistinguishable to a remote system may want multiple objects to map to the same PUID.""" class IRemotelyCallable(Interface): """This object is remotely callable. This means it defines some remote_* methods and may have a schema which describes how those methods may be invoked. """ def getInterfaceNames(): """Return a list of RemoteInterface names to which this object knows how to respond.""" def doRemoteCall(methodname, args, kwargs): """Invoke the given remote method. This method may raise an exception, return normally, or return a Deferred.""" class ITub(Interface): """This marks a Tub.""" class IBroker(Interface): """This marks a broker.""" class IRemoteReference(Interface): """This marks a RemoteReference.""" def notifyOnDisconnect(callback, *args, **kwargs): """Register a callback to run when we lose this connection. The callback will be invoked with whatever extra arguments you provide to this function. For example:: def my_callback(name, number): print name, number+4 cookie = rref.notifyOnDisconnect(my_callback, 'bob', number=3) This function returns an opaque cookie. If you want to cancel the notification, pass this same cookie back to dontNotifyOnDisconnect:: rref.dontNotifyOnDisconnect(cookie) Note that if the Tub is shutdown (via stopService), all notifyOnDisconnect handlers are cancelled. """ def dontNotifyOnDisconnect(cookie): """Deregister a callback that was registered with notifyOnDisconnect. """ def callRemote(name, *args, **kwargs): """Invoke a method on the remote object with which I am associated. I always return a Deferred. This will fire with the results of the method when and if the remote end finishes. It will errback if any of the following things occur:: the arguments do not match the schema I believe is in use by the far end (causes a Violation exception) the connection to the far end has been lost (DeadReferenceError) the arguments are not accepted by the schema in use by the far end (Violation) the method executed by the far end raises an exception (arbitrary) the return value of the remote method is not accepted by the schema in use by the far end (Violation) the connection is lost before the response is returned (ConnectionLost) the return value is not accepted by the schema I believe is in use by the far end (Violation) """ def callRemoteOnly(name, *args, **kwargs): """Invoke a method on the remote object with which I am associated. This form is for one-way messages that do not require results or even acknowledgement of completion. I do not wait for the method to finish executing. The remote end will be instructed to not send any response. There is no way to know whether the method was successfully delivered or not. I always return None. """ foolscap-0.13.1/src/foolscap/logging/0000755000076500000240000000000013204747603020054 5ustar warnerstaff00000000000000foolscap-0.13.1/src/foolscap/logging/__init__.py0000644000076500000240000000000012766553111022154 0ustar warnerstaff00000000000000foolscap-0.13.1/src/foolscap/logging/app_versions.py0000644000076500000240000000060412766553111023137 0ustar warnerstaff00000000000000 import twisted import foolscap # You might want to modify this to include the version of your application. # Just do: # # from foolscap.logging import app_versions # app_versions.add_version("myapp", myversion) versions = {"twisted": twisted.__version__, "foolscap": foolscap.__version__, } def add_version(name, version): versions[name] = str(version) foolscap-0.13.1/src/foolscap/logging/cli.py0000644000076500000240000000644112766553111021203 0ustar warnerstaff00000000000000 import sys from StringIO import StringIO from twisted.python import usage import foolscap from foolscap.logging.tail import TailOptions, LogTail from foolscap.logging.gatherer import \ CreateGatherOptions, create_log_gatherer, \ CreateIncidentGatherOptions, create_incident_gatherer from foolscap.logging.dumper import DumpOptions, LogDumper from foolscap.logging.web import WebViewerOptions, WebViewer from foolscap.logging.filter import FilterOptions, Filter from foolscap.logging.incident import ClassifyOptions, IncidentClassifier class Options(usage.Options): synopsis = "Usage: flogtool (tail|create-gatherer|dump|filter|web-viewer)" subCommands = [ ("tail", None, TailOptions, "follow logs of the target node"), ("create-gatherer", None, CreateGatherOptions, "Make a .tac which will record all logs to a given directory"), ("create-incident-gatherer", None, CreateIncidentGatherOptions, "Make a .tac which will record all incidents to a given directory"), ("dump", None, DumpOptions, "dump the logs recorded by 'logtool gather'"), ("filter", None, FilterOptions, "produce a new file with a subset of the events from another file"), ("web-viewer", None, WebViewerOptions, "view the logs through a web page"), ("classify-incident", None, ClassifyOptions, "classify a stored Incident file"), ] def postOptions(self): if not hasattr(self, 'subOptions'): raise usage.UsageError("must specify a command") def opt_help(self): print self.synopsis sys.exit(0) def opt_version(self): from twisted import copyright print "Foolscap version:", foolscap.__version__ print "Twisted version:", copyright.version sys.exit(0) def dispatch(command, options): if command == "tail": lt = LogTail(options) return lt.run(options.target_furl) elif command == "create-gatherer": return create_log_gatherer(options) elif command == "create-incident-gatherer": return create_incident_gatherer(options) elif command == "dump": ld = LogDumper() return ld.run(options) elif command == "filter": f = Filter() return f.run(options) elif command == "web-viewer": wv = WebViewer() return wv.run(options) elif command == "classify-incident": ic = IncidentClassifier() return ic.run(options) else: print "unknown command '%s'" % command raise NotImplementedError def run_flogtool(argv=None, run_by_human=True): if argv: command_name = argv[0] else: command_name = sys.argv[0] config = Options() try: config.parseOptions(argv) except usage.error, e: if not run_by_human: raise print "%s: %s" % (command_name, e) print c = getattr(config, 'subOptions', config) print str(c) sys.exit(1) command = config.subCommand so = config.subOptions if not run_by_human: so.stdout = StringIO() so.stderr = StringIO() rc = dispatch(command, so) if rc is None: rc = 0 if run_by_human: sys.exit(rc) else: return (so.stdout.getvalue(), so.stderr.getvalue()) foolscap-0.13.1/src/foolscap/logging/dumper.py0000644000076500000240000001170713204511065021717 0ustar warnerstaff00000000000000 import sys, errno, textwrap from twisted.python import usage from foolscap.logging import flogfile from foolscap.logging.log import format_message from foolscap.util import format_time, FORMAT_TIME_MODES class DumpOptions(usage.Options): stdout = sys.stdout stderr = sys.stderr synopsis = "Usage: flogtool dump DUMPFILE.flog[.bz2]" optParameters = [ ("timestamps", "t", "short-local", "Format for timestamps: " + " ".join(FORMAT_TIME_MODES)), ] optFlags = [ ("verbose", "v", "Show all event arguments"), ("just-numbers", "n", "Show only event numbers"), ("rx-time", "r", "Show event receipt time (in addition to emit time)"), ] def opt_timestamps(self, arg): if arg not in FORMAT_TIME_MODES: raise usage.UsageError("--timestamps= must be one of (%s)" % ", ".join(FORMAT_TIME_MODES)) self["timestamps"] = arg def parseArgs(self, dumpfile): self.dumpfile = dumpfile class LogDumper: def __init__(self): self.trigger = None def run(self, options): try: for e in flogfile.get_events(options.dumpfile): if "header" in e: self.print_header(e, options) if "d" in e: self.print_event(e, options) except EnvironmentError, e: # "flogtool dump FLOGFILE |less" is very common, and if you quit # it early with "q", the stdout pipe is broken and python dies # with a messy stacktrace. Catch and ignore that. if e.errno == errno.EPIPE: return 1 raise except flogfile.ThisIsActuallyAFurlFileError: print >>options.stderr, textwrap.dedent("""\ Error: %s appears to be a FURL file. Perhaps you meant to run 'flogtool tail' instead of 'flogtool dump'?""" % (options.dumpfile,)) return 1 except flogfile.EvilPickleFlogFile: print >>options.stderr, textwrap.dedent("""\ Error: %s appears to be an old-style (pickle-based) flogfile, which cannot be loaded safely. If you wish to allow the author of the flogfile to take over your computer (and incidentally allow you to view the content), please use the flogtool from a copy of foolscap-0.12.7 or earlier.""" % (options.dumpfile,)) return 1 except flogfile.BadMagic as e: print >>options.stderr, textwrap.dedent("""\ Error: %s does not appear to be a flogfile. """ % (options.dumpfile,)) return 1 except ValueError, ex: print >>options.stderr, ( "truncated pickle file? (%s): %s" % (options.dumpfile, ex)) return 1 def print_header(self, e, options): stdout = options.stdout h = e["header"] if h["type"] == "incident": t = h["trigger"] self.trigger = (t["incarnation"], t["num"]) if options['verbose']: print >>stdout, e if not options["just-numbers"] and not options["verbose"]: if "versions" in h: print >>stdout, "Application versions (embedded in logfile):" versions = h["versions"] longest = max([len(name) for name in versions] + [0]) fmt = "%" + str(longest) + "s: %s" for name in sorted(versions.keys()): print >>stdout, fmt % (name, versions[name]) if "pid" in h: print >>stdout, "PID: %s" % (h["pid"],) print >>stdout def print_event(self, e, options): stdout = options.stdout short = e['from'][:8] d = e['d'] when = format_time(d['time'], options["timestamps"]) if options['just-numbers']: print >>stdout, when, d.get('num') return eid = (d["incarnation"], d["num"]) # let's mark the trigger event from incident reports with # [INCIDENT-TRIGGER] at the end of the line is_trigger = bool(self.trigger and (eid == self.trigger)) try: text = format_message(d) except: print "unformattable event", d raise t = "%s#%d " % (short, d['num']) if options['rx-time']: rx_when = format_time(e['rx_time'], options["timestamps"]) t += "rx(%s) " % rx_when t += "emit(%s)" % when else: t += "%s" % when t += ": %s" % text if options['verbose']: t += ": %r" % d if is_trigger: t += " [INCIDENT-TRIGGER]" print >>stdout, t if 'failure' in d: print >>stdout," FAILURE:" lines = str(d['failure'].get('str', d['failure'])).split("\n") for line in lines: print >>stdout, " %s" % (line,) foolscap-0.13.1/src/foolscap/logging/filter.py0000644000076500000240000001042713204511065021706 0ustar warnerstaff00000000000000 from twisted.python import usage import sys, os, bz2, time from foolscap.logging import log, flogfile from foolscap.util import move_into_place class FilterOptions(usage.Options): stdout = sys.stdout stderr = sys.stderr synopsis = "Usage: flogtool filter [options] OLDFILE NEWFILE" optParameters = [ ["after", None, None, "include events after timestamp (seconds since epoch)"], ["before", None, None, "include events before timestamp"], ["strip-facility", None, None, "remove events with the given facility prefix"], ["above", None, None, "include events at the given severity level or above"], ["from", None, None, "include events from the given tubid prefix"], ] optFlags = [ ["verbose", "v", "emit event numbers during processing (useful to isolate an unloadable event pickle"], ] def parseArgs(self, oldfile, newfile=None): self.oldfile = oldfile self.newfile = newfile if newfile is None: self.newfile = oldfile def opt_after(self, arg): self['after'] = int(arg) def opt_before(self, arg): self['before'] = int(arg) def opt_above(self, arg): try: self['above'] = int(arg) except ValueError: levelmap = {"NOISY": log.NOISY, "OPERATIONAL": log.OPERATIONAL, "UNUSUAL": log.UNUSUAL, "INFREQUENT": log.INFREQUENT, "CURIOUS": log.CURIOUS, "WEIRD": log.WEIRD, "SCARY": log.SCARY, "BAD": log.BAD, } self['above'] = levelmap[arg] class Filter: def run(self, options): stdout = options.stdout newfilename = options.newfile if options.newfile == options.oldfile: print >>stdout, "modifying event file in place" newfilename = newfilename + ".tmp" if options.newfile.endswith(".bz2"): newfile = bz2.BZ2File(newfilename, "w") else: newfile = open(newfilename, "wb") newfile.write(flogfile.MAGIC) after = options['after'] if after is not None: print >>stdout, " --after: removing events before %s" % time.ctime(after) before = options['before'] if before is not None: print >>stdout, " --before: removing events after %s" % time.ctime(before) above = options['above'] if above: print >>stdout, " --above: removing events below level %d" % above from_tubid = options['from'] if from_tubid: print >>stdout, " --from: retaining events only from tubid prefix %s" % from_tubid strip_facility = options['strip-facility'] if strip_facility is not None: print >>stdout, "--strip-facility: removing events for %s and children" % strip_facility total = 0 copied = 0 for e in flogfile.get_events(options.oldfile): if options['verbose']: if "d" in e: print >>stdout, e['d']['num'] else: print >>stdout, "HEADER" total += 1 if "d" in e: if before is not None and e['d']['time'] >= before: continue if after is not None and e['d']['time'] <= after: continue if above is not None and e['d']['level'] < above: continue if from_tubid is not None and not e['from'].startswith(from_tubid): continue if (strip_facility is not None and e['d'].get('facility', "").startswith(strip_facility)): continue copied += 1 flogfile.serialize_raw_wrapper(newfile, e) newfile.close() if options.newfile == options.oldfile: if sys.platform == "win32": # Win32 can't do an atomic rename to an existing file. try: os.unlink(options.newfile) except OSError: pass move_into_place(newfilename, options.newfile) print >>stdout, "copied %d of %d events into new file" % (copied, total) foolscap-0.13.1/src/foolscap/logging/flogfile.py0000644000076500000240000000631013204511065022204 0ustar warnerstaff00000000000000import json from contextlib import closing from twisted.python import failure class ExtendedEncoder(json.JSONEncoder): def default(self, o): if isinstance(o, failure.Failure): # this includes CopyableFailure # # pickled Failures get the following modified attributes: frames, # tb=None, stack=, pickled=1 return {"@": "Failure", "str": str(o), "repr": repr(o), "traceback": o.getTraceback(), # o.frames? .stack? .type? } try: return {"@": "UnJSONable", "message": "log.msg() was given an object that could not be encoded into JSON. I've replaced it with this UnJSONable object. The object's repr is in .repr", "repr": repr(o), } except Exception as e: try: return {"@": "Unreprable", "message": "log.msg() was given an object that could not be encoded into JSON, and when I tried to repr() it I got an error too. I've put the repr of the exception in .exception_repr", "exception_repr": repr(e), } except Exception: return {"@": "ReallyUnreprable", "message": "log.msg() was given an object that could not be encoded into JSON, and when I tried to repr() it I got an error too. That exception wasn't repr()able either. I give up. Good luck.", } def serialize_raw_header(f, header): json.dump({"header": header}, f, cls=ExtendedEncoder) f.write("\n") def serialize_header(f, type, **kwargs): header = {"header": {"type": type} } for k,v in kwargs.items(): header["header"][k] = v json.dump(header, f, cls=ExtendedEncoder) f.write("\n") def serialize_raw_wrapper(f, wrapper): json.dump(wrapper, f, cls=ExtendedEncoder) f.write("\n") def serialize_wrapper(f, ev, from_, rx_time): wrapper = {"from": from_, "rx_time": rx_time, "d": ev} json.dump(wrapper, f, cls=ExtendedEncoder) f.write("\n") MAGIC = "# foolscap flogfile v1\n" class BadMagic(Exception): """The file is not a flogfile: wrong magic number.""" class EvilPickleFlogFile(BadMagic): """This is an old (pickle-based) flogfile, and cannot be loaded safely.""" class ThisIsActuallyAFurlFileError(BadMagic): pass def get_events(fn): if fn.endswith(".bz2"): import bz2 f = bz2.BZ2File(fn, "r") # note: BZ2File in py2.6 is not a context manager else: f = open(fn, "rb") with closing(f): maybe_magic = f.read(len(MAGIC)) if maybe_magic != MAGIC: if maybe_magic.startswith("(dp0"): raise EvilPickleFlogFile() if maybe_magic.startswith("pb:"): # this happens when you point "flogtool dump" at a furlfile # (e.g. logport.furl) by mistake. Emit a useful error # message. raise ThisIsActuallyAFurlFileError raise BadMagic(repr(maybe_magic)) for line in f.readlines(): yield json.loads(line) foolscap-0.13.1/src/foolscap/logging/gatherer.py0000644000076500000240000005365213204511065022231 0ustar warnerstaff00000000000000 import os, sys, time, bz2 signal = None try: import signal except ImportError: pass from zope.interface import implements from twisted.internet import reactor, utils, defer from twisted.python import usage, procutils, filepath, log as tw_log from twisted.application import service, internet from foolscap.api import Tub, Referenceable from foolscap.logging.interfaces import RILogGatherer, RILogObserver from foolscap.logging.incident import IncidentClassifierBase, TIME_FORMAT from foolscap.logging import flogfile from foolscap.util import move_into_place class BadTubID(Exception): pass class ObsoleteGatherer(Exception): pass class GatheringBase(service.MultiService, Referenceable): # requires self.furlFile and self.tacFile to be set on the class, both of # which should be relative to the basedir. use_local_addresses = True def __init__(self, basedir): service.MultiService.__init__(self) if basedir is None: # This instance was created by a gatherer.tac file. Confirm that # we're running from the right directory (the one with the .tac # file), otherwise we'll put the logfiles in the wrong place. basedir = os.getcwd() tac = os.path.join(basedir, self.tacFile) if not os.path.exists(tac): raise RuntimeError("running in the wrong directory") self.basedir = basedir certFile = os.path.join(self.basedir, "gatherer.pem") portfile = os.path.join(self.basedir, "port") locationfile = os.path.join(self.basedir, "location") furlFile = os.path.join(self.basedir, self.furlFile) # Foolscap-0.11.0 was the last release that used # automatically-determined listening addresses and ports. New ones # (created with "flogtool create-gatherer" or # "create-incident-gathererer" now require --location and --port # arguments to provide these values. If you really don't want to # create a new one, you can write "tcp:3117" (or some other port # number of your choosing) to BASEDIR/port, and "tcp:$HOSTNAME:3117" # (with your hostname or IP address) to BASEDIR/location if (not os.path.exists(portfile) or not os.path.exists(locationfile)): raise ObsoleteGatherer("Please create a new gatherer, with both " "--port and --location") try: with open(portfile, "r") as f: port = f.read().strip() except EnvironmentError: raise ObsoleteGatherer("Please create a new gatherer, with both " "--port and --location") try: with open(locationfile, "r") as f: location = f.read().strip() except EnvironmentError: raise ObsoleteGatherer("Please create a new gatherer, with both " "--port and --location") self._tub = Tub(certFile=certFile) self._tub.setServiceParent(self) self._tub.listenOn(port) self._tub.setLocation(location) self.my_furl = self._tub.registerReference(self, furlFile=furlFile) if self.verbose: print "Gatherer waiting at:", self.my_furl class CreateGatherOptions(usage.Options): """flogtool create-gatherer GATHERER_DIRECTORY""" stdout = sys.stdout stderr = sys.stderr optFlags = [ ("bzip", "b", "Compress each output file with bzip2"), ("quiet", "q", "Don't print instructions to stdout"), ] optParameters = [ ("port", "p", "tcp:3117", "TCP port to listen on (strports string)"), ("location", "l", None, "(required) Tub location hints to use in generated FURLs. e.g. 'tcp:example.org:3117'"), ("rotate", "r", None, "Rotate the output file every N seconds."), ] def opt_port(self, port): assert not port.startswith("ssl:") assert port != "tcp:0" self["port"] = port def parseArgs(self, gatherer_dir): self["basedir"] = gatherer_dir def postOptions(self): if not self["location"]: raise usage.UsageError("--location= is mandatory") class Observer(Referenceable): implements(RILogObserver) def __init__(self, nodeid_s, gatherer): self.nodeid_s = nodeid_s # printable string self.gatherer = gatherer def remote_msg(self, d): self.gatherer.msg(self.nodeid_s, d) class GathererService(GatheringBase): # create this with 'flogtool create-gatherer BASEDIR' # run this as 'cd BASEDIR && twistd -y gatherer.tac' """Run a service that gathers logs from multiple applications. The LogGatherer sits in a corner and receives log events from many applications at once. At startup, it runs a Tub and emits the gatherer's long-term FURL. You can then configure your applications to connect to this FURL when they start and pass it a reference to their LogPublisher. The gatherer will subscribe to the publisher and save all the resulting messages in a serialized flogfile. Applications can use code like the following to create a LogPublisher and pass it to the gatherer:: def tub_ready(self): # called when the Tub is available for registerReference lp = LogPublisher('logport.furl') lp.setServiceParent(self.tub) log_gatherer_furl = self.get_config('log_gatherer.furl') if log_gatherer_furl: self.tub.connectTo(log_gatherer_furl, self._log_gatherer_connected, lp) def _log_gatherer_connected(self, rref, lp): rref.callRemote('logport', self.nodeid, lp) This LogGatherer class is meant to be run by twistd from a .tac file, but applications that want to provide the same functionality can just instantiate it with a distinct basedir= and call startService. """ implements(RILogGatherer) verbose = True furlFile = "log_gatherer.furl" tacFile = "gatherer.tac" def __init__(self, rotate, use_bzip, basedir=None): GatheringBase.__init__(self, basedir) if rotate: # int or None rotator = internet.TimerService(rotate, self.do_rotate) rotator.setServiceParent(self) bzip = None if use_bzip: bzips = procutils.which("bzip2") if bzips: bzip = bzips[0] self.bzip = bzip if signal and hasattr(signal, "SIGHUP"): signal.signal(signal.SIGHUP, self._handle_SIGHUP) self._savefile = None def _handle_SIGHUP(self, *args): reactor.callFromThread(self.do_rotate) def startService(self): # note: the rotator (if any) will fire as soon as startService is # called, since TimerService uses now=True. To deal with this, # do_rotate() tests self._savefile before doing anything else, and # we're careful to upcall to startService before we do the first # call to _open_savefile(). GatheringBase.startService(self) now = time.time() self._open_savefile(now) def format_time(self, when): return time.strftime(TIME_FORMAT, time.gmtime(when)) + "Z" def _open_savefile(self, now): new_filename = "from-%s---to-present.flog" % self.format_time(now) self._savefile_name = os.path.join(self.basedir, new_filename) self._savefile = open(self._savefile_name, "ab", 0) self._savefile.write(flogfile.MAGIC) self._starting_timestamp = now flogfile.serialize_header(self._savefile, "gatherer", start=self._starting_timestamp) def do_rotate(self): if not self._savefile: return self._savefile.close() now = time.time() from_time = self.format_time(self._starting_timestamp) to_time = self.format_time(now) new_name = "from-%s---to-%s.flog" % (from_time, to_time) new_name = os.path.join(self.basedir, new_name) move_into_place(self._savefile_name, new_name) self._open_savefile(now) if self.bzip: # we spawn an external bzip process because it's easier than # using the stdlib bz2 module and spreading the work out over # several ticks. We're trying to resume accepting log events # quickly here. We don't save the events using BZ2File because # the gatherer might be killed at any moment, and BZ2File doesn't # flush its output until the file is closed. d = utils.getProcessOutput(self.bzip, [new_name], env=os.environ) new_name = new_name + ".bz2" def _compression_error(f): print f d.addErrback(_compression_error) # note that by returning this Deferred, the rotation timer won't # start again until the bzip process finishes else: d = defer.succeed(None) d.addCallback(lambda res: new_name) return d # for tests def remote_logport(self, nodeid, publisher): # nodeid is actually a printable string nodeid_s = nodeid o = Observer(nodeid_s, self) d = publisher.callRemote("subscribe_to_all", o) d.addCallback(lambda res: None) return d # mostly for testing def msg(self, nodeid_s, d): try: flogfile.serialize_wrapper(self._savefile, d, from_=nodeid_s, rx_time=time.time()) except Exception, ex: print "GATHERER: unable to serialize %s: %s" % (d, ex) LOG_GATHERER_TACFILE = """\ # -*- python -*- # we record the path when 'flogtool create-gatherer' is run, in case flogtool # was run out of a source tree. This is somewhat fragile, of course. stashed_path = [ %(path)s] import sys needed = [p for p in stashed_path if p not in sys.path] sys.path = needed + sys.path print 'NEEDED', needed from foolscap.logging import gatherer from twisted.application import service rotate = %(rotate)s use_bzip = %(use_bzip)s gs = gatherer.GathererService(rotate, use_bzip) application = service.Application('log_gatherer') gs.setServiceParent(application) """ def create_log_gatherer(config): basedir = config["basedir"] stdout = config.stdout assert config["port"] assert config["location"] if not os.path.exists(basedir): os.makedirs(basedir) f = open(os.path.join(basedir, "port"), "w") f.write("%s\n" % config["port"]) f.close() f = open(os.path.join(basedir, "location"), "w") f.write("%s\n" % config["location"]) f.close() f = open(os.path.join(basedir, "gatherer.tac"), "w") stashed_path = "" for p in sys.path: stashed_path += " %r,\n" % p if config["rotate"]: rotate = config["rotate"] else: rotate = "None" f.write(LOG_GATHERER_TACFILE % { 'path': stashed_path, 'rotate': rotate, 'use_bzip': bool(config["bzip"]), }) f.close() if not config["quiet"]: print >>stdout, "Gatherer created in directory %s" % basedir print >>stdout, "Now run '(cd %s && twistd -y gatherer.tac)' to launch the daemon" % basedir ################### # Incident Gatherer class CreateIncidentGatherOptions(usage.Options): """flogtool create-incident-gatherer BASEDIR""" stdout = sys.stdout stderr = sys.stderr optFlags = [ ("quiet", "q", "Don't print instructions to stdout"), ] optParameters = [ ("port", "p", "tcp:3118", "TCP port to listen on (strports string)"), ("location", "l", None, "(required) Tub location hints to use in generated FURLs. e.g. 'tcp:example.org:3118'"), ] def opt_port(self, port): assert not port.startswith("ssl:") assert port != "tcp:0" self["port"] = port def parseArgs(self, basedir): self["basedir"] = basedir def postOptions(self): if not self["location"]: raise usage.UsageError("--location= is mandatory") class IncidentObserver(Referenceable): implements(RILogObserver) def __init__(self, basedir, tubid_s, gatherer, publisher, stdout): if not os.path.isdir(basedir): os.makedirs(basedir) self.basedir = filepath.FilePath(basedir) self.tubid_s = tubid_s # printable string self.gatherer = gatherer self.publisher = publisher self.stdout = stdout self.caught_up_d = defer.Deferred() self.incidents_wanted = [] self.incident_fetch_outstanding = False def connect(self): # look for a local state file, to see what incidents we've already # got statefile = self.basedir.child("latest").path latest = "" try: latest = open(statefile, "r").read().strip() except EnvironmentError: pass print >>self.stdout, "connected to %s, last known incident is %s" \ % (self.tubid_s, latest) # now subscribe to everything since then d = self.publisher.callRemote("subscribe_to_incidents", self, catch_up=True, since=latest) # for testing, we arrange for this Deferred (which governs the return # from remote_logport) to not fire until we've finished catching up # on all incidents. d.addCallback(lambda res: self.caught_up_d) return d def remote_new_incident(self, name, trigger): print >>self.stdout, "new incident", name # name= should look like "incident-2008-07-29-204211-aspkxoi". We # prevent name= from containing path metacharacters like / or : by # using FilePath later on. self.incidents_wanted.append( (name, trigger) ) self.maybe_fetch_incident() def maybe_fetch_incident(self): # only fetch one incident at a time, to keep the sender's outbound # memory usage to a reasonable level if self.incident_fetch_outstanding: return if not self.incidents_wanted: return self.incident_fetch_outstanding = True (name, trigger) = self.incidents_wanted.pop(0) print >>self.stdout, "fetching incident", name d = self.publisher.callRemote("get_incident", name) def _clear_outstanding(res): self.incident_fetch_outstanding = False return res d.addBoth(_clear_outstanding) d.addCallback(self._got_incident, name, trigger) d.addErrback(tw_log.err, "IncidentObserver.get_incident or _got_incident") d.addBoth(lambda ign: self.maybe_fetch_incident()) def _got_incident(self, incident, name, trigger): # We always save the incident to a .bz2 file. abs_fn = self.basedir.child(name).path # this prevents evil abs_fn += ".flog.bz2" # we need to record the relative pathname of the savefile, for use by # the classifiers (they write it into their output files) rel_fn = os.path.join("incidents", self.tubid_s, name) + ".flog.bz2" self.save_incident(abs_fn, incident) self.update_latest(name) self.gatherer.new_incident(abs_fn, rel_fn, self.tubid_s, incident) def save_incident(self, filename, incident): now = time.time() (header, events) = incident f = bz2.BZ2File(filename, "w") f.write(flogfile.MAGIC) flogfile.serialize_raw_header(f, header) for e in events: flogfile.serialize_wrapper(f, e, from_=self.tubid_s, rx_time=now) f.close() def update_latest(self, name): f = open(self.basedir.child("latest").path, "w") f.write(name + "\n") f.close() def remote_done_with_incident_catchup(self): self.caught_up_d.callback(None) return None class IncidentGathererService(GatheringBase, IncidentClassifierBase): # create this with 'flogtool create-incident-gatherer BASEDIR' # run this as 'cd BASEDIR && twistd -y gatherer.tac' """Run a service that gathers Incidents from multiple applications. The IncidentGatherer sits in a corner and receives incidents from many applications at once. At startup, it runs a Tub and emits the gatherer's long-term FURL. You can then configure your applications to connect to this FURL when they start and pass it a reference to their LogPublisher. The gatherer will subscribe to the publisher and save all the resulting incidents in the incidents/ directory, organized by the publisher's tubid. The gatherer will also run a set of user-supplied classifier functions on the incidents and put the filenames (one line per incident) into files in the categories/ directory. This IncidentGatherer class is meant to be run as a standalone service from bin/flogtool, but by careful subclassing and setup it could be run as part of some other application. """ implements(RILogGatherer) verbose = True furlFile = "log_gatherer.furl" tacFile = "gatherer.tac" def __init__(self, classifiers=[], basedir=None, stdout=None): GatheringBase.__init__(self, basedir) IncidentClassifierBase.__init__(self) self.classifiers.extend(classifiers) self.stdout = stdout self.incidents_received = 0 # for tests def startService(self): indir = os.path.join(self.basedir, "incidents") if not os.path.isdir(indir): os.makedirs(indir) outputdir = os.path.join(self.basedir, "classified") if not os.path.isdir(outputdir): os.makedirs(outputdir) self.add_classify_files(self.basedir) self.classify_stored_incidents(indir) GatheringBase.startService(self) def classify_stored_incidents(self, indir): stdout = self.stdout or sys.stdout print >>stdout, "classifying stored incidents" # now classify all stored incidents that aren't already classified already = set() outputdir = os.path.join(self.basedir, "classified") for category in os.listdir(outputdir): for line in open(os.path.join(outputdir, category), "r"): fn = line.strip() abs_fn = os.path.join(self.basedir, fn) already.add(abs_fn) print >>stdout, "%d incidents already classified" % len(already) count = 0 for tubid_s in os.listdir(indir): nodedir = os.path.join(indir, tubid_s) for fn in os.listdir(nodedir): if fn.startswith("incident-"): abs_fn = os.path.join(nodedir, fn) if abs_fn in already: continue incident = self.load_incident(abs_fn) rel_fn = os.path.join("incidents", tubid_s, fn) self.move_incident(rel_fn, tubid_s, incident) count += 1 print >>stdout, "done classifying %d stored incidents" % count def remote_logport(self, nodeid, publisher): # we ignore nodeid (which is a printable string), and get the tubid # from the publisher remoteReference. getRemoteTubID() protects us # from .. and / and other nasties. tubid_s = publisher.getRemoteTubID() basedir = os.path.join(self.basedir, "incidents", tubid_s) stdout = self.stdout or sys.stdout o = IncidentObserver(basedir, tubid_s, self, publisher, stdout) d = o.connect() d.addCallback(lambda res: None) return d # mostly for testing def new_incident(self, abs_fn, rel_fn, tubid_s, incident): self.move_incident(rel_fn, tubid_s, incident) self.incidents_received += 1 def move_incident(self, rel_fn, tubid_s, incident): stdout = self.stdout or sys.stdout categories = self.classify_incident(incident) for c in categories: fn = os.path.join(self.basedir, "classified", c) f = open(fn, "a") f.write(rel_fn + "\n") f.close() print >>stdout, "classified %s as [%s]" % (rel_fn, ",".join(categories)) return categories INCIDENT_GATHERER_TACFILE = """\ # -*- python -*- # we record the path when 'flogtool create-incident-gatherer' is run, in case # flogtool was run out of a source tree. This is somewhat fragile, of course. stashed_path = [ %(path)s] import sys needed = [p for p in stashed_path if p not in sys.path] sys.path = needed + sys.path print 'NEEDED', needed from foolscap.logging import gatherer from twisted.application import service gs = gatherer.IncidentGathererService() # To add a classifier function, store it in a neighboring file named # classify_*.py, in a function named classify_incident(). All such files will # be loaded at startup: # # %% cat classify_foolscap.py # import re # TUBCON_RE = re.compile(r'^Tub.connectorFinished: WEIRD, is not in \[') # def classify_incident(trigger): # # match some foolscap messages # m = trigger.get('message', '') # if TUBCON_RE.search(m): # return 'foolscap-tubconnector' # %% application = service.Application('incident_gatherer') gs.setServiceParent(application) """ def create_incident_gatherer(config): basedir = config["basedir"] stdout = config.stdout assert config["port"] assert config["location"] if not os.path.exists(basedir): os.makedirs(basedir) f = open(os.path.join(basedir, "port"), "w") f.write("%s\n" % config["port"]) f.close() f = open(os.path.join(basedir, "location"), "w") f.write("%s\n" % config["location"]) f.close() f = open(os.path.join(basedir, "gatherer.tac"), "w") stashed_path = "" for p in sys.path: stashed_path += " %r,\n" % p f.write(INCIDENT_GATHERER_TACFILE % { 'path': stashed_path, }) f.close() if not config["quiet"]: print >>stdout, "Incident Gatherer created in directory %s" % basedir print >>stdout, "Now run '(cd %s && twistd -y gatherer.tac)' to launch the daemon" % basedir foolscap-0.13.1/src/foolscap/logging/incident.py0000644000076500000240000002303013204511065022210 0ustar warnerstaff00000000000000 import sys, os.path, time, bz2 from pprint import pprint from zope.interface import implements from twisted.python import usage from twisted.internet import reactor from foolscap.logging.interfaces import IIncidentReporter from foolscap.logging import levels, app_versions, flogfile from foolscap.eventual import eventually from foolscap.util import move_into_place from foolscap import base32 TIME_FORMAT = "%Y-%m-%d--%H-%M-%S" class IncidentQualifier: """I am responsible for deciding what qualifies as an Incident. I look at the event stream and watch for a 'triggering event', then signal my handler when the events that I've seen are severe enought to warrant recording the recent history in an 'incident log file'. My event() method should be called with each event. When I declare an incident, I will call my handler's declare_incident(ev) method, with the triggering event. Since event() will be fired from an eventual-send queue, the incident will be declared slightly later than the triggering event. """ def set_handler(self, handler): self.handler = handler def check_event(self, ev): if ev['level'] >= levels.WEIRD: return True return False def event(self, ev): if self.check_event(ev) and self.handler: self.handler.declare_incident(ev) class IncidentReporter: """Once an Incident has been declared, I am responsible for making a durable record all relevant log events. I do this by creating a logfile (a series of JSON lines, one per log event dictionary) and copying everything from the history buffer into it. I can copy a small number of future events into it as well, to record what happens as the application copes with the situtation. I am responsible for just a single incident. I am created with a reference to a FoolscapLogger instance, from which I will grab the contents of the history buffer. When I have closed the incident logfile, I will notify the logger by calling their incident_recorded() method, passing it the local filename of the logfile I created and the triggering event. This can be used to notify remote subscribers about the incident that just occurred. """ implements(IIncidentReporter) TRAILING_DELAY = 5.0 # gather 5 seconds of post-trigger events TRAILING_EVENT_LIMIT = 100 # or 100 events, whichever comes first def __init__(self, basedir, logger, tubid_s): self.basedir = basedir self.logger = logger self.tubid_s = tubid_s self.active = True self.timer = None def is_active(self): return self.active def format_time(self, when): return time.strftime(TIME_FORMAT, time.gmtime(when)) + "Z" def incident_declared(self, triggering_event): self.trigger = triggering_event # choose a name for the logfile now = time.time() unique = os.urandom(4) unique_s = base32.encode(unique) self.name = "incident-%s-%s" % (self.format_time(now), unique_s) filename = self.name + ".flog" self.abs_filename = os.path.join(self.basedir, filename) self.abs_filename_bz2 = self.abs_filename + ".bz2" self.abs_filename_bz2_tmp = self.abs_filename + ".bz2.tmp" # open logfile. We use both an uncompressed one and a compressed one. self.f1 = open(self.abs_filename, "wb") self.f2 = bz2.BZ2File(self.abs_filename_bz2_tmp, "wb") # write header with triggering_event self.f1.write(flogfile.MAGIC) self.f2.write(flogfile.MAGIC) flogfile.serialize_header(self.f1, "incident", trigger=triggering_event, versions=app_versions.versions, pid=os.getpid()) flogfile.serialize_header(self.f2, "incident", trigger=triggering_event, versions=app_versions.versions, pid=os.getpid()) if self.TRAILING_DELAY is not None: # subscribe to events that occur after this one self.still_recording = True self.remaining_events = self.TRAILING_EVENT_LIMIT self.logger.addObserver(self.trailing_event) # use self.logger.buffers, copy events into logfile events = list(self.logger.get_buffered_events()) events.sort(lambda a,b: cmp(a['num'], b['num'])) for e in events: flogfile.serialize_wrapper(self.f1, e, from_=self.tubid_s, rx_time=now) flogfile.serialize_wrapper(self.f2, e, from_=self.tubid_s, rx_time=now) self.f1.flush() # the BZ2File has no flush method if self.TRAILING_DELAY is None: self.active = False eventually(self.finished_recording) else: # now we wait for the trailing events to arrive self.timer = reactor.callLater(self.TRAILING_DELAY, self.stop_recording) def trailing_event(self, ev): if not self.still_recording: return self.remaining_events -= 1 if self.remaining_events >= 0: now = time.time() flogfile.serialize_wrapper(self.f1, ev, from_=self.tubid_s, rx_time=now) flogfile.serialize_wrapper(self.f2, ev, from_=self.tubid_s, rx_time=now) return self.stop_recording() def new_trigger(self, ev): # it is too late to add this to the header. We could add it to a # trailer, though. pass def stop_recording(self): self.still_recording = False self.active = False if self.timer and self.timer.active(): self.timer.cancel() self.logger.removeObserver(self.trailing_event) # Observers are notified through an eventually() call, so we might # get a few more after the observer is removed. We use # self.still_recording to hush them. eventually(self.finished_recording) def finished_recording(self): self.f2.close() move_into_place(self.abs_filename_bz2_tmp, self.abs_filename_bz2) # the compressed logfile has closed successfully. We no longer care # about the uncompressed one. self.f1.close() os.unlink(self.abs_filename) # now we can tell the world about our new incident report eventually(self.logger.incident_recorded, self.abs_filename_bz2, self.name, self.trigger) class NonTrailingIncidentReporter(IncidentReporter): TRAILING_DELAY = None class ClassifyOptions(usage.Options): stdout = sys.stdout stderr = sys.stderr synopsis = "Usage: flogtool classify-incident [options] INCIDENTFILE.." optFlags = [ ("verbose", "v", "show trigger details for unclassifiable incidents"), ] optParameters = [ ("classifier-directory", "c", ".", "directory with classify_*.py functions to import"), ] def parseArgs(self, *files): self.files = files class IncidentClassifierBase: def __init__(self): self.classifiers = [] def add_classifier(self, f): # there are old .tac files that call this explicitly self.classifiers.append(f) def add_classify_files(self, plugindir): plugindir = os.path.expanduser(plugindir) for fn in os.listdir(plugindir): if not (fn.startswith("classify_") and fn.endswith(".py")): continue f = open(os.path.join(plugindir, fn), "r") localdict = {} exec f in localdict self.add_classifier(localdict["classify_incident"]) def load_incident(self, abs_fn): assert abs_fn.endswith(".bz2") events = flogfile.get_events(abs_fn) header = next(events)["header"] wrapped_events = [event["d"] for event in events] return (header, wrapped_events) def classify_incident(self, incident): categories = set() for f in self.classifiers: (header, events) = incident trigger = header["trigger"] c = f(trigger) if c: # allow the classifier to return None, or [], or ["foo"] if isinstance(c, str): c = [c] # or just "foo" categories.update(c) if not categories: categories.add("unknown") return categories class IncidentClassifier(IncidentClassifierBase): def run(self, options): self.add_classify_files(options["classifier-directory"]) out = options.stdout for f in options.files: abs_fn = os.path.expanduser(f) incident = self.load_incident(abs_fn) categories = self.classify_incident(incident) print >>out, "%s: %s" % (f, ",".join(sorted(categories))) if list(categories) == ["unknown"] and options["verbose"]: (header, events) = incident trigger = header["trigger"] from foolscap.logging.log import format_message print >>out, format_message(trigger) pprint(trigger, stream=out) if 'failure' in trigger: print >>out," FAILURE:" lines = str(trigger['failure']).split("\n") for line in lines: print >>out, " %s" % (line,) print >>out, "" foolscap-0.13.1/src/foolscap/logging/interfaces.py0000644000076500000240000001425113204455737022560 0ustar warnerstaff00000000000000 from zope.interface import Interface from foolscap.remoteinterface import RemoteInterface from foolscap.schema import DictOf, ListOf, Any, Optional, ChoiceOf TubID = str # printable, base32 encoded Incarnation = (str, ChoiceOf(str, None)) Header = DictOf(str, Any()) Event = DictOf(str, Any()) # this has message:, level:, facility:, etc EventWrapper = DictOf(str, Any()) # this has from:, rx_time:, and d: class RILogObserver(RemoteInterface): __remote_name__ = "RILogObserver.foolscap.lothar.com" def msg(logmsg=Event): return None def done(): return None def new_incident(name=str, trigger=Event): # should this give (tubid, incarnation, trigger) like list_incidents? return None def done_with_incident_catchup(): return None class RILogFile(RemoteInterface): __remote_name__ = "RILogFile.foolscap.lothar.com" def get_header(): # (tubid, incarnation, # (first_event: number, time), (last_event: number, time), # num_events, # level_map, # maps string severity to count of messages # ) return (TubID, int, (int, int), (int, int), int, DictOf(str, int)) def get_events(receiver=RILogObserver): """The designated receiver will be sent every event in the logfile, followed by a done() call.""" return None class RISubscription(RemoteInterface): __remote_name__ = "RISubscription.foolscap.lothar.com" def unsubscribe(): """Cancel a subscription. Once this method has been completed (and its Deferred has fired), no further messages will be received by the observer (i.e. the response to unsubscribe() will wait until all pending messages have been queued). This method is idempotent: calling it multiple times has the same effect as calling it just once.""" return None class RILogPublisher(RemoteInterface): __remote_name__ = "RILogPublisher.foolscap.lothar.com" def get_versions(): return DictOf(str, str) def get_pid(): return int def subscribe_to_all(observer=RILogObserver, catch_up=Optional(bool, False)): """ Call unsubscribe() on the returned RISubscription object to stop receiving messages. """ return RISubscription def unsubscribe(subscription=Any()): # NOTE: this is deprecated. Use subscription.unsubscribe() instead. # I don't know how to get the constraint right: unsubscribe() should # accept return value of subscribe_to_all() return None def enumerate_logfiles(): return ListOf(RILogFile) # Incident support def list_incidents(since=Optional(str, "")): """Return a dict that maps an 'incident name' (a string of the form 'incident-TIMESTAMP-UNIQUE') to the triggering event (a single event dictionary). The incident name can be passed to get_incident() to obtain the list of events (including header) contained inside the incident report. Incident names will sort in chronological order. If the optional since= argument is provided, then this will only return incident names that are alphabetically greater (and thus chronologically later) than the given string. This can be used to poll an application for incidents that have occurred since a previous query. For real-time reporting, use subscribe_to_incidents() instead. """ return DictOf(str, Event) def subscribe_to_incidents(observer=RILogObserver, catch_up=Optional(bool, False), since=Optional(str, "")): """Subscribe to hear about new Incidents, optionally catching up on old ones. Each new Incident will be reported by name+trigger to the observer by a new_incident() message. This message will be sent after the incident reporter has finished working (usually a few seconds after the triggering event). If catch_up=True, then old Incidents will be sent to the observer before any new ones are reported. When the publisher has finished sending the names of all old events, it will send a done_with_incident_catchup() message to the observer. Only old Incidents with a name that is alphabetically greater (and thus later) than the since= argument will be sent. Use since='' to catch up on all old Incidents. Call unsubscribe() on the returned RISubscription object to stop receiving messages. """ return RISubscription def get_incident(incident_name=str): """Given an incident name, return the header dict and list of event dicts for that incident.""" # note that this puts all the events in memory at the same time, but # we expect the logfiles to be of a reasonable size: not much larger # than the circular buffers that we keep around anyways. return (Header, ListOf(Event)) class RILogGatherer(RemoteInterface): __remote_name__ = "RILogGatherer.foolscap.lothar.com" def logport(nodeid=TubID, logport=RILogPublisher): return None class IIncidentReporter(Interface): def incident_declared(triggering_event): """This is called when an Incident needs to be recorded.""" def new_trigger(triggering_event): """This is called when a triggering event occurs while an incident is already being reported. If the event happened later, it would trigger a new incident. Since it overlapped with the existing incident, it will just be added to that incident. The triggering event will also be reported through the usual event-publish-subscribe mechanism. This method is provided to give the reporter the opportunity to mark the event somehow, for the benefit of incident-file analysis tools. """ def is_active(): """Returns True if the reporter is still running. While in this state, new Incident triggers will be passed to the existing reporter instead of causing a new Incident to be declared. This will tend to coalesce back-to-back problems into a single Incident.""" foolscap-0.13.1/src/foolscap/logging/levels.py0000644000076500000240000000035112766553111021720 0ustar warnerstaff00000000000000 import logging NOISY = logging.DEBUG # 10 OPERATIONAL = logging.INFO # 20 UNUSUAL = logging.INFO+3 INFREQUENT = logging.INFO+5 CURIOUS = logging.INFO+8 WEIRD = logging.WARNING # 30 SCARY = logging.WARNING+5 BAD = logging.ERROR # 40 foolscap-0.13.1/src/foolscap/logging/log.py0000644000076500000240000004542213204511065021205 0ustar warnerstaff00000000000000 import os, sys, time, weakref, binascii import traceback import collections from twisted.python import log as twisted_log from twisted.python import failure from foolscap import eventual from foolscap.logging.interfaces import IIncidentReporter from foolscap.logging.incident import IncidentQualifier, IncidentReporter from foolscap.logging import app_versions, flogfile from foolscap.logging.levels import NOISY, OPERATIONAL, UNUSUAL, \ INFREQUENT, CURIOUS, WEIRD, SCARY, BAD llmap = {} try: import logging as py_logging from twisted.logger import LogLevel # added in Twisted-15.2.0 # twisted.logger._stdlib.toStdlibLogLevelMapping is private, alas llmap = { LogLevel.debug: py_logging.DEBUG, # == NOISY LogLevel.info: py_logging.INFO, # == OPERATIONAL LogLevel.warn: py_logging.WARNING, # == WEIRD LogLevel.error: py_logging.ERROR, # == BAD, LogLevel.critical: py_logging.CRITICAL, # == BAD+10 } except ImportError: pass # Twisted < 15.2.0 # hush pyflakes, these are imported to be available to other callers _unused = [NOISY, OPERATIONAL, UNUSUAL, INFREQUENT, CURIOUS, WEIRD, SCARY, BAD] def format_message(e): try: if "format" in e: assert isinstance(e['format'], (str,unicode)) return e['format'] % e elif "args" in e: assert "message" in e assert isinstance(e['message'], (str,unicode)) return e['message'] % e['args'] elif "message" in e: assert isinstance(e['message'], (str,unicode)) return e['message'] else: return "" except (ValueError, TypeError): return e.get('message', "[no message]") + " [formatting failed]" class Count: """A fixed version of itertools.count . This class counts up from zero, just like the Python 2.5.2 docs claim that itertools.count() does, but this class does not overflow with an error like itertools.count() does: File 'foolscap/logging/log.py', line 137, in msg num = self.seqnum.next() exceptions.OverflowError: cannot count beyond PY_SSIZE_T_MAX """ def __init__(self, firstval=0): self.n = firstval - 1 def next(self): self.n += 1 return self.n class FoolscapLogger: DEFAULT_SIZELIMIT = 100 DEFAULT_THRESHOLD = NOISY MAX_RECORDED_INCIDENTS = 20 # records filenames of incident logfiles def __init__(self): self.incarnation = self.get_incarnation() self.seqnum = Count() self.facility_explanations = {} self.buffer_sizes = {} # k: facility or None, v: dict(level->sizelimit) self.buffer_sizes[None] = {} self.buffers = {} # k: facility or None, v: dict(level->deque) self.thresholds = {} self._observers = [] self._immediate_observers = [] self._immediate_incident_observers = [] self.logdir = None # nowhere to put our incidents self.inactive_incident_qualifier = IncidentQualifier() self.active_incident_qualifier = None self.incident_reporter_factory = IncidentReporter self.active_incident_reporter_weakref = None self.incidents_declared = 0 self.incidents_recorded = 0 self.recent_recorded_incidents = [] def get_incarnation(self): unique = binascii.b2a_hex(os.urandom(8)) sequential = None return (unique, sequential) def addObserver(self, observer): self._observers.append(observer) def removeObserver(self, observer): self._observers.remove(observer) def addImmediateObserver(self, observer): # by using this, you solemly swear that your observer will not raise # an exception, nor will it recurse or cause more log messages to be # emitted. Immediate Observers are notified without an eventual-send. self._immediate_observers.append(observer) def removeImmediateObserver(self, observer): self._immediate_observers.remove(observer) def setLogDir(self, directory): # TODO: change self.incarnation to reflect next seqnum self.logdir = os.path.abspath(os.path.expanduser(directory)) if not os.path.isdir(self.logdir): os.makedirs(self.logdir) self.activate_incident_qualifier() def setIncidentQualifier(self, iq): assert iq.event self.deactivate_incident_qualifier() self.inactive_incident_qualifier = iq if self.logdir: self.activate_incident_qualifier() def deactivate_incident_qualifier(self): if self.active_incident_qualifier: self.active_incident_qualifier.set_handler(None) self.active_incident_qualifier = None def activate_incident_qualifier(self): self.active_incident_qualifier = self.inactive_incident_qualifier self.active_incident_qualifier.set_handler(self) def setIncidentReporterFactory(self, ir): assert IIncidentReporter.implementedBy(ir) self.incident_reporter_factory = ir def addImmediateIncidentObserver(self, observer): self._immediate_incident_observers.append(observer) def removeImmediateIncidentObserver(self, observer): self._immediate_incident_observers.remove(observer) def explain_facility(self, facility, description): self.facility_explanations[facility] = description def set_buffer_size(self, level, sizelimit, facility=None): if facility not in self.buffer_sizes: self.buffer_sizes[facility] = {} self.buffer_sizes[facility][level] = sizelimit def set_generation_threshold(self, level, facility=None): self.thresholds[facility] = level def get_generation_threshold(self, facility=None): return self.thresholds.get(facility, self.DEFAULT_THRESHOLD) def msg(self, *args, **kwargs): """ @param parent: the event number of the most direct parent of this event @param facility: the slash-joined facility name, or None @param level: the numeric severity level, like NOISY or SCARY @param stacktrace: a string stacktrace, or True to generate one @returns: the event number for this logevent, intended to be passed to parent= in a subsequent call to msg() """ if "num" not in kwargs: num = self.seqnum.next() kwargs['num'] = num else: num = kwargs['num'] try: self._msg(*args, **kwargs) except Exception as e: try: errormsg = ("internal error in log._msg," " args=%r, kwargs=%r, exception=%r" % (args, kwargs, e)) self._msg(errormsg, num=num, level=WEIRD, facility="foolscap/internal-error") except: pass # bummer return num def _msg(self, *args, **kwargs): facility = kwargs.get('facility') if "level" not in kwargs: kwargs['level'] = OPERATIONAL level = kwargs["level"] threshold = self.get_generation_threshold(facility) if level < threshold: return # not worth logging event = kwargs # kwargs always has 'num' if "format" in event: pass elif "message" in event: event['message'] = str(event['message']) elif args: event['message'], posargs = str(args[0]), args[1:] if posargs: event['args'] = posargs else: event['message'] = "" if "time" not in event: event['time'] = time.time() if event.get('stacktrace', False) is True: event['stacktrace'] = traceback.format_stack() event['incarnation'] = self.incarnation self.add_event(facility, level, event) def err(self, _stuff=None, _why=None, **kw): """ Write a failure to the log. """ if _stuff is None: _stuff = failure.Failure() if isinstance(_stuff, failure.Failure): return self.msg(failure=_stuff, why=_why, isError=1, **kw) elif isinstance(_stuff, Exception): return self.msg(failure=failure.Failure(_stuff), why=_why, isError=1, **kw) else: return self.msg(repr(_stuff), why=_why, isError=1, **kw) def add_event(self, facility, level, event): # send to observers for o in self._immediate_observers: o(event) for o in self._observers: eventual.eventually(o, event) # buffer locally d1 = self.buffers.get(facility) if not d1: d1 = self.buffers[facility] = {} buffer = d1.get(level) if not buffer: buffer = d1[level] = collections.deque() buffer.append(event) # enforce size limits on local buffers d2 = self.buffer_sizes.get(facility) if d2: sizelimit = d2.get(level, self.DEFAULT_SIZELIMIT) else: sizelimit = self.DEFAULT_SIZELIMIT while len(buffer) > sizelimit: buffer.popleft() # check with incident reporter. This is done synchronously rather # than via the usual eventual-send to allow the application to do: # log.msg("abandon ship", level=log.BAD) # sys.exit(1) # # This means the IncidentReporter will do most of its work right # here. The reporter is not allowed to make any foolscap calls, and # the call to incident_recorded() is required to pass through an # eventual-send. if self.active_incident_qualifier: # this might call declare_incident self.active_incident_qualifier.event(event) def declare_incident(self, triggering_event): self.incidents_declared += 1 ir = self.get_active_incident_reporter() if ir: ir.new_trigger(triggering_event) return if self.logdir: # just in case ir = self.incident_reporter_factory(self.logdir, self, "local") self.active_incident_reporter_weakref = weakref.ref(ir) ir.incident_declared(triggering_event) # this takes a few seconds def incident_recorded(self, filename, name, trigger): # 'name' is incident-TIMESTAMP-UNIQUE, whereas filename is an # absolute pathname to the NAME.flog.bz2 file. self.incidents_recorded += 1 self.recent_recorded_incidents.append(filename) while len(self.recent_recorded_incidents) > self.MAX_RECORDED_INCIDENTS: self.recent_recorded_incidents.pop(0) # publish these to interested parties for o in self._immediate_incident_observers: o(name, trigger) def get_active_incident_reporter(self): if self.active_incident_reporter_weakref: ir = self.active_incident_reporter_weakref() if ir and ir.is_active(): return ir return None def setLogPort(self, logport): self._logport = logport def getLogPort(self): return self._logport def get_buffered_events(self): # iterates over all current log events in no particular order. The # caller should sort them by event number. If this isn't iterated # quickly enough, more events may arrive. for facility,b1 in self.buffers.iteritems(): for level,q in b1.iteritems(): for event in q: yield event theLogger = FoolscapLogger() # def msg(stuff): msg = theLogger.msg err = theLogger.err setLogDir = theLogger.setLogDir explain_facility = theLogger.explain_facility set_buffer_size = theLogger.set_buffer_size set_generation_threshold = theLogger.set_generation_threshold get_generation_threshold = theLogger.get_generation_threshold # code to bridge twisted.python.log.msg() to foolscap class TwistedLogBridge: def __init__(self, tubID=None, foolscap_logger=theLogger): self.tubID = tubID self.logger = foolscap_logger # we currently depend on Twisted >= 10.1.0, so we can use # t.p.log.textFromEventDict . However we cannot add ourselves as a # new-style observer (t.l.globalLogPublisher.addObserver()) because that # wasn't added until 15.2.0. So even on newer Twisteds, we'll be wrapped # by t.l._legacy.LegacyLogObserverWrapper def observer(self, d): # Twisted will remove this for us if it fails. if "from-foolscap" in d: return # Twisted-8.2.0's ILogObserver tends to give these keys: # log.msg(): message=*args, system, time, isError=False # log.err() adds: isError=True, failure, why # plus any kwargs provided to msg()/err(), like format= # With Twisted-15.2.0 we are wrapped by # t.l._legacy.LegacyLogObserverWrapper , so we still get those keys, # but we'll also see some log_* keys that the new logging system # adds. Some of the new keys are non-serializable. # So we stringify the Twisted event right now, and produce a new # event with a small set of known keys. message = twisted_log.textFromEventDict(d) kwargs = {'tubID': self.tubID, 'from-twisted': True} # log_level was added in 15.2.0 if "log_level" in d: # d["log_level"] might be a non-serializable ConstantString. # Transform it into the corresponding (integer) Foolscap log # level. log_level = d.pop("log_level") new_log_level = llmap.get(log_level, log_level) if not isinstance(new_log_level, (int, long, str, unicode, bool)): # it was something weird: just stringify it in-place new_log_level = str(new_log_level) kwargs["level"] = new_log_level # foolscap level, not twisted # d["isError"]=1 for pre-15.2.0 calls to t.p.log.err(), and is # synthesized by the LegacyLogObserverWrapper for post-15.2.0 calls # when the event includes a Failure or a log_level of "error" or # "critical". In post-15.2.0 calls, "time" and "system" are copied # from log_time and log_system, and "log_namespace" seems pretty # useful. for k in ["isError", "why", "time", "system", "log_namespace"]: if k in d: kwargs[k] = d[k] # we don't copy d["failure"] or d["why"], because its text should # already be copied into "message". self.logger.msg(message, **kwargs) _bridges = {} # maps (twisted_logger,foolscap_logger) to TwistedLogBridge def bridgeLogsFromTwisted(tubID=None, twisted_logger=twisted_log.theLogPublisher, foolscap_logger=theLogger): """Called without arguments, this arranges for all twisted log messages to be bridged into the default foolscap logger. I can also be called with a specific twisted and/or foolscap logger, mostly for unit tests that don't want to modify the default instances. For their benefit, I return the bridge. I only add one bridge per (twisted_logger,foolscap_logger) pair, even if called multiple times with different TubIDs, so multiple Tubs in a single process that all call tub.setOption(bridge-twisted-logs) will only see one foolscap copy of each twisted event, with the first Tub's tubID. """ key = (twisted_logger, foolscap_logger) if key not in _bridges: tlb = TwistedLogBridge(tubID, foolscap_logger) _bridges[key] = tlb twisted_logger.addObserver(tlb.observer) return _bridges[key] def unbridgeLogsFromTwisted(twisted_logger, tlb): # for tests foolscap_logger = tlb.logger key = (twisted_logger, foolscap_logger) del _bridges[key] twisted_logger.removeObserver(tlb.observer) def bridgeLogsToTwisted(filter=None, foolscap_logger=theLogger, twisted_logger=twisted_log): # foolscap_logger and twisted_logger are for testing purposes def non_foolscap_operational_or_better(e): if e.get("facility","").startswith("foolscap"): return False if e['level'] < OPERATIONAL: return False return True if not filter: filter = non_foolscap_operational_or_better def _to_twisted(event): if "from-twisted" in event: return if not filter(event): return args = {"from-foolscap": True, "num": event["num"], "level": event["level"], } twisted_logger.msg(format_message(event), **args) foolscap_logger.addObserver(_to_twisted) class LogFileObserver: def __init__(self, filename, level=OPERATIONAL): if filename.endswith(".bz2"): import bz2 self._logFile = bz2.BZ2File(filename, "w") else: self._logFile = open(filename, "wb") self._level = level self._logFile.write(flogfile.MAGIC) flogfile.serialize_header(self._logFile, "log-file-observer", versions=app_versions.versions, pid=os.getpid(), threshold=level) def stop_on_shutdown(self): from twisted.internet import reactor reactor.addSystemEventTrigger("after", "shutdown", self._stop) def msg(self, event): threshold = self._level #if event.get('facility', '').startswith('foolscap'): # threshold = UNUSUAL if event['level'] >= threshold: flogfile.serialize_wrapper(self._logFile, event, from_="local", rx_time=time.time()) def _stop(self): self._logFile.close() del self._logFile # remove the key, so any child processes won't try to log to (and thus # clobber) the same file. This doesn't always seem to work reliably # (allmydata.test.test_runner.RunNode.test_client uses os.system and the # child process still has $FLOGFILE set). _flogfile = os.environ.pop("FLOGFILE", None) if _flogfile: try: _floglevel = int(os.environ.get("FLOGLEVEL", str(OPERATIONAL))) lfo = LogFileObserver(_flogfile, _floglevel) lfo.stop_on_shutdown() theLogger.addObserver(lfo.msg) #theLogger.set_generation_threshold(UNUSUAL, "foolscap.negotiation") except IOError: print >>sys.stderr, "FLOGFILE: unable to write to %s, ignoring" % \ (_flogfile,) if "FLOGTWISTED" in os.environ: bridgeLogsFromTwisted() if "FLOGTOTWISTED" in os.environ: _floglevel = int(os.environ.get("FLOGLEVEL", str(OPERATIONAL))) def non_foolscap_FLOGLEVEL_or_better(e): if e.get("facility","").startswith("foolscap"): return False if e['level'] < _floglevel: return False return True bridgeLogsToTwisted(filter=non_foolscap_FLOGLEVEL_or_better) foolscap-0.13.1/src/foolscap/logging/publish.py0000644000076500000240000002275613204511065022077 0ustar warnerstaff00000000000000 import os from collections import deque from zope.interface import implements from twisted.python import filepath from foolscap.referenceable import Referenceable from foolscap.logging.interfaces import RISubscription, RILogPublisher from foolscap.logging import app_versions, flogfile from foolscap.eventual import eventually class Subscription(Referenceable): implements(RISubscription) # used as a marker, and as an unsubscribe() method. We use this to manage # the outbound size-limited queue. MAX_QUEUE_SIZE = 2000 MAX_IN_FLIGHT = 10 def __init__(self, observer, logger): self.observer = observer self.logger = logger self.subscribed = False self.queue = deque() self.in_flight = 0 self.marked_for_sending = False #self.messages_dropped = 0 def subscribe(self, catch_up): self.subscribed = True # If we have to discard messages, discard them as early as possible, # and provide backpressure. So we add our method as an "immediate # observer" instead of a regular one. self.logger.addImmediateObserver(self.send) self._nod_marker = self.observer.notifyOnDisconnect(self.unsubscribe) if catch_up: # send any catch-up events in a single batch, before we allow any # other events to be generated (and sent). This lets the # subscriber see events in sorted order. We bypass the bounded # queue for this. events = list(self.logger.get_buffered_events()) events.sort(lambda a,b: cmp(a['num'], b['num'])) for e in events: self.observer.callRemoteOnly("msg", e) def unsubscribe(self): if self.subscribed: self.logger.removeImmediateObserver(self.send) self.observer.dontNotifyOnDisconnect(self._nod_marker) self.subscribed = False def remote_unsubscribe(self): return self.unsubscribe() def send(self, event): if len(self.queue) < self.MAX_QUEUE_SIZE: self.queue.append(event) else: # preserve old messages, discard new ones. #self.messages_dropped += 1 pass if not self.marked_for_sending: self.marked_for_sending = True eventually(self.start_sending) def start_sending(self): self.marked_for_sending = False while self.queue and (self.MAX_IN_FLIGHT - self.in_flight > 0): event = self.queue.popleft() self.in_flight += 1 d = self.observer.callRemote("msg", event) d.addCallback(self._event_received) d.addErrback(self._error) def _event_received(self, res): self.in_flight -= 1 # the following would be nice to have, but requires very careful # analysis to avoid recursion, reentrancy, or even more overload #if self.messages_dropped and not self.queue: # count = self.messages_dropped # self.messages_dropped = 0 # log.msg(format="log-publisher: %(dropped)d messages dropped", # dropped=count, # facility="foolscap.log.publisher", # level=log.UNUSUAL) if not self.marked_for_sending: self.marked_for_sending = True eventually(self.start_sending) def _error(self, f): #print "PUBLISH FAILED: %s" % f self.unsubscribe() class IncidentSubscription(Referenceable): implements(RISubscription) def __init__(self, observer, logger, publisher): self.observer = observer self.logger = logger self.publisher = publisher self.subscribed = False def subscribe(self, catch_up=False, since=None): self.subscribed = True self.logger.addImmediateIncidentObserver(self.send) self._nod_marker = self.observer.notifyOnDisconnect(self.unsubscribe) if catch_up: self.catch_up(since) def catch_up(self, since): new = dict(self.publisher.list_incident_names(since)) for name in sorted(new.keys()): fn = new[name] trigger = self.publisher.get_incident_trigger(fn) if trigger: self.observer.callRemoteOnly("new_incident", name, _keys_to_bytes(trigger)) self.observer.callRemoteOnly("done_with_incident_catchup") def unsubscribe(self): if self.subscribed: self.logger.removeImmediateIncidentObserver(self.send) self.observer.dontNotifyOnDisconnect(self._nod_marker) self.subscribed = False def remote_unsubscribe(self): return self.unsubscribe() def send(self, name, trigger): d = self.observer.callRemote("new_incident", name, trigger) d.addErrback(self._error) def _error(self, f): print "INCIDENT PUBLISH FAILED: %s" % f self.unsubscribe() def _keys_to_bytes(d): # the interfaces.Header and Event schema require the keys to be bytes # ("str", since we're in py2), but we get unicode since we're pulling # from a JSON-serialized incident file. These keys are fixed strings # like (message, level, facility, from, rx_time, d). Encode to ASCII to # make this clear. The user-provided data lives in the *values* of # these dicts, which are unconstrained (the schemas use Any()) return dict([ (k.encode("ascii"), v) for (k,v) in d.iteritems()]) class LogPublisher(Referenceable): """Publish log events to anyone subscribed to our 'logport'. This class manages the subscriptions. Enable this by asking the Tub for a reference to me, or by telling the Tub to offer me to a log gatherer:: lp = tub.getLogPort() rref.callRemote('have_a_logport', lp) print 'logport at:', tub.getLogPortFURL() tub.setOption('log-gatherer-furl', gatherer_furl) Running 'flogtool tail LOGPORT_FURL' will connect to the logport and print all events that subsequently get logged. To make the logport use the same furl from one run to the next, give the Tub a filename where it can store the furl. Make sure you do this before touching the logport:: logport_furlfile = 'logport.furl' tub.setOption('logport-furlfile', logport_furlfile) If you're using one or more LogGatherers, pass their FURLs into the Tub with tub.setOption('log-gatherer-furl'), or pass the name of a file where it is stored with tub.setOption('log-gatherer-furlfile'). This will cause the Tub to connect to the gatherer and grant it access to the logport. """ implements(RILogPublisher) # the 'versions' dict used to live here in LogPublisher, but now it lives # in foolscap.logging.app_versions and should be accessed from there. # This copy remains for backwards-compatibility. versions = app_versions.versions def __init__(self, logger): self._logger = logger logger.setLogPort(self) def remote_get_versions(self): return app_versions.versions def remote_get_pid(self): return os.getpid() def remote_subscribe_to_all(self, observer, catch_up=False): s = Subscription(observer, self._logger) eventually(s.subscribe, catch_up) # allow the call to return before we send them any events return s def remote_unsubscribe(self, s): return s.unsubscribe() def trim(self, s, *suffixes): for suffix in suffixes: if s.endswith(suffix): s = s[:-len(suffix)] return s def list_incident_names(self, since=""): # yields (name, absfilename) pairs basedir = self._logger.logdir for fn in os.listdir(basedir): if fn.startswith("incident") and not fn.endswith(".tmp"): basename = self.trim(fn, ".bz2", ".flog") if basename > since: yield (basename, os.path.join(basedir, fn)) def get_incident_trigger(self, abs_fn): events = flogfile.get_events(abs_fn) try: header = next(iter(events)) except (EOFError, ValueError): return None assert header["header"]["type"] == "incident" trigger = header["header"]["trigger"] return trigger def remote_list_incidents(self, since=""): incidents = {} for (name,fn) in self.list_incident_names(since): trigger = self.get_incident_trigger(fn) if trigger: incidents[name] = _keys_to_bytes(trigger) return incidents def remote_get_incident(self, name): if not name.startswith("incident"): raise KeyError("bad incident name %s" % name) incident_dir = filepath.FilePath(self._logger.logdir) abs_fn = incident_dir.child(name).path + ".flog" try: fn = abs_fn + ".bz2" if not os.path.exists(fn): fn = abs_fn events = flogfile.get_events(fn) # note the generator isn't actually cycled yet, not until next() header = _keys_to_bytes(next(events)["header"]) except EnvironmentError: raise KeyError("no incident named %s" % name) wrapped_events = [_keys_to_bytes(event["d"]) for event in events] return (header, wrapped_events) def remote_subscribe_to_incidents(self, observer, catch_up=False, since=""): s = IncidentSubscription(observer, self._logger, self) eventually(s.subscribe, catch_up, since) # allow the call to return before we send them any events return s foolscap-0.13.1/src/foolscap/logging/tail.py0000644000076500000240000001342513204511065021353 0ustar warnerstaff00000000000000 import os, sys, time from zope.interface import implements from twisted.internet import reactor from twisted.python import usage from foolscap import base32 from foolscap.api import Tub, Referenceable, fireEventually from foolscap.logging import log, flogfile from foolscap.referenceable import SturdyRef from foolscap.util import format_time, FORMAT_TIME_MODES from interfaces import RILogObserver def short_tubid_b2a(tubid): return base32.encode(tubid)[:8] class LogSaver(Referenceable): implements(RILogObserver) def __init__(self, nodeid_s, savefile): self.nodeid_s = nodeid_s self.f = savefile # we own this, and may close it self.f.write(flogfile.MAGIC) def emit_header(self, versions, pid): flogfile.serialize_header(self.f, "tail", versions=versions, pid=pid) def remote_msg(self, d): try: flogfile.serialize_wrapper(self.f, d, from_=self.nodeid_s, rx_time=time.time()) except Exception, ex: print "GATHERER: unable to serialize %s: %s" % (d, ex) def disconnected(self): self.f.close() del self.f class TailOptions(usage.Options): synopsis = "Usage: flogtool tail (LOGPORT.furl/furlfile/nodedir)" optFlags = [ ("verbose", "v", "Show all event arguments"), ("catch-up", "c", "Catch up with recent events"), ] optParameters = [ ("save-to", "s", None, "Save events to the given file. The file will be overwritten."), ("timestamps", "t", "short-local", "Format for timestamps: " + " ".join(FORMAT_TIME_MODES)), ] def opt_timestamps(self, arg): if arg not in FORMAT_TIME_MODES: raise usage.UsageError("--timestamps= must be one of (%s)" % ", ".join(FORMAT_TIME_MODES)) self["timestamps"] = arg def parseArgs(self, target): if target.startswith("pb:"): self.target_furl = target elif os.path.isfile(target): self.target_furl = open(target, "r").read().strip() elif os.path.isdir(target): fn = os.path.join(target, "logport.furl") self.target_furl = open(fn, "r").read().strip() else: raise RuntimeError("Can't use tail target: %s" % target) class LogPrinter(Referenceable): implements(RILogObserver) def __init__(self, options, target_tubid_s, output=sys.stdout): self.options = options self.saver = None if options["save-to"]: self.saver = LogSaver(target_tubid_s[:8], open(options["save-to"], "wb")) self.output = output def got_versions(self, versions, pid=None): print >>self.output, "Remote Versions:" for k in sorted(versions.keys()): print >>self.output, " %s: %s" % (k, versions[k]) if self.saver: self.saver.emit_header(versions, pid) def remote_msg(self, d): if self.options['verbose']: self.simple_print(d) else: self.formatted_print(d) if self.saver: self.saver.remote_msg(d) def simple_print(self, d): print >>self.output, d def formatted_print(self, d): time_s = format_time(d['time'], self.options["timestamps"]) msg = log.format_message(d) level = d.get('level', log.OPERATIONAL) tubid = "" # TODO print >>self.output, "%s L%d [%s]#%d %s" % (time_s, level, tubid, d["num"], msg) if 'failure' in d: print >>self.output, " FAILURE:" lines = str(d['failure']).split("\n") for line in lines: print >>self.output, " %s" % (line,) class LogTail: def __init__(self, options): self.options = options def run(self, target_furl): target_tubid = SturdyRef(target_furl).getTubRef().getTubID() d = fireEventually(target_furl) d.addCallback(self.start, target_tubid) d.addErrback(self._error) print "starting.." reactor.run() def _error(self, f): print "ERROR", f reactor.stop() def start(self, target_furl, target_tubid): print "Connecting.." self._tub = Tub() self._tub.startService() self._tub.connectTo(target_furl, self._got_logpublisher, target_tubid) def _got_logpublisher(self, publisher, target_tubid): d = publisher.callRemote("get_pid") def _announce(pid_or_failure): if isinstance(pid_or_failure, int): print "Connected (to pid %d)" % pid_or_failure return pid_or_failure else: # the logport is probably foolscap-0.2.8 or earlier and # doesn't offer get_pid() print "Connected (unable to get pid)" return None d.addBoth(_announce) publisher.notifyOnDisconnect(self._lost_logpublisher) lp = LogPrinter(self.options, target_tubid) def _ask_for_versions(pid): d = publisher.callRemote("get_versions") d.addCallback(lp.got_versions, pid) return d d.addCallback(_ask_for_versions) catch_up = bool(self.options["catch-up"]) if catch_up: d.addCallback(lambda res: publisher.callRemote("subscribe_to_all", lp, True)) else: # provide compatibility with foolscap-0.2.4 and earlier, which # didn't accept a catchup= argument d.addCallback(lambda res: publisher.callRemote("subscribe_to_all", lp)) d.addErrback(self._error) return d def _lost_logpublisher(publisher): print "Disconnected" foolscap-0.13.1/src/foolscap/logging/web.py0000644000076500000240000004255213204511065021202 0ustar warnerstaff00000000000000 import time, urllib from twisted.internet import reactor, endpoints from twisted.internet.defer import inlineCallbacks, returnValue from twisted.python import usage from foolscap import base32 from foolscap.eventual import fireEventually from foolscap.logging import log, flogfile from foolscap.util import format_time, FORMAT_TIME_MODES, allocate_tcp_port from twisted.web import server, static, html, resource class WebViewerOptions(usage.Options): synopsis = "Usage: flogtool web-viewer DUMPFILE.flog[.bz2]" optFlags = [ ("quiet", "q", "Don't print instructions to stdout"), ("open", "o", "Open the page in your webbrowser automatically"), ] optParameters = [ ("port", "p", None, "endpoint specification of where the web server should listen."), ("timestamps", "t", "short-local", "Format for timestamps: " + " ".join(FORMAT_TIME_MODES)), ] def parseArgs(self, dumpfile): self.dumpfile = dumpfile def opt_timestamps(self, arg): if arg not in FORMAT_TIME_MODES: raise usage.UsageError("--timestamps= must be one of (%s)" % ", ".join(FORMAT_TIME_MODES)) self["timestamps"] = arg FLOG_CSS = """ span.MODELINE { font-size: 60%; } span.NOISY { color: #000080; } span.OPERATIONAL { color: #000000; } span.UNUSUAL { color: #000000; background-color: #ff8080; } span.INFREQUENT { color: #000000; background-color: #ff8080; } span.CURIOUS { color: #000000; background-color: #ff8080; } span.WEIRD { color: #000000; background-color: #ff4040; } span.SCARY { color: #000000; background-color: #ff4040; } span.BAD { color: #000000; background-color: #ff0000; } """ def web_format_time(t, mode="short-local"): time_s = format_time(t, mode) time_utc = format_time(t, "utc") time_local = format_time(t, "long-local") time_ctime = time.ctime(t).replace(" ", " ") extended = "Local=%s Local=%s UTC=%s" % (time_ctime, time_local, time_utc) return time_s, extended def web_escape(u): return html.escape(u.encode("utf-8")) class Welcome(resource.Resource): def __init__(self, viewer, timestamps): self.viewer = viewer self.default_timestamps = timestamps resource.Resource.__init__(self) def fromto_time(self, t, timestamps): if t is None: return "?" ign, extended = web_format_time(float(t), timestamps) tz = time.strftime("%z", time.localtime(t)) return '%s (%s)' % (extended, time.ctime(t), tz) def render(self, req): timestamps = self.default_timestamps data = "" data += "Foolscap Log Viewer\n" data += "\n" data += "

Foolscap Log Viewer

\n" data += "

Logfiles:

\n" if self.viewer.logfiles: data += "
    \n" for lfnum,lf in enumerate(self.viewer.logfiles): data += "
  • %s:\n" % html.escape(lf) data += "
      \n" ((first_number, first_time), (last_number, last_time), num_events, levels, pid, versions) = self.viewer.summaries[lf] # remember: the logfile uses JSON, so all strings will be # unicode, and twisted.web requires bytes data += "
    • PID %s
      • \n" % html.escape(str(pid)) if versions: data += "
    • Application Versions:\n" data += "
        \n" for name in sorted(versions.keys()): ver = versions[name] data += "
      • %s: %s
        • \n" % (web_escape(name), web_escape(ver)) data += "
      \n" data += "
      • \n" if first_time and last_time: duration = int(last_time - first_time) else: duration = "?" data += ("
    • %s events covering %s seconds
      • \n" % (num_events, duration)) from_time_s = self.fromto_time(float(first_time), timestamps) to_time_s = self.fromto_time(float(last_time), timestamps) data += '
    • from %s to %s
      • \n' % (from_time_s, to_time_s) for level in sorted(levels.keys()): data += ('
    • %d events ' 'at level %s
      • \n' % (lfnum, level, len(levels[level]), level)) if self.viewer.triggers: data += "
    • Incident Triggers:\n" data += "
        \n" for t in self.viewer.triggers: le = self.viewer.number_map[t] data += "
      • " href_base = "/all-events?timestamps=%s" % timestamps data += le.to_html(href_base, timestamps) data += "
        • \n" data += "
      \n" data += "
      • \n" data += "
    \n" data += "
\n" else: data += "none!" data += '

' % timestamps data += 'View All Events

\n' data += '
\n' data += ' \n' data += '
\n' data += "" req.setHeader("content-type", "text/html") return data class Summary(resource.Resource): def __init__(self, viewer): self._viewer = viewer resource.Resource.__init__(self) def getChild(self, path, req): if "-" in path: lfnum,levelnum = map(int, path.split("-")) lf = self._viewer.logfiles[lfnum] (first, last, num_events, levels, pid, versions) = self._viewer.summaries[lf] events = levels[levelnum] return SummaryView(events, levelnum) return resource.Resource.getChild(self, path, req) class SummaryView(resource.Resource): def __init__(self, events, levelnum): self._events = events self._levelnum = levelnum resource.Resource.__init__(self) def render(self, req): data = "" data += "Foolscap Log Viewer\n" data += '' data += "\n" data += "\n" data += "

Events at level %d

\n" % self._levelnum data += "
    \n" for e in self._events: data += "
  • " + e.to_html("/all-events") + "
    • \n" data += "
\n" data += "\n" data += "\n" return data class EventView(resource.Resource): def __init__(self, viewer): self.viewer = viewer resource.Resource.__init__(self) def render(self, req): sortby = req.args.get("sort", ["nested"])[0] timestamps = req.args.get("timestamps", ["short-local"])[0] data = "" data += "Foolscap Log Viewer\n" data += '' data += "\n" data += "\n" data += "

Event Log

\n" data += "%d root events " % len(self.viewer.root_events) url = "/all-events?sort=%s" % sortby other_timestamps = ['local' % url, 'utc' % url] url = "/all-events?timestamps=%s" % timestamps other_sortby = ['nested' % url, 'number' % url, 'time' % url] modeline = ''.join(['', 'timestamps=%s ' % timestamps, '(switch to %s) ' % ", ".join(other_timestamps), 'sort=%s ' % sortby, '(switch to %s)' % ", ".join(other_sortby), '\n']) data += modeline data += "
    \n" if sortby == "nested": for e in self.viewer.root_events: data += self._emit_events(0, e, timestamps) elif sortby == "number": numbers = sorted(self.viewer.number_map.keys()) for n in numbers: e = self.viewer.number_map[n] data += '
  • ' % e.level_class() data += e.to_html(timestamps=timestamps) data += '
    • \n' elif sortby == "time": events = self.viewer.number_map.values() events.sort(lambda a,b: cmp(a.e['d']['time'], b.e['d']['time'])) for e in events: data += '
  • ' % e.level_class() data += e.to_html(timestamps=timestamps) data += '
    • \n' else: data += "unknown sort argument '%s'\n" % sortby data += "
\n" req.setHeader("content-type", "text/html") return data def _emit_events(self, indent, event, timestamps): indent_s = " " * indent data = (indent_s + '
  • ' % event.level_class() + event.to_html(timestamps=timestamps) + "
    • \n" ) if event.children: data += indent_s + "
      \n" for child in event.children: data += self._emit_events(indent+1, child, timestamps) data += indent_s + "
    \n" return data class LogEvent: def __init__(self, e): self.e = e self.parent = None self.children = [] self.index = None self.anchor_index = "no-number" self.incarnation = base32.encode(e['d']['incarnation'][0]) if 'num' in e['d']: self.index = (e['from'], e['d']['num']) self.anchor_index = "%s_%s_%d" % (urllib.quote(e['from'].encode("utf-8")), self.incarnation.encode("utf-8"), e['d']['num']) self.parent_index = None if 'parent' in e['d']: self.parent_index = (e['from'], e['d']['parent']) self.is_trigger = False LEVELMAP = { log.NOISY: "NOISY", log.OPERATIONAL: "OPERATIONAL", log.UNUSUAL: "UNUSUAL", log.INFREQUENT: "INFREQUENT", log.CURIOUS: "CURIOUS", log.WEIRD: "WEIRD", log.SCARY: "SCARY", log.BAD: "BAD", } def level_class(self): level = self.e['d'].get('level', log.OPERATIONAL) return self.LEVELMAP.get(level, "UNKNOWN") def to_html(self, href_base="", timestamps="short-local"): # this must return bytes to satisfy twisted.web, but the logfile is # JSON so we get unicode here d = self.e['d'] time_short, time_extended = web_format_time(d['time'], timestamps) msg = web_escape(log.format_message(d)) if 'failure' in d: lines = str(d['failure']).split("\n") html_lines = [web_escape(line) for line in lines] f_html = "\n".join(html_lines) msg += " FAILURE:
    %s
    " % f_html level = d.get('level', log.OPERATIONAL) level_s = "" if level >= log.UNUSUAL: level_s = self.LEVELMAP.get(level, "") + " " details = " ".join(["Event #%d" % d['num'], "TubID=%s" % web_escape(self.e['from']), "Incarnation=%s" % web_escape(self.incarnation), time_extended]) label = '%s' % (details, time_short) data = '%s [%d]: %s%s' \ % (label, self.anchor_index, href_base, self.anchor_index, d['num'], level_s, msg) if self.is_trigger: data += " [INCIDENT-TRIGGER]" return data class Reload(resource.Resource): def __init__(self, viewer): self.viewer = viewer resource.Resource.__init__(self) def render_POST(self, req): self.viewer.load_logfiles() req.redirect("/") return '' class WebViewer: def run(self, options): d = fireEventually(options) d.addCallback(self.start) d.addErrback(self._error) print "starting.." reactor.run() def _error(self, f): print "ERROR", f reactor.stop() @inlineCallbacks def start(self, options): root = static.Data("placeholder", "text/plain") welcome = Welcome(self, options["timestamps"]) root.putChild("", welcome) root.putChild("welcome", welcome) # we used to only do this root.putChild("reload", Reload(self)) root.putChild("all-events", EventView(self)) root.putChild("summary", Summary(self)) root.putChild("flog.css", static.Data(FLOG_CSS, "text/css")) s = server.Site(root) port = options["port"] if not port: port = "tcp:%d:interface=127.0.0.1" % allocate_tcp_port() ep = endpoints.serverFromString(reactor, port) self.lp = yield ep.listen(s) portnum = self.lp.getHost().port # TODO: this makes all sort of assumptions: HTTP-vs-HTTPS, localhost. url = "http://localhost:%d/" % portnum if not options["quiet"]: print "scanning.." self.logfiles = [options.dumpfile] self.load_logfiles() if not options["quiet"]: print "please point your browser at:" print url if options["open"]: import webbrowser webbrowser.open(url) returnValue(url) # for tests def stop(self): return self.lp.stopListening() def load_logfiles(self): #self.summary = {} # keyed by logfile name (self.summaries, self.root_events, self.number_map, self.triggers) = self.process_logfiles(self.logfiles) def process_logfiles(self, logfiles): summaries = {} # build up a tree of events based upon parent/child relationships number_map = {} roots = [] trigger_numbers = [] first_event_from = None for lf in logfiles: (first_event_number, first_event_time) = (None, None) (last_event_number, last_event_time) = (None, None) num_events = 0 levels = {} pid = None for e in flogfile.get_events(lf): if "header" in e: h = e["header"] if h["type"] == "incident": t = h["trigger"] trigger_numbers.append(t["num"]) pid = h.get("pid") versions = h.get("versions", {}) if "d" not in e: continue # skip headers if not first_event_from: first_event_from = e['from'] le = LogEvent(e) if le.index: number_map[le.index] = le if le.parent_index in number_map: le.parent = number_map[le.parent_index] le.parent.children.append(le) else: roots.append(le) d = e['d'] level = d.get("level", "NORMAL") number = d.get("num", None) when = d.get("time") if number in trigger_numbers: le.is_trigger = True if False: # this is only meaningful if the logfile contains events # from just a single tub and incarnation, but our current # LogGatherer combines multiple processes' logs into a # single file. if first_event_number is None: first_event_number = number elif number is not None: first_event_number = min(first_event_number, number) if last_event_number is None: last_event_number = number elif number is not None: last_event_number = max(last_event_number, number) if first_event_time is None: first_event_time = when elif when is not None: first_event_time = min(first_event_time, when) if last_event_time is None: last_event_time = when elif when is not None: last_event_time = max(last_event_time, when) num_events += 1 if level not in levels: levels[level] = [] levels[level].append(le) summary = ( (first_event_number, first_event_time), (last_event_number, last_event_time), num_events, levels, pid, versions ) summaries[lf] = summary triggers = [(first_event_from, num) for num in trigger_numbers] return summaries, roots, number_map, triggers foolscap-0.13.1/src/foolscap/negotiate.py0000644000076500000240000015040113204160675020756 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test.test_negotiate -*- import time from twisted.python.failure import Failure from twisted.internet import protocol, reactor, defer from twisted.internet.error import ConnectionDone from foolscap import broker, referenceable, vocab from foolscap.eventual import eventually from foolscap.tokens import (SIZE_LIMIT, ERROR, BananaError, NegotiationError, RemoteNegotiationError, DuplicateConnection) from foolscap.ipb import DeadReferenceError from foolscap.banana import int2b128 from foolscap.logging import log from foolscap.logging.log import NOISY, OPERATIONAL, WEIRD, UNUSUAL, CURIOUS from foolscap.util import isSubstring from foolscap import crypto def best_overlap(my_min, my_max, your_min, your_max, name): """Find the highest integer which is in both ranges (inclusive). Raise NegotiationError (using 'name' in the error message) if there is no overlap.""" best = min(my_max, your_max) if best < my_min: raise NegotiationError("I can't handle %s %d" % (name, best)) if best < your_min: raise NegotiationError("You can't handle %s %d" % (name, best)) return best def check_inrange(my_min, my_max, decision, name): if decision < my_min or decision > my_max: raise NegotiationError("I can't handle %s %d" % (name, decision)) # negotiation phases PLAINTEXT, ENCRYPTED, DECIDING, BANANA, ABANDONED = range(5) # version number history: # 1 (0.1.0): offer includes initial-vocab-table-range, # decision includes initial-vocab-table-index # 2 (0.1.1): no changes to offer or decision # reqID=0 was commandeered for use by callRemoteOnly() # 3 (0.1.3): added PING and PONG tokens class Negotiation(protocol.Protocol): """This is the first protocol to speak over the wire. It is responsible for negotiating the connection parameters, then switching the connection over to the actual Banana protocol. This removes all the details of negotiation from Banana, and makes it easier to use a more complex scheme (including a STARTTLS transition) in PB. Negotiation consists of three phases. In the PLAINTEXT phase, the client side (i.e. the one which initiated the connection) sends an HTTP-compatible GET request for the target Tub ID. This request includes an Connection: Upgrade header. The GET request serves a couple of purposes: if a PB client is accidentally pointed at an HTTP server, it will trigger a sensible 404 Error instead of getting confused. A regular HTTP server can be used to send back a 303 Redirect, allowing Apache (or whatever) to be used as a redirection server. After sending the GET request, the client waits for the server to send a 101 Switching Protocols command, then starts the TLS session. It may also receive a 303 Redirect command, in which case it drops the connection and tries again with the new target. In the PLAINTEXT phase, the server side (i.e. the one which accepted the connection) waits for the client's GET request, extracts the TubID from the first line, consults the local Listener object to locate the appropriate Tub (and its certificate), sends back a 101 Switching Protocols response, then starts the TLS session with the Tub's certificate. If the Listener reports that the requested Tub is listening elsewhere, the server sends back a 303 Redirect instead, and then drops the connection. By the end of the PLAINTEXT phase, both ends know which Tub they are using (self.tub has been set). Both sides send a Hello Block upon entering the ENCRYPTED phase, which in practice means just after starting the TLS session. The Hello block contains the negotiation offer, as a series of Key: Value lines separated by \\r\\n delimiters and terminated by a blank line. Upon receiving the other end's Hello block, each side switches to the DECIDING phase, and then evaluates the received Hello message. Each side compares TubIDs, and the side with the lexicographically higher value becomes the Master. (If, for some reason, one side does not claim a TubID, its value is treated as None, which always compares *less* than any actual TubID, so the non-TubID side will probably not be the Master. Any possible ties are resolved by having the server side be the master). Both sides know the other's TubID, so both sides know whether they are the Master or not. The Master has two jobs to do. The first is that it compares the negotiation offer against its own capabilities, and comes to a decision about what the connection parameters shall be. It may decide that the two sides are not compatible, in which case it will abandon the connection. The second job is to decide whether to continue to use the connection at all: if the Master already has a connection to the other Tub, it will drop this new one. This decision must be made by the Master (as opposed to the Server) because it is possible for both Tubs to connect to each other simultaneously, and this design avoids a race condition that could otherwise drop *both* connections. If the Master decides to continue with the connection, it sends the Decision block to the non-master side. It then swaps out the Negotiation protocol for a new Banana protocol instance that has been created with the same parameters that were used to create the Decision block. The non-master side is waiting in the DECIDING phase for this block. Upon receiving it, the non-master side evaluates the connection parameters and either drops the connection or swaps in a new Banana protocol instance with the same parameters. At this point, negotiation is complete and the Negotiation instances are dropped. @ivar negotationOffer: a dict which describes what we will offer to the far side. Each key/value pair will be put into a rfc822-style header and sent from the client to the server when the connection is established. On the server side, handleNegotiation() uses negotationOffer to indicate what we are locally capable of. Subclasses may influence the negotiation process by modifying this dictionary before connectionMade() is called. @ivar negotiationResults: a dict which describes what the two ends have agreed upon. This is computed by the server, stored locally, and sent down to the client. The client receives it and stores it without modification (server chooses). In general, the negotiationResults are the same on both sides of the same connection. However there may be certain parameters which are sent as part of the negotiation block (the PB TubID, for example) which will not. """ myTubID = None tub = None theirTubID = None receive_phase = PLAINTEXT # we are expecting this send_phase = PLAINTEXT # the other end is expecting this doNegotiation = True forceNegotiation = None minVersion = 3 maxVersion = 3 brokerClass = broker.Broker initialVocabTableRange = vocab.getVocabRange() SERVER_TIMEOUT = 120 # You have 2 minutes to complete negotiation, or # else. The only reason this isn't closer to 10s is # that Tor/I2P connection establishment might # include spinning up a local Tor/I2P daemon, which # can take 30-50 seconds from a cold start. negotiationTimer = None def __init__(self, logparent=None): self._logparent = log.msg("Negotiation started", parent=logparent, facility="foolscap.negotiation") for i in range(self.minVersion, self.maxVersion+1): assert hasattr(self, "evaluateNegotiationVersion%d" % i), i assert hasattr(self, "acceptDecisionVersion%d" % i), i assert isinstance(self.initialVocabTableRange, tuple) self.negotiationOffer = { "banana-negotiation-range": "%d %d" % (self.minVersion, self.maxVersion), "initial-vocab-table-range": "%d %d" % self.initialVocabTableRange, } # TODO: for testing purposes, it might be useful to be able to add # some keys to this offer if self.forceNegotiation is not None: # TODO: decide how forcing should work. Maybe forceNegotiation # should be a dict of keys or something. distinguish between # offer and decision. self.negotiationOffer['negotiation-forced'] = "True" self.buffer = "" self._test_options = {} # to trigger specific race conditions during unit tests, it is useful # to allow certain operations to be stalled for a moment. # self._test_options will contain a key like # debug_slow_connectionMade to indicate that there should be a 1 # second delay between the real connectionMade and the time our # self.connectionMade() method is invoked. To support this, the first # time connectionMade() is invoked, # self.debugTimers['connectionMade'] is set to a 1s DelayedCall, # which fires self.debug_fireTimer('connectionMade', callable, # *args). That will set self.debugTimers['connectionMade'] to None, # so the condition is not fired again, then invoke the actual # connectionMade method. When the connection is lost, all remaining # timers will be canceled. self.debugTimers = {} self.debugPauses = {} # similar, but holds a Deferred # if anything goes wrong during negotiation (version mismatch, # malformed headers, assertion checks), we stash the Failure in this # attribute and then drop the connection. For client-side # connections, we notify our parent TubConnector when the # connectionLost() message is finally delivered. self.failureReason = None def log(self, *args, **kwargs): # we log as NOISY by default, because nobody should hear about # negotiation unless it goes wrong. if 'parent' not in kwargs: kwargs['parent'] = self._logparent if 'facility' not in kwargs: kwargs['facility'] = "foolscap.negotiation" if 'level' not in kwargs: kwargs['level'] = log.NOISY return log.msg(*args, **kwargs) def initClient(self, connector, targetHost, connectionInfo): # clients do connectTCP and speak first with a GET self.log("initClient: to target %s" % connector.target, target=connector.target.getTubID()) self.isClient = True self.tub = connector.tub self.brokerClass = self.tub.brokerClass self.myTubID = self.tub.tubID self.connector = connector self.target = connector.target self.targetHost = targetHost self._connectionInfo = connectionInfo self._test_options = self.tub._test_options.copy() tubID = self.target.getTubID() slave_record = self.tub.slave_table.get(tubID, ("none",0)) assert isinstance(slave_record, tuple), slave_record self.negotiationOffer['last-connection'] = "%s %s" % slave_record def initServer(self, listener, connectionInfo): # servers do listenTCP and respond to the GET self.log("initServer", listener=repr(listener)) self.isClient = False self.listener = listener self._connectionInfo = connectionInfo self._test_options = self.listener._test_options.copy() # the broker class is set when we find out which Tub we should use def parseLines(self, header): lines = header.split("\r\n") block = {} for line in lines: colon = line.index(":") key = line[:colon].lower() value = line[colon+1:].lstrip() block[key] = value return block def sendBlock(self, block): keys = block.keys() keys.sort() for k in keys: self.transport.write("%s: %s\r\n" % (k.lower(), block[k])) self.transport.write("\r\n") # end block def debug_doTimer(self, name, timeout, call, *args): if (self._test_options.has_key("debug_slow_%s" % name) and not self.debugTimers.has_key(name)): self.log("debug_doTimer(%s)" % name) t = reactor.callLater(timeout, self.debug_fireTimer, name) self.debugTimers[name] = (t, [(call, args)]) cb = self._test_options["debug_slow_%s" % name] if cb is not None and cb is not True: cb() return True return False def debug_doPause(self, name, call, *args): cb = self._test_options.get("debug_pause_%s" % name, None) if not cb: return False if self.debugPauses.has_key(name): return False self.log("debug_doPause(%s)" % name) self.debugPauses[name] = d = defer.Deferred() d.addCallback(lambda _: call(*args)) try: cb(d) except Exception, e: print e # otherwise failures are hard to track down raise return True def debug_addTimerCallback(self, name, call, *args): if self.debugTimers.get(name): self.debugTimers[name][1].append((call, args)) return True return False def debug_forceTimer(self, name): if self.debugTimers.get(name): self.debugTimers[name][0].cancel() self.debug_fireTimer(name) def debug_forceAllTimers(self): for name in self.debugTimers: if self.debugTimers.get(name): self.debugTimers[name][0].cancel() self.debug_fireTimer(name) def debug_cancelAllTimers(self): for name in self.debugTimers: if self.debugTimers.get(name): self.debugTimers[name][0].cancel() self.debugTimers[name] = None def debug_fireTimer(self, name): calls = self.debugTimers[name][1] self.debugTimers[name] = None for call,args in calls: call(*args) def connectionMade(self): # once connected, this Negotiation instance must either invoke # self.switchToBanana or self.negotiationFailed, to insure that the # TubConnector (if any) gets told about the results of the connection # attempt. if self.doNegotiation: if self.isClient: self.connectionMadeClient() else: self.connectionMadeServer() else: self.switchToBanana({}) def connectionMadeClient(self): assert self.receive_phase == PLAINTEXT # the client needs to send the HTTP-compatible tubid GET, # along with the TLS upgrade request self.sendPlaintextClient() # now we wait for the TLS Upgrade acceptance to come back def sendPlaintextClient(self): req = [] self.log("sendPlaintextClient: GET for tubID %s" % self.target.tubID) req.append("GET /id/%s HTTP/1.1" % self.target.tubID) req.append("Host: %s" % self.targetHost) self.log("sendPlaintextClient: wantEncryption=True") req.append("Upgrade: TLS/1.0") req.append("Connection: Upgrade") self.transport.write("\r\n".join(req)) self.transport.write("\r\n\r\n") # the next thing the other end expects to see is the encrypted phase self.send_phase = ENCRYPTED def connectionMadeServer(self): # the server just waits for the GET message to arrive, but set up the # server timeout first if self.debug_doTimer("connectionMade", 1, self.connectionMade): return timeout = self._test_options.get('server_timeout', self.SERVER_TIMEOUT) if timeout: # oldpb clients will hit this case. self.negotiationTimer = reactor.callLater(timeout, self.negotiationTimedOut) def sendError(self, why): pass # TODO def negotiationTimedOut(self): del self.negotiationTimer why = Failure(NegotiationError("negotiation timeout")) self.sendError(why) self.failureReason = why self.transport.loseConnection() def stopNegotiationTimer(self): if self.negotiationTimer: self.negotiationTimer.cancel() del self.negotiationTimer def dataReceived(self, chunk): self.log("dataReceived(isClient=%s,phase=%s,options=%s): %r" % (self.isClient, self.receive_phase, self._test_options, chunk), level=NOISY) if self.receive_phase == ABANDONED: return self.buffer += chunk if self.debug_addTimerCallback("connectionMade", self.dataReceived, ''): return try: # we accumulate a header block for each phase if len(self.buffer) > 4096: raise BananaError("Header too long") eoh = self.buffer.find('\r\n\r\n') if eoh == -1: return header, self.buffer = self.buffer[:eoh], self.buffer[eoh+4:] if self.receive_phase == PLAINTEXT: if self.isClient: self.handlePLAINTEXTClient(header) else: self.handlePLAINTEXTServer(header) elif self.receive_phase == ENCRYPTED: self.handleENCRYPTED(header) elif self.receive_phase == DECIDING: self.handleDECIDING(header) else: assert 0, "should not get here" # there might be some leftover data for the next phase. # self.buffer will be emptied when we switchToBanana, so in that # case we won't call the wrong dataReceived. if self.buffer: self.dataReceived("") except Exception, e: why = Failure() if isinstance(e, RemoteNegotiationError): pass # they've already hung up else: # there's a chance we can provide a little bit more information # to the other end before we hang up on them if isinstance(e, NegotiationError): errmsg = str(e) else: self.log("negotiation had internal error:", failure=why, level=UNUSUAL) errmsg = "internal server error, see logs" errmsg = errmsg.replace("\n", " ").replace("\r", " ") if self.send_phase == PLAINTEXT: resp = ("HTTP/1.1 500 Internal Server Error: %s\r\n\r\n" % errmsg) self.transport.write(resp) elif self.send_phase in (ENCRYPTED, DECIDING): block = {'banana-decision-version': 1, 'error': errmsg, } self.sendBlock(block) elif self.send_phase == BANANA: self.sendBananaError(errmsg) self.failureReason = why self.transport.loseConnection() return def sendBananaError(self, msg): if len(msg) > SIZE_LIMIT: msg = msg[:SIZE_LIMIT-10] + "..." int2b128(len(msg), self.transport.write) self.transport.write(ERROR) self.transport.write(msg) # now you should drop the connection def connectionLost(self, reason): # force connectionMade to happen, so connectionLost can occur # normally self.debug_forceTimer("connectionMade") # cancel the other slowdown timers, since they all involve sending # data, and the connection is no longer available self.debug_cancelAllTimers() for k,t in self.debugTimers.items(): if t: t[0].cancel() self.debugTimers[k] = None if self.isClient: l = self.tub._test_options.get("debug_gatherPhases") if l is not None: l.append(self.receive_phase) if not self.failureReason: self.failureReason = reason self.negotiationFailed() def handlePLAINTEXTServer(self, header): # the client sends us a GET message lines = header.split("\r\n") if not lines[0].startswith("GET "): raise BananaError("not right") command, url, version = lines[0].split() if not url.startswith("/id/"): # probably a web browser raise BananaError("not right") targetTubID = url[4:] self.log("handlePLAINTEXTServer: targetTubID='%s'" % targetTubID, level=NOISY) if targetTubID == "": # they're asking for an old UnauthenticatedTub. Refuse. raise NegotiationError("secure Tubs require encryption") if isSubstring("Upgrade: TLS/1.0\r\n", header): wantEncrypted = True else: wantEncrypted = False self.log("handlePLAINTEXTServer: wantEncrypted=%s" % wantEncrypted, level=NOISY) # we ignore the rest of the lines # now that we know which Tub the client wants to connect to, either # send a Redirect, or start the ENCRYPTED phase tub, redirect = self.listener.lookupTubID(targetTubID) if tub: self.tub = tub # our tub self._test_options.update(self.tub._test_options) self.brokerClass = self.tub.brokerClass self.myTubID = tub.tubID self.sendPlaintextServerAndStartENCRYPTED() elif redirect: self.sendRedirect(redirect) else: raise NegotiationError("unknown TubID %s" % targetTubID) def sendPlaintextServerAndStartENCRYPTED(self): # this is invoked on the server side if self.debug_doTimer("sendPlaintextServer", 1, self.sendPlaintextServerAndStartENCRYPTED): return resp = "\r\n".join(["HTTP/1.1 101 Switching Protocols", "Upgrade: TLS/1.0, PB/1.0", "Connection: Upgrade", ]) self.transport.write(resp) self.transport.write("\r\n\r\n") # the next thing they expect is the encrypted block self.send_phase = ENCRYPTED self.startENCRYPTED() def sendRedirect(self, redirect): # this is invoked on the server side # send the redirect message, then close the connection. make sure the # data gets flushed, though. raise NotImplementedError # TODO def handlePLAINTEXTClient(self, header): self.log("handlePLAINTEXTClient: header='%s'" % header) lines = header.split("\r\n") tokens = lines[0].split() # TODO: accept a 303 redirect if tokens[1] != "101": raise BananaError("not right, got '%s', " "expected 101 Switching Protocols" % lines[0]) if not isSubstring("Upgrade: TLS/1.0", header): raise BananaError("header didn't contain TLS upgrade: %r" % (header,)) # we ignore everything else # now we upgrade to TLS self.startENCRYPTED() # and wait for their Hello to arrive def startENCRYPTED(self): # this is invoked on both sides. We move to the "ENCRYPTED" phase, # which involves a TLS-encrypted session. self.log("startENCRYPTED(isClient=%s)" % (self.isClient,)) self.startTLS(self.tub.myCertificate) # TODO: can startTLS trigger dataReceived? self.receive_phase = ENCRYPTED self.sendHello() def sendHello(self): """This is called on both sides as soon as the encrypted connection is established. This causes a negotiation block to be sent to the other side as an offer.""" if self.debug_doTimer("sendHello", 1, self.sendHello): return if self.debug_doPause("sendHello", self.sendHello): return hello = self.negotiationOffer.copy() assert self.myTubID # This indicates which identity we wish to claim. This is the hash of # the certificate we're using. hello['my-tub-id'] = self.myTubID if self.tub: IR = self.tub.getIncarnationString() hello['my-incarnation'] = IR self.log("Negotiate.sendHello (isClient=%s): %s" % (self.isClient, hello)) self.sendBlock(hello) def handleENCRYPTED(self, header): # both ends have sent a Hello message if self.debug_addTimerCallback("sendHello", self.handleENCRYPTED, header): return self.theirCertificate = None # We should be encrypted now. Get the peer's certificate. them = crypto.peerFromTransport(self.transport) if them and them.original: self.theirCertificate = them hello = self.parseLines(header) if hello.has_key("error"): raise RemoteNegotiationError(hello["error"]) self.evaluateHello(hello) def evaluateHello(self, offer): """Evaluate the HELLO message sent by the other side. We compare TubIDs, and the higher value becomes the 'master' and makes the negotiation decisions. This method returns a tuple of DECISION,PARAMS. There are a few different possibilities:: - We are the master, we make a negotiation decision: DECISION is the block of data to send back to the non-master side, PARAMS are the connection parameters we will use ourselves. - We are the master, we can't accomodate their request: raise NegotiationError - We are not the master: DECISION is None """ self.log("evaluateHello(isClient=%s): offer=%s" % (self.isClient, offer)) if not offer.has_key('banana-negotiation-range'): if offer.has_key('banana-negotiation-version'): msg = ("Peer is speaking foolscap-0.0.5 or earlier, " "which is not compatible with this version. " "Please upgrade the peer.") raise NegotiationError(msg) raise NegotiationError("No valid banana-negotiation sequence seen") min_s, max_s = offer['banana-negotiation-range'].split() theirMinVer = int(min_s) theirMaxVer = int(max_s) # best_overlap() might raise a NegotiationError best = best_overlap(self.minVersion, self.maxVersion, theirMinVer, theirMaxVer, "banana version") negfunc = getattr(self, "evaluateNegotiationVersion%d" % best) self.decision_version = best return negfunc(offer) def evaluateNegotiationVersion1(self, offer): forced = False f = offer.get('negotiation-forced', None) if f and f.lower() == "true": forced = True # 'forced' means the client is on a one-way link (or is really # stubborn) and has already made up its mind about the connection # parameters. If we are unable to handle exactly what they have # offered, we must hang up. assert not forced # TODO: implement # glyph says: look at Juice, it does rfc822 parsing, startTLS, # switch-to-other-protocol, etc. grep for retrieveConnection in q2q. # TODO: oh, if we see an HTTP client, send a good HTTP error like # "protocol not supported", or maybe even an HTML page that explains # what a PB server is # there are four distinct dicts here: # self.negotiationOffer: what we want # clientOffer: what they sent to us, the client's requests. # serverOffer: what we send to them, the server's decision # self.negotiationResults: the negotiated settings # # [my-tub-id] is not present in self.negotiationResults # the server's tubID is in [my-tub-id] for both self.negotiationOffer # and serverOffer # the client's tubID is in [my-tub-id] for clientOffer myTubID = self.myTubID theirTubID = offer.get("my-tub-id") if self.theirCertificate is None: # no client certificate if theirTubID is not None: # this is where a poor MitM attack is detected, one which # doesn't even pretend to encrypt the connection raise BananaError("you must use a certificate to claim a " "TubID") else: # verify that their claimed TubID matches their SSL certificate. # TODO: handle chains digest = crypto.digest32(self.theirCertificate.digest("sha1")) if digest != theirTubID: # this is where a good MitM attack is detected, one which # encrypts the connection but which of course uses the wrong # certificate raise BananaError("TubID mismatch") assert theirTubID theirTubRef = referenceable.TubRef(theirTubID) self.theirTubRef = theirTubRef # for use by non-master side, later if self.isClient: # verify that we connected to the Tub we expected to. if theirTubRef != self.target: # TODO: how (if at all) should this error message be # communicated to the other side? raise BananaError("connected to the wrong Tub") if myTubID is None and theirTubID is None: iAmTheMaster = not self.isClient elif myTubID is None: iAmTheMaster = False elif theirTubID is None: iAmTheMaster = True else: # this is the most common case iAmTheMaster = myTubID > theirTubID self.log(format="iAmTheMaster: %(master)s", master=iAmTheMaster) decision, params = None, None if iAmTheMaster: # we get to decide everything. The other side is now waiting for # a decision block. self.send_phase = DECIDING decision = {} params = {} # combine their 'offer' and our own self.negotiationOffer to come # up with a 'decision' to be sent back to the other end, and the # 'params' to be used on our connection # first, do we continue with this connection? we might have an # existing connection for this particular tub if theirTubRef and theirTubRef in self.tub.brokers: # there is an existing connection.. we might want to prefer # this new offer, because the old connection might be stale # (NAT boxes and laptops that disconnect abruptly are two # ways for a single process to disappear silently and then # reappear with a different IP address). lp = self.log("got offer for an existing connection", level=UNUSUAL) existing = self.tub.brokers[theirTubRef] acceptOffer = self.compareOfferAndExisting(offer, existing, lp) if acceptOffer: # drop the old one self.log("accepting new offer, dropping existing connection", parent=lp) err = DeadReferenceError("[%s] replaced by a new connection" % theirTubRef.getShortTubID()) why = Failure(err) existing.shutdown(why) else: # reject the new one self.log("rejecting the offer: we already have one", parent=lp) raise DuplicateConnection("Duplicate connection") if theirTubRef: # generate a new seqnum, one higher than the last one we've # used. old_seqnum = self.tub.master_table.get(theirTubRef.getTubID(), 0) new_seqnum = old_seqnum + 1 new_slave_IR = offer.get('my-incarnation', None) self.tub.master_table[theirTubRef.getTubID()] = new_seqnum my_IR = self.tub.getIncarnationString() decision['current-connection'] = "%s %s" % (my_IR, new_seqnum) # these params will be copied into the Broker where we can # retrieve them later, when we need to compare it against a new # offer. params['current-slave-IR'] = new_slave_IR params['current-seqnum'] = new_seqnum # what initial vocab set should we use? theirVocabRange_s = offer.get("initial-vocab-table-range", "0 0") theirVocabRange = theirVocabRange_s.split() theirVocabMin = int(theirVocabRange[0]) theirVocabMax = int(theirVocabRange[1]) vocab_index = best_overlap( self.initialVocabTableRange[0], self.initialVocabTableRange[1], theirVocabMin, theirVocabMax, "initial vocab set") vocab_hash = vocab.hashVocabTable(vocab_index) decision['initial-vocab-table-index'] = "%d %s" % (vocab_index, vocab_hash) decision['banana-decision-version'] = str(self.decision_version) # v1: handle vocab table index params['banana-decision-version'] = self.decision_version params['initial-vocab-table-index'] = vocab_index else: # otherwise, the other side gets to decide. The next thing they # expect to hear from us is banana. self.send_phase = BANANA if iAmTheMaster: # I am the master, so I send the decision self.log("Negotiation.sendDecision: %s" % decision, level=OPERATIONAL) # now we send the decision and switch to Banana. they might hang # up. self.sendDecision(decision, params) else: # I am not the master, I receive the decision self.receive_phase = DECIDING def evaluateNegotiationVersion2(self, offer): # version 2 changes the meaning of reqID=0 in a 'call' sequence, to # support the implementation of callRemoteOnly. No other protocol # changes were made, and no changes were made to the offer or # decision blocks. return self.evaluateNegotiationVersion1(offer) def evaluateNegotiationVersion3(self, offer): # version 3 adds PING and PONG tokens, to enable keepalives and # idle-disconnect. No other protocol changes were made, and no # changes were made to the offer or decision blocks. return self.evaluateNegotiationVersion1(offer) def compareOfferAndExisting(self, offer, existing, lp): """Compare the new offer against the existing connection, and decide which to keep. @return: True to accept the new offer, False to stick with the existing connection. """ def log(*args, **kwargs): if 'parent' not in kwargs: kwargs['parent'] = lp return self.log(*args, **kwargs) existing_slave_IR = existing.current_slave_IR existing_seqnum = existing.current_seqnum log(format="existing connection has slave_IR=%(slave_IR)s, seqnum=%(seqnum)s", slave_IR=existing_slave_IR, seqnum=existing_seqnum) # TESTING: force handle-old stuff #lp2 = log("TESTING: forcing use of handle-old logic") #return self.handle_old(offer, existing, 60, lp2) # step one: does the inbound offer have a my-incarnation header? If # not, this is an older peer ( existing_seqnum indicates something really # weird has taken place. log(format="offer_master_seqnum %(offer)d > existing_seqnum %(existing)d", offer=offer_master_seqnum, existing=existing_seqnum, level=WEIRD) return False # reject weirdness def handle_old(self, offer, existing, threshold, lp): # determine the age of the existing broker age = time.time() - existing.creation_timestamp if age < threshold: self.log("the existing broker is too new (%d<%d), rejecting offer" % (age, threshold), parent=lp) return False # reject the offer self.log("the existing broker is old enough to replace", parent=lp) return True # accept the offer def sendDecision(self, decision, params): if self.debug_doTimer("sendDecision", 1, self.sendDecision, decision, params): return if self.debug_addTimerCallback("sendHello", self.sendDecision, decision, params): return self.sendBlock(decision) self.send_phase = BANANA self.switchToBanana(params) def handleDECIDING(self, header): # this gets called on the non-master side self.log("handleDECIDING(isClient=%s): %s" % (self.isClient, header), level=NOISY) if self.debug_doTimer("handleDECIDING", 1, self.handleDECIDING, header): # for testing purposes, wait a moment before accepting the # decision. This insures that we trigger the "Duplicate # Broker" condition. NOTE: This will interact badly with the # "there might be some leftover data for the next phase" call # in dataReceived return decision = self.parseLines(header) params = self.acceptDecision(decision) self.switchToBanana(params) def acceptDecision(self, decision): """This is called on the client end when it receives the results of the negotiation from the server. The client must accept this decision (and return the connection parameters dict), or raise NegotiationError to hang up.negotiationResults.""" self.log("Banana.acceptDecision: got %s" % decision, level=OPERATIONAL) version = decision.get('banana-decision-version') if not version: raise NegotiationError("No banana-decision-version value") acceptfunc = getattr(self, "acceptDecisionVersion%d" % int(version)) if not acceptfunc: raise NegotiationError("I cannot handle banana-decision-version " "value of %d" % int(version)) return acceptfunc(decision) def acceptDecisionVersion1(self, decision): if decision.has_key("error"): error = decision["error"] raise RemoteNegotiationError("Banana negotiation failed: %s" % error) # parse the decision here, create the connection parameters dict ver = int(decision['banana-decision-version']) vocab_index_string = decision.get('initial-vocab-table-index') if vocab_index_string: vocab_index, vocab_hash = vocab_index_string.split() vocab_index = int(vocab_index) else: vocab_index = 0 check_inrange(self.initialVocabTableRange[0], self.initialVocabTableRange[1], vocab_index, "initial vocab table index") our_hash = vocab.hashVocabTable(vocab_index) if vocab_index > 0 and our_hash != vocab_hash: msg = ("Our hash for vocab-table-index %d (%s) does not match " "your hash (%s)" % (vocab_index, our_hash, vocab_hash)) raise NegotiationError(msg) if self.theirTubRef in self.tub.brokers: # we're the slave, so we need to drop our existing connection and # use the one picked by the master self.log("master told us to use a new connection, " "so we must drop the existing one", level=UNUSUAL) err = DeadReferenceError("replaced by a new connection") why = Failure(err) self.tub.brokers[self.theirTubRef].shutdown(why) current_connection = decision.get('current-connection') if current_connection: tubID = self.theirTubRef.getTubID() self.tub.slave_table[tubID] = tuple(current_connection.split()) else: self.log("no current-connection in decision from %s" % self.theirTubRef, level=UNUSUAL) params = { 'banana-decision-version': ver, 'initial-vocab-table-index': vocab_index, } return params def acceptDecisionVersion2(self, decision): # this only affects the interpretation of reqID=0, so we can use the # same accept function return self.acceptDecisionVersion1(decision) def acceptDecisionVersion3(self, decision): # this adds PING and PONG tokens, so we can use the same accept # function return self.acceptDecisionVersion1(decision) def loopbackDecision(self): # if we were talking to ourselves, what negotiation decision would we # reach? This is used for loopback connections max_vocab = self.initialVocabTableRange[1] params = { 'banana-decision-version': self.maxVersion, 'initial-vocab-table-index': max_vocab, } return params def startTLS(self, cert): # the TLS connection (according to glyph) is "ready" immediately, but # really the negotiation is going on behind the scenes (OpenSSL is # trying a little too hard to be transparent). I think you have to # write some bytes to trigger the negotiation. getPeerCertificate() # can't be called until you receive some bytes, so grab it when a # negotiation block arrives that claims to have an authenticated # TubID. # Instead of this: # opts = self.tub.myCertificate.options() # We use the MyOptions class to fix up the verify stuff: we request a # certificate from the client, but do not verify it against a list of # root CAs self.log("startTLS, client=%s" % self.isClient) kwargs = {} if cert: kwargs['privateKey'] = cert.privateKey.original kwargs['certificate'] = cert.original ctxFactory = crypto.FoolscapContextFactory(**kwargs) self.transport.startTLS(ctxFactory) def switchToBanana(self, params): # switch over to the new protocol (a Broker instance). This # Negotiation protocol goes away after this point. lp = self.log("Negotiate.switchToBanana(isClient=%s)" % self.isClient, level=NOISY) self.log("params: %s" % (params,), parent=lp) self.stopNegotiationTimer() if self.isClient: theirTubRef = self.target else: theirTubRef = self.theirTubRef b = self.brokerClass(theirTubRef, params, self.tub.keepaliveTimeout, self.tub.disconnectTimeout, self._connectionInfo, ) b.factory = self.factory # not used for PB code b.setTub(self.tub) # we leave ourselves as the protocol, but redirect incoming messages # (from the transport) to the broker #self.transport.protocol = b self.dataReceived = b.dataReceived self.connectionLost = b.connectionLost b.makeConnection(self.transport) buf, self.buffer = self.buffer, "" # empty our buffer, just in case b.dataReceived(buf) # and hand it to the new protocol self._connectionInfo._set_connected(True) # if we were created as a client, we'll have a TubConnector. Let them # know that this connection has succeeded, so they can stop any other # connection attempts still in progress. if self.isClient: self.connector.connectorNegotiationComplete(self, self.factory.location) else: self._connectionInfo._set_listener_status("successful") # finally let our Tub know that they can start using the new Broker. # This will wake up anyone who initiated an outbound connection. self.tub.brokerAttached(theirTubRef, b, self.isClient) def negotiationFailed(self): reason = self.failureReason self.stopNegotiationTimer() if self.receive_phase != ABANDONED and self.isClient: eventually(self.connector.connectorNegotiationFailed, self, self.factory.location, reason) self.receive_phase = ABANDONED if not self.isClient: description = "negotiation failed: %s" % str(reason.value) self._connectionInfo._set_listener_status(description) cb = self._test_options.get("debug_negotiationFailed_cb") if cb: # note that this gets called with a NegotiationError, not a # Failure. ACTUALLY: not true, gets a Failure eventually(cb, reason) # Negotiations fail all the time, for benign reasons, so limit how # much we log (the full Failure and traceback is frequently useless # and noisy). Parallel connection-hints cause the slower connection # to be rejected as a duplicate, as do full-mesh applications (like # Tahoe) that construct cross-linked connections. if reason.check(DuplicateConnection): # this happens when we reject a connection during negotiation self.log("negotiationFailed: DuplicateConnection", level=NOISY, umid="XRFlRA") elif reason.check(ConnectionDone): # this happens to our other losing parallel connection attempts self.log("negotiationFailed: ConnectionDone", level=NOISY, umid="9khFxA") elif reason.check(RemoteNegotiationError): # and this is how the remote side tells us they rejected or # abandoned a connection. Sometimes it's due to a duplicate # connection, sometimes due to code problems. In either case, the # traceback would only show local code, and is unhelpful. self.log("negotiationFailed: remote: %s" % reason.value.args[0], level=NOISY, umid="yAsbmA") else: # This shouldn't happen very often. self.log("negotiationFailed", failure=reason, level=OPERATIONAL, umid="pm2kjg") # TODO: make sure code that examines self.receive_phase handles ABANDONED foolscap-0.13.1/src/foolscap/observer.py0000644000076500000240000000312012766553111020624 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test_observer -*- # many thanks to AllMyData for contributing the initial version of this code from twisted.internet import defer from foolscap import eventual class OneShotObserverList(object): """A one-shot event distributor. Subscribers can get a Deferred that will fire with the results of the event once it finally occurs. The caller does not need to know whether the event has happened yet or not: they get a Deferred in either case. The Deferreds returned to subscribers are guaranteed to not fire in the current reactor turn; instead, eventually() is used to fire them in a later turn. Look at Mark Miller's 'Concurrency Among Strangers' paper on erights.org for a description of why this property is useful. I can only be fired once.""" def __init__(self): self._fired = False self._result = None self._watchers = [] self.__repr__ = self._unfired_repr def _unfired_repr(self): return "" % (self._watchers, ) def _fired_repr(self): return " %s>" % (self._result, ) def whenFired(self): if self._fired: return eventual.fireEventually(self._result) d = defer.Deferred() self._watchers.append(d) return d def fire(self, result): assert not self._fired self._fired = True self._result = result for w in self._watchers: eventual.eventually(w.callback, result) del self._watchers self.__repr__ = self._fired_repr foolscap-0.13.1/src/foolscap/pb.py0000644000076500000240000012033313204511065017372 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test.test_pb -*- import os.path, weakref, binascii, re from warnings import warn from zope.interface import implements from twisted.internet import (reactor, defer, protocol, error, interfaces, endpoints) from twisted.application import service from twisted.python.failure import Failure from twisted.python.deprecate import deprecated from twisted.python.versions import Version from foolscap import ipb, base32, negotiate, broker, eventual, storage from foolscap import connection, util, info from foolscap.connections import tcp from foolscap.referenceable import SturdyRef from .furl import BadFURLError from foolscap.tokens import PBError, BananaError, WrongTubIdError, \ WrongNameError, NoLocationError from foolscap.reconnector import Reconnector from foolscap.logging import log as flog from foolscap.logging import log from foolscap.logging import publish as flog_publish from foolscap.logging.log import UNUSUAL from foolscap import crypto class Listener(protocol.ServerFactory, service.Service): """I am responsible for a single listening port, which connects to a single Tub. I listen on an Endpoint, and can be constructed with either the Endpoint, or a string (which I will pass to serverFromString()).""" # this also serves as the ServerFactory def __init__(self, tub, endpoint_or_description, _test_options={}, negotiationClass=negotiate.Negotiation): assert isinstance(tub, Tub) self._tub = tub if interfaces.IStreamServerEndpoint.providedBy(endpoint_or_description): self._ep = endpoint_or_description elif isinstance(endpoint_or_description, str): self._ep = endpoints.serverFromString(reactor, endpoint_or_description) else: raise TypeError("I require an endpoint, or a string description that can be turned into one") self._lp = None self._test_options = _test_options self._negotiationClass = negotiationClass self._redirects = {} def startService(self): service.Service.startService(self) d = self._ep.listen(self) def _listening(lp): self._lp = lp d.addCallback(_listening) def stopService(self): service.Service.stopService(self) if self._lp: return self._lp.stopListening() @deprecated(Version("Foolscap", 0, 12, 0), # "please use .." "pre-allocated port numbers") def getPortnum(self): """When this Listener was created with a port string of '0' or 'tcp:0' (meaning 'please allocate me something'), and if the Listener is active (it is attached to a Tub which is in the 'running' state), this method will return the port number that was allocated. This is useful for the following pattern:: t = Tub() l = t.listenOn('tcp:0') t.setLocation('localhost:%d' % l.getPortnum()) """ assert self._lp return self._lp.getHost().port def __repr__(self): return ("" % (abs(id(self)), str(self._ep), str(self._tub.tubID))) def addRedirect(self, tubID, location): assert tubID is not None self._redirects[tubID] = location def removeRedirect(self, tubID): del self._redirects[tubID] def buildProtocol(self, addr): """Return a Broker attached to me (as the service provider). """ lp = log.msg("%s accepting connection from %s" % (self, addr), addr=(addr.host, addr.port), facility="foolscap.listener") proto = self._negotiationClass(logparent=lp) ci = info.ConnectionInfo() ci._set_listener_description(self._describe()) ci._set_listener_status("negotiating") proto.initServer(self, ci) proto.factory = self return proto def lookupTubID(self, tubID): tub = None if tubID == self._tub.tubID: tub = self._tub return (tub, self._redirects.get(tubID)) def _describe(self): desc = "Listener" if self._lp: desc += " on %s" % str(self._lp.getHost()) return desc def generateSwissnumber(bits): bytes = os.urandom(bits/8) return base32.encode(bytes) class Tub(service.MultiService): """I am a presence in the PB universe, also known as a Tub. I am a Service (in the twisted.application.service.Service sense), so you either need to call my startService() method before using me, or setServiceParent() me to a running service. This is the primary entry point for all PB-using applications, both clients and servers. I am known to the outside world by a base URL, which may include authentication information (a yURL). This is my 'TubID'. I contain Referenceables, and manage RemoteReferences to Referenceables that live in other Tubs. @param certData: if provided, use it as a certificate rather than generating a new one. This is a PEM-encoded private/public keypair, as returned by Tub.getCertData() @param certFile: if provided, the Tub will store its certificate in this file. If the file does not exist when the Tub is created, the Tub will generate a new certificate and store it here. If the file does exist, the certificate will be loaded from this file. The simplest way to use the Tub is to choose a long-term location for the certificate, use certFile= to tell the Tub about it, and then let the Tub manage its own certificate. You may provide certData, or certFile, (or neither), but not both. @param _test_options: a dictionary of options that can influence connection connection negotiation. Currently defined keys are: - debug_slow: if True, wait half a second between each negotiation response @ivar brokers: maps TubIDs to L{Broker} instances @ivar referenceToName: maps Referenceable to a name @ivar nameToReference: maps name to Referenceable @type tubID: string @ivar tubID: a global identifier for this Tub, possibly including authentication information, hash of SSL certificate """ implements(ipb.ITub) unsafeTracebacks = True # TODO: better way to enable this logLocalFailures = False logRemoteFailures = False debugBanana = False NAMEBITS = 160 # length of swissnumber for each reference TUBIDBITS = 16 # length of non-crypto tubID negotiationClass = negotiate.Negotiation brokerClass = broker.Broker keepaliveTimeout = 4*60 # ping when connection has been idle this long disconnectTimeout = None # disconnect after this much idle time tubID = None def __init__(self, certData=None, certFile=None, _test_options={}): service.MultiService.__init__(self) self.setup(_test_options) if certFile: self.setupEncryptionFile(certFile) else: self.setupEncryption(certData) def __repr__(self): return "" % self.tubID def setupEncryptionFile(self, certFile): try: certData = open(certFile, "rb").read() except EnvironmentError: certData = None self.setupEncryption(certData) if certData is None: f = open(certFile, "wb") f.write(self.getCertData()) f.close() def setupEncryption(self, certData): if certData: cert = crypto.loadCertificate(certData) else: cert = self.createCertificate() self.myCertificate = cert self.tubID = crypto.digest32(cert.digest("sha1")) def make_incarnation(self): unique = binascii.b2a_hex(os.urandom(8)) # TODO: it'd be nice to have a sequential component, so incarnations # could be ordered, but it requires disk space sequential = None self.incarnation = (unique, sequential) self.incarnation_string = unique def getIncarnationString(self): return self.incarnation_string def setup(self, _test_options): self._test_options = _test_options self.logger = flog.theLogger self.listeners = [] self.locationHints = [] # duplicate-connection management self.make_incarnation() # the master_table records the master-seqnum we used for the last # established connection with the given tubid. It only contains # entries for which we were the master. self.master_table = {} # k:tubid, v:seqnum # the slave_table records the (master-IR,master-seqnum) pair for the # last established connection with the given tubid. It only contains # entries for which we were the slave. self.slave_table = {} # k:tubid, v:(master-IR,seqnum) # local Referenceables self.nameToReference = weakref.WeakValueDictionary() self.referenceToName = weakref.WeakKeyDictionary() self.strongReferences = [] self.nameLookupHandlers = [] # remote stuff. Most of these use a TubRef as a dictionary key self.tubConnectors = {} # maps TubRef to a TubConnector self.waitingForBrokers = {} # maps TubRef to list of Deferreds self.brokers = {} # maps TubRef to a Broker that connects to them self.reconnectors = [] self._connectionHandlers = {"tcp": tcp.default()} self._activeConnectors = [] self._pending_getReferences = [] # list of (d, furl) pairs self._logport = None self._logport_furl = None self._logport_furlfile = None self._log_gatherer_furls = [] self._log_gatherer_furlfile = None self._log_gatherer_connectors = {} # maps furl to reconnector self._handle_old_duplicate_connections = False self._expose_remote_exception_types = True self.accept_gifts = True def setOption(self, name, value): if name == "logLocalFailures": # log (with log.err) any exceptions that occur during the # execution of a local Referenceable's method, which is invoked # on behalf of a remote caller. These exceptions are reported to # the remote caller through their callRemote's Deferred as usual: # this option enables logging on the callee's side (i.e. our # side) as well. # # TODO: This does not yet include Violations which were raised # because the inbound callRemote had arguments that didn't meet # our specifications. But it should. self.logLocalFailures = value elif name == "logRemoteFailures": # log (with log.err) any exceptions that occur during the # execution of a remote Referenceabe's method, invoked on behalf # of a local RemoteReference.callRemote(). These exceptions are # reported to our local caller through the usual Deferred.errback # mechanism: this enables logging on the caller's side (i.e. our # side) as well. self.logRemoteFailures = value elif name == "keepaliveTimeout": self.keepaliveTimeout = value elif name == "disconnectTimeout": self.disconnectTimeout = value elif name == "logport-furlfile": self.setLogPortFURLFile(value) elif name == "log-gatherer-furl": self.setLogGathererFURL(value) elif name == "log-gatherer-furlfile": self.setLogGathererFURLFile(value) elif name == "bridge-twisted-logs": assert value is not False, "cannot unbridge twisted logs" if value is True: return flog.bridgeLogsFromTwisted(self.tubID) else: # for tests, bridge logs from a specific twisted LogPublisher return flog.bridgeLogsFromTwisted(self.tubID, twisted_logger=value) elif name == "handle-old-duplicate-connections": if value is True: value = 60 self._handle_old_duplicate_connections = int(value) elif name == "expose-remote-exception-types": self._expose_remote_exception_types = bool(value) elif name == "accept-gifts": self.accept_gifts = bool(value) else: raise KeyError("unknown option name '%s'" % name) def removeAllConnectionHintHandlers(self): self._connectionHandlers = {} def addConnectionHintHandler(self, hint_type, handler): assert ipb.IConnectionHintHandler.providedBy(handler) self._connectionHandlers[hint_type] = handler def setLogGathererFURL(self, gatherer_furl_or_furls): assert not self._log_gatherer_furls if isinstance(gatherer_furl_or_furls, basestring): self._log_gatherer_furls.append(gatherer_furl_or_furls) else: self._log_gatherer_furls.extend(gatherer_furl_or_furls) self._maybeConnectToGatherer() def setLogGathererFURLFile(self, gatherer_furlfile): assert not self._log_gatherer_furlfile self._log_gatherer_furlfile = gatherer_furlfile self._maybeConnectToGatherer() def _maybeConnectToGatherer(self): if not self.locationHints: return furls = [] if self._log_gatherer_furls: furls.extend(self._log_gatherer_furls) if self._log_gatherer_furlfile: try: # allow multiple lines for line in open(self._log_gatherer_furlfile, "r").readlines(): furl = line.strip() if furl: furls.append(furl) except EnvironmentError: pass for f in furls: if f in self._log_gatherer_connectors: continue connector = self.connectTo(f, self._log_gatherer_connected) self._log_gatherer_connectors[f] = connector def _log_gatherer_connected(self, rref): # we want the logport's furl to be nailed down now, so we'll use the # right (persistent) name even if the user never calls # tub.getLogPortFURL() directly. ignored = self.getLogPortFURL() del ignored tubID = self.tubID rref.callRemoteOnly('logport', tubID, self.getLogPort()) def getLogPort(self): if not self.locationHints: raise NoLocationError return self._maybeCreateLogPort() def _maybeCreateLogPort(self): if not self._logport: self._logport = flog_publish.LogPublisher(self.logger) return self._logport def setLogPortFURLFile(self, furlfile): self._logport_furlfile = furlfile self._maybeCreateLogPortFURLFile() def _maybeCreateLogPortFURLFile(self): if not self._logport_furlfile: return if not self.locationHints: return # getLogPortFURL() creates the logport-furlfile as a side-effect ignored = self.getLogPortFURL() del ignored def getLogPortFURL(self): if not self.locationHints: raise NoLocationError if self._logport_furl: return self._logport_furl furlfile = self._logport_furlfile # the Tub must be running and configured (setLocation) by now self._logport_furl = self.registerReference(self.getLogPort(), furlFile=furlfile) return self._logport_furl def log(self, *args, **kwargs): kwargs['tubID'] = self.tubID return log.msg(*args, **kwargs) def createCertificate(self): return crypto.createCertificate() def getCertData(self): # the string returned by this method can be used as the certData= # argument to create a new Tub with the same identity. TODO: actually # test this, I don't know if dump/keypair.newCertificate is the right # pair of methods. return self.myCertificate.dumpPEM() def setLocation(self, *hints): """Tell this service what its location is: a host:port description of how to reach it from the outside world. You need to use this because the Tub can't do it without help. If you do a C{s.listenOn('tcp:1234')}, and the host is known as C{foo.example.com}, then it would be appropriate to do:: s.setLocation('foo.example.com:1234') You must set the location before you can register any references. Tubs can have multiple location hints, just provide multiple arguments. """ if self.locationHints: raise PBError("Tub.setLocation() can only be called once") self.locationHints = hints self._maybeCreateLogPortFURLFile() self._maybeConnectToGatherer() @deprecated(Version("Foolscap", 0, 12, 0), # "please use .." "user-provided hostnames") def setLocationAutomatically(self, *extra_addresses): """Determine one of this host's publically-visible IP addresses and use it to set our location. This uses whatever source address would be used to get to a well-known public host (A.ROOT-SERVERS.NET), which is effectively the interface on which a default route lives. This is neither very pretty (IP address instead of hostname) nor guaranteed to work (it may very well be a 192.168 'private' address), but for publically-visible hosts this will probably produce a useable FURL. This method returns a Deferred that will fire once the location is actually established. Calls to registerReference() must be put off until the location has been set. And of course, you must call listenOn() before calling setLocationAutomatically().""" # first, make sure the reactor is actually running, by using the # eventual-send queue d = eventual.fireEventually() def _reactor_running(res): assert self.running # we can't use get_local_ip_for until the reactor is running return util.get_local_ip_for() d.addCallback(_reactor_running) def _got_local_ip(local_address): local_addresses = set(extra_addresses) if local_address: local_addresses.add(local_address) local_addresses.add("127.0.0.1") locations = set() for l in self.getListeners(): portnum = l.getPortnum() for addr in local_addresses: locations.add("%s:%d" % (addr, portnum)) locations = list(locations) locations.sort() assert len(locations) >= 1 location = ",".join(locations) self.setLocation(location) d.addCallback(_got_local_ip) return d def listenOn(self, what, _test_options={}): """Start listening for connections. @type what: string @param what: a L{twisted.internet.endpoints.serverFromString} -style description @param _test_options: a dictionary of options that can influence connection negotiation before the target Tub has been determined @return: The Listener object that was created. This can be used to stop listening later on.""" if what in ("0", "tcp:0"): warningString = ("Tub.listenOn('tcp:0') was deprecated " "in Foolscap 0.12.0; please use pre-allocated " "port numbers instead") warn(warningString, DeprecationWarning, stacklevel=2) if isinstance(what, str) and re.search(r"^\d+$", what): warn("Tub.listenOn('12345') was deprecated " "in Foolscap 0.12.0; please use qualified endpoint " "descriptions like 'tcp:12345'", DeprecationWarning, stacklevel=2) what = "tcp:%s" % what l = Listener(self, what, _test_options, self.negotiationClass) self.listeners.append(l) l.setServiceParent(self) return l def stopListeningOn(self, l): # this returns a Deferred when the port is shut down self.listeners.remove(l) return l.disownServiceParent() def getListeners(self): """Return the set of Listener objects that allow the outside world to connect to this Tub.""" return self.listeners[:] def getTubID(self): return self.tubID def getShortTubID(self): return self.tubID[:4] def getConnectionInfoForFURL(self, furl): try: tubref = SturdyRef(furl).getTubRef() except (ValueError, BadFURLError): return None # unparseable FURL return self._getConnectionInfoForTubRef(tubref) def _getConnectionInfoForTubRef(self, tubref): if tubref in self.brokers: return self.brokers[tubref].getConnectionInfo() if tubref in self.tubConnectors: return self.tubConnectors[tubref].getConnectionInfo() return None # currently have no established or in-progress connection def connectorStarted(self, c): assert self.running # TODO: why a list? shouldn't there only ever be one TubConnector? self._activeConnectors.append(c) def connectorFinished(self, c): if c in self._activeConnectors: self._activeConnectors.remove(c) def startService(self): service.MultiService.startService(self) for d,sturdy in self._pending_getReferences: d1 = eventual.fireEventually(sturdy) d1.addCallback(self.getReference) d1.addBoth(lambda res, d=d: d.callback(res)) del self._pending_getReferences for rc in self.reconnectors: eventual.eventually(rc.startConnecting, self) def _tubsAreNotRestartable(self, *args, **kwargs): raise RuntimeError("Sorry, but Tubs cannot be restarted.") def _tubHasBeenShutDown(self, *args, **kwargs): raise RuntimeError("Sorry, but this Tub has been shut down.") def stopService(self): # note that once you stopService a Tub, I cannot be restarted. (at # least this code is not designed to make that possible.. it might be # doable in the future). assert self.running self.startService = self._tubsAreNotRestartable self.getReference = self._tubHasBeenShutDown self.connectTo = self._tubHasBeenShutDown # Tell everything to shut down now. We assume that it will stop # twitching by the next tick, so Trial unit tests won't complain # about a dirty reactor. We wait on a few things that might not # behave. dl = [] for rc in list(self.reconnectors): rc.stopConnecting() del self.reconnectors for c in list(self._activeConnectors): c.shutdown() why = Failure(error.ConnectionDone("Tub.stopService was called")) for b in self.brokers.values(): broker_disconnected = defer.Deferred() dl.append(broker_disconnected) b._notifyOnConnectionLost( lambda d=broker_disconnected: d.callback(None) ) b.shutdown(why, fireDisconnectWatchers=False) d = defer.DeferredList(dl) d.addCallback(lambda _: service.MultiService.stopService(self)) d.addCallback(eventual.fireEventually) return d def generateSwissnumber(self, bits): return generateSwissnumber(bits) def buildURL(self, name): # TODO: IPv6 dotted-quad addresses have colons, but need to have # host:port hints = ",".join(self.locationHints) return "pb://" + self.tubID + "@" + hints + "/" + name def registerReference(self, ref, name=None, furlFile=None): """Make a Referenceable available to the outside world. A URL is returned which can be used to access this object. This registration will remain in effect (and the Tub will retain a reference to the object to keep it meaningful) until explicitly unregistered, or the Tub is shut down. @type name: string (optional) @param name: if provided, the object will be registered with this name. If not, a random (unguessable) string will be used. @param furlFile: if provided, get the name from this file (if it exists), and write the new FURL to this file. If 'name=' is also provided, it is used for the name, but the FURL is still written to this file. @rtype: string @return: the URL which points to this object. This URL can be passed to Tub.getReference() in any Tub on any host which can reach this one. """ if not self.locationHints: raise NoLocationError("you must setLocation() before " "you can registerReference()") oldfurl = None if furlFile: try: oldfurl = open(furlFile, "r").read().strip() except EnvironmentError: pass if oldfurl: sr = SturdyRef(oldfurl) if name is None: name = sr.name if self.tubID != sr.tubID: raise WrongTubIdError("I cannot keep using the old FURL from %s" " because it does not have the same" " TubID as I do (%s)" % (furlFile, self.tubID)) if name != sr.name: raise WrongNameError("I cannot keep using the old FURL from %s" " because you called registerReference" " with a new name (%s)" % (furlFile, name)) name = self._assignName(ref, name) assert name if ref not in self.strongReferences: self.strongReferences.append(ref) furl = self.buildURL(name) if furlFile: need_to_chmod = not os.path.exists(furlFile) f = open(furlFile, "w") f.write(furl + "\n") f.close() if need_to_chmod: # XXX: open-to-chmod race here os.chmod(furlFile, 0600) return furl # this is called by either registerReference or by # getOrCreateURLForReference def _assignName(self, ref, preferred_name=None): """Make a Referenceable available to the outside world, but do not retain a strong reference to it. If we must create a new name, use preferred_name. If that is None, use a random unguessable name. """ if not self.locationHints: # without a location, there is no point in giving it a name return None if self.referenceToName.has_key(ref): return self.referenceToName[ref] name = preferred_name if not name: name = self.generateSwissnumber(self.NAMEBITS) self.referenceToName[ref] = name self.nameToReference[name] = ref return name def getReferenceForName(self, name): if name in self.nameToReference: return self.nameToReference[name] for lookup in self.nameLookupHandlers: ref = lookup(name) if ref: if ref not in self.referenceToName: self.referenceToName[ref] = name return ref # don't reveal the full swissnum hint = name[:2] raise KeyError("unable to find reference for name starting with '%s'" % hint) def getReferenceForURL(self, url): # TODO: who should this be used by? sturdy = SturdyRef(url) assert sturdy.tubID == self.tubID return self.getReferenceForName(sturdy.name) def getOrCreateURLForReference(self, ref): """Return the global URL for the reference, if there is one, or None if there is not.""" name = self._assignName(ref) if name: return self.buildURL(name) return None def revokeReference(self, ref): # TODO pass def unregisterURL(self, url): sturdy = SturdyRef(url) name = sturdy.name ref = self.nameToReference[name] del self.nameToReference[name] del self.referenceToName[ref] self.revokeReference(ref) def unregisterReference(self, ref): name = self.referenceToName[ref] url = self.buildURL(name) sturdy = SturdyRef(url) name = sturdy.name del self.nameToReference[name] del self.referenceToName[ref] if ref in self.strongReferences: self.strongReferences.remove(ref) self.revokeReference(ref) def registerNameLookupHandler(self, lookup): """Add a function to help convert names to Referenceables. When remote systems pass a FURL to their Tub.getReference(), our Tub will be asked to locate a Referenceable for the name inside that furl. The normal mechanism for this is to look at the table maintained by registerReference() and unregisterReference(). If the name does not exist in that table, other 'lookup handler' functions are given a chance. Each lookup handler is asked in turn, and the first which returns a non-None value wins. This may be useful for cases where the furl represents an object that lives on disk, or is generated on demand: rather than creating all possible Referenceables at startup, the lookup handler can create or retrieve the objects only when someone asks for them. Note that constructing the FURLs of these objects may be non-trivial. It is safe to create an object, use tub.registerReference in one invocation of a program to obtain (and publish) the furl, parse the furl to extract the name, save the contents of the object on disk, then in a later invocation of the program use a lookup handler to retrieve the object from disk. This approach means the objects that are created in a given invocation stick around (inside tub.strongReferences) for the rest of that invocation. An alternatve approach is to create the object but *not* use tub.registerReference, but in that case you have to construct the FURL yourself, and the Tub does not currently provide any support for doing this robustly. @param lookup: a callable which accepts a name (as a string) and returns either a Referenceable or None. Note that these strings should not contain a slash, a question mark, or an ampersand, as these are reserved in the FURL for later expansion (to add parameters beyond the object name) """ self.nameLookupHandlers.append(lookup) def unregisterNameLookupHandler(self, lookup): self.nameLookupHandlers.remove(lookup) def getReference(self, sturdyOrURL): """Acquire a RemoteReference for the given SturdyRef/URL. The Tub must be running (i.e. Tub.startService()) when this is invoked. Future releases may relax this requirement. @return: a Deferred that fires with the RemoteReference. Any failures are returned asynchronously. """ return defer.maybeDeferred(self._getReference, sturdyOrURL) def _getReference(self, sturdyOrURL): if isinstance(sturdyOrURL, SturdyRef): sturdy = sturdyOrURL else: sturdy = SturdyRef(sturdyOrURL) if not self.running: # queue their request for service once the Tub actually starts log.msg("Tub.getReference(%s) queued until Tub.startService called" % sturdy, facility="foolscap.tub") d = defer.Deferred() self._pending_getReferences.append((d, sturdy)) return d name = sturdy.name d = self.getBrokerForTubRef(sturdy.getTubRef()) d.addCallback(lambda b: b.getYourReferenceByName(name)) return d def connectTo(self, _sturdyOrURL, _cb, *args, **kwargs): """Establish (and maintain) a connection to a given PBURL. I establish a connection to the PBURL and run a callback to inform the caller about the newly-available RemoteReference. If the connection is lost, I schedule a reconnection attempt for the near future. If that one fails, I keep trying at longer and longer intervals (exponential backoff). I accept a callback which will be fired each time a connection attempt succeeds. This callback is run with the new RemoteReference and any additional args/kwargs provided to me. The callback should then use rref.notifyOnDisconnect() to get a message when the connection goes away. At some point after it goes away, the Reconnector will reconnect. The Tub must be running (i.e. Tub.startService()) when this is invoked. Future releases may relax this requirement. I return a Reconnector object. When you no longer want to maintain this connection, call the stopConnecting() method on the Reconnector. I promise to not invoke your callback after you've called stopConnecting(), even if there was already a connection attempt in progress. If you had an active connection before calling stopConnecting(), you will still have access to it, until it breaks on its own. (I will not attempt to break existing connections, I will merely stop trying to create new ones). All my Reconnector objects will be shut down when the Tub is stopped. Usage:: def _got_ref(rref, arg1, arg2): rref.callRemote('hello again') # etc rc = tub.connectTo(_got_ref, 'arg1', 'arg2') ... rc.stopConnecting() # later """ rc = Reconnector(_sturdyOrURL, _cb, args, kwargs) if self.running: rc.startConnecting(self) else: self.log("Tub.connectTo(%s) queued until Tub.startService called" % _sturdyOrURL, level=UNUSUAL) self.reconnectors.append(rc) return rc def serialize(self, obj): b = broker.StorageBroker(None) b.setTub(self) d = storage.serialize(obj, banana=b) return d def unserialize(self, data): b = broker.StorageBroker(None) b.setTub(self) d = storage.unserialize(data, banana=b) assert isinstance(d, defer.Deferred) return d # beyond here are internal methods, not for use by application code # _removeReconnector is called by the Reconnector def _removeReconnector(self, rc): self.reconnectors.remove(rc) def getBrokerForTubRef(self, tubref): if tubref in self.brokers: return defer.succeed(self.brokers[tubref]) if tubref.getTubID() == self.tubID: b = self._createLoopbackBroker(tubref) # _createLoopbackBroker will call brokerAttached, which will add # it to self.brokers # TODO: stash this in self.brokers, so we don't create multiples return defer.succeed(b) d = defer.Deferred() if tubref not in self.waitingForBrokers: self.waitingForBrokers[tubref] = [] self.waitingForBrokers[tubref].append(d) if tubref not in self.tubConnectors: # the TubConnector will call our brokerAttached when it finishes # negotiation, which will fire waitingForBrokers[tubref]. c = connection.TubConnector(self, tubref, self._connectionHandlers) self.tubConnectors[tubref] = c c.connect() return d def _createLoopbackBroker(self, tubref): t1,t2 = broker.LoopbackTransport(), broker.LoopbackTransport() t1.setPeer(t2); t2.setPeer(t1) n = negotiate.Negotiation() params = n.loopbackDecision() ci = info.ConnectionInfo() b1 = self.brokerClass(tubref, params, connectionInfo=ci) b2 = self.brokerClass(tubref, params) # we treat b1 as "our" broker, and b2 as "theirs", and we pretend # that b2 has just connected to us. We keep track of b1, and b2 keeps # track of us. b1.setTub(self) b2.setTub(self) t1.protocol = b1; t2.protocol = b2 b1.makeConnection(t1); b2.makeConnection(t2) ci._set_connected(True) ci._set_winning_hint("loopback") ci._set_connection_status("loopback", "connected") ci._set_established_at(b1.creation_timestamp) self.brokerAttached(tubref, b1, False) return b1 def connectionFailed(self, tubref, why): # we previously initiated an outbound TubConnector to this tubref, but # it was unable to establish a connection. 'why' is the most useful # Failure that occurred (i.e. it is a NegotiationError if we made it # that far, otherwise it's a ConnectionFailed). if tubref in self.tubConnectors: del self.tubConnectors[tubref] if tubref in self.brokers: # oh, but fortunately an inbound connection must have succeeded. # Nevermind. return # inform hopeful Broker-waiters that they aren't getting one if tubref in self.waitingForBrokers: waiting = self.waitingForBrokers[tubref] del self.waitingForBrokers[tubref] for d in waiting: d.errback(why) def brokerAttached(self, tubref, broker, isClient): assert self.running assert tubref if tubref in self.tubConnectors: # we initiated an outbound connection to this tubref if not isClient: # however, the connection we got was from an inbound # connection. The completed (inbound) connection wins, so # abandon the outbound TubConnector self.tubConnectors[tubref].shutdown() # we don't need the TubConnector any more del self.tubConnectors[tubref] if tubref in self.brokers: # this shouldn't happen: acceptDecision is supposed to drop any # existing old connection first. self.log("ERROR: unexpected duplicate connection from %s" % tubref) raise BananaError("unexpected duplicate connection") self.brokers[tubref] = broker # now inform everyone who's been waiting on it if tubref in self.waitingForBrokers: for d in self.waitingForBrokers[tubref]: eventual.eventually(d.callback, broker) del self.waitingForBrokers[tubref] def brokerDetached(self, broker, why): # a loopback connection will produce two Brokers that both use the # same tubref. Both will shut down about the same time. Make sure # this doesn't confuse us. # the Broker will have already severed all active references for tubref in self.brokers.keys(): if self.brokers[tubref] is broker: del self.brokers[tubref] def debug_listBrokers(self): # return a list of (tubref, inbound, outbound) tuples. The tubref # tells you which broker this is, 'inbound' is a list of # InboundDelivery objects (one per outstanding inbound message), and # 'outbound' is a list of PendingRequest objects (one per message # that's waiting on a remote broker to complete). output = [] all_brokers = self.brokers.items() for tubref,_broker in all_brokers: inbound = _broker.inboundDeliveryQueue[:] outbound = [pr for (reqID, pr) in sorted(_broker.waitingForAnswers.items()) ] output.append( (str(tubref), inbound, outbound) ) output.sort(lambda x,y: cmp( (len(x[1]), len(x[2])), (len(y[1]), len(y[2])) )) return output foolscap-0.13.1/src/foolscap/promise.py0000644000076500000240000002453412766553111020467 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test.test_promise -*- from twisted.python.failure import Failure from twisted.internet import defer from foolscap.eventual import eventually EVENTUAL, CHAINED, NEAR, BROKEN = range(4) class UsageError(Exception): """Raised when you do something inappropriate to a Promise.""" def _ignore(results): pass class Promise(object): """I am a promise of a future result. I am a lot like a Deferred, except that my promised result is usually an instance. I make it possible to schedule method invocations on this future instance, returning Promises for the results. Promises are always in one of three states: Eventual, Fulfilled, and Broken. (see http://www.erights.org/elib/concurrency/refmech.html for a pretty picture). They start as Eventual, meaning we do not yet know whether they will resolve or not. In this state, method invocations are queued. Eventually the Promise will be 'resolved' into either the Fulfilled or the Broken state. Fulfilled means that the promise contains a live object to which methods can be dispatched synchronously. Broken promises are incapable of invoking methods: they all result in Failure. Method invocation is always asynchronous: it always returns a Promise. The only thing you can do with a promise 'p1' is to perform an eventual-send on it, like so:: sendOnly(p1).foo(args) # ignores the result p2 = send(p1).bar(args) # creates a Promise for the result p2 = p1.bar(args) # same as send(p1).bar(args) Or wait for it to resolve, using one of the following:: d = when(p); d.addCallback(cb) # provides a Deferred p._then(cb, *args, **kwargs) # like when(p).addCallback(cb,*a,**kw) p._except(cb, *args, **kwargs) # like when(p).addErrback(cb,*a,**kw) The _then and _except forms return the same Promise. You can set up chains of calls that will be invoked in the future, using a dataflow style, like this:: p = getPromiseForServer() d = p.getDatabase('db1') r = d.getRecord(name) def _print(record): print 'the record says', record def _oops(failure): print 'something failed:', failure r._then(_print) r._except(_oops) Or all collapsed in one sequence like:: getPromiseForServer().getDatabase('db1').getRecord(name)._then(_print) The eventual-send will eventually invoke the method foo(args) on the promise's resolution. This will return a new Promise for the results of that method call. """ # all our internal methods are private, to avoid a confusing lack of an # error message if someone tries to make a synchronous method call on us # with a name that happens to match an internal one. _state = EVENTUAL _useDataflowStyle = True # enables p.foo(args) def __init__(self): self._watchers = [] self._pendingMethods = [] # list of (methname, args, kwargs, p) # _then and _except are our only public methods. All other access is # through normal (not underscore-prefixed) attribute names, which # indicate names of methods on the target object that should be called # later. def _then(self, cb, *args, **kwargs): d = self._wait_for_resolution() d.addCallback(cb, *args, **kwargs) d.addErrback(lambda ignore: None) return self def _except(self, cb, *args, **kwargs): d = self._wait_for_resolution() d.addErrback(cb, *args, **kwargs) return self # everything beyond here is private to this module def __repr__(self): return "" % id(self) def __getattr__(self, name): if not self._useDataflowStyle: raise AttributeError("no such attribute %s" % name) def newmethod(*args, **kwargs): return self._send(name, args, kwargs) return newmethod # _send and _sendOnly are used by send() and sendOnly(). _send is also # used by regular attribute access. def _send(self, methname, args, kwargs): """Return a Promise (for the result of the call) when the call is eventually made. The call is guaranteed to not fire in this turn.""" # this is called by send() p, resolver = makePromise() if self._state in (EVENTUAL, CHAINED): self._pendingMethods.append((methname, args, kwargs, resolver)) else: eventually(self._deliver, methname, args, kwargs, resolver) return p def _sendOnly(self, methname, args, kwargs): """Send a message like _send, but discard the result.""" # this is called by sendOnly() if self._state in (EVENTUAL, CHAINED): self._pendingMethods.append((methname, args, kwargs, _ignore)) else: eventually(self._deliver, methname, args, kwargs, _ignore) # _wait_for_resolution is used by when(), as well as _then and _except def _wait_for_resolution(self): """Return a Deferred that will fire (with whatever was passed to _resolve) when this Promise moves to a RESOLVED state (either NEAR or BROKEN).""" # this is called by when() if self._state in (EVENTUAL, CHAINED): d = defer.Deferred() self._watchers.append(d) return d if self._state == NEAR: return defer.succeed(self._target) # self._state == BROKEN return defer.fail(self._target) # _resolve is our resolver method, and is handed out by makePromise() def _resolve(self, target_or_failure): """Resolve this Promise to refer to the given target. If called with a Failure, the Promise is now BROKEN. _resolve may only be called once.""" # E splits this method into two pieces resolve(result) and # smash(problem). It is easier for us to keep them in one piece, # because d.addBoth(p._resolve) is convenient. if self._state != EVENTUAL: raise UsageError("Promises may not be resolved multiple times") self._resolve2(target_or_failure) # the remaining methods are internal, for use by this class only def _resolve2(self, target_or_failure): # we may be called with a Promise, an immediate value, or a Failure if isinstance(target_or_failure, Promise): self._state = CHAINED when(target_or_failure).addBoth(self._resolve2) return if isinstance(target_or_failure, Failure): self._break(target_or_failure) return self._target = target_or_failure self._deliver_queued_messages() self._state = NEAR def _break(self, failure): # TODO: think about what you do to break a resolved promise. Once the # Promise is in the NEAR state, it can't be broken, but eventually # we're going to have a FAR state, which *can* be broken. """Put this Promise in the BROKEN state.""" if not isinstance(failure, Failure): raise UsageError("Promises must be broken with a Failure") if self._state == BROKEN: raise UsageError("Broken Promises may not be re-broken") self._target = failure if self._state in (EVENTUAL, CHAINED): self._deliver_queued_messages() self._state == BROKEN def _invoke_method(self, name, args, kwargs): if isinstance(self._target, Failure): return self._target method = getattr(self._target, name) res = method(*args, **kwargs) return res def _deliverOneMethod(self, methname, args, kwargs): method = getattr(self._target, methname) return method(*args, **kwargs) def _deliver(self, methname, args, kwargs, resolver): # the resolver will be fired with both success and Failure t = self._target if isinstance(t, Promise): resolver(t._send(methname, args, kwargs)) elif isinstance(t, Failure): resolver(t) else: d = defer.maybeDeferred(self._deliverOneMethod, methname, args, kwargs) d.addBoth(resolver) def _deliver_queued_messages(self): for (methname, args, kwargs, resolver) in self._pendingMethods: eventually(self._deliver, methname, args, kwargs, resolver) del self._pendingMethods # Q: what are the partial-ordering semantics between queued messages # and when() clauses that are waiting on this Promise to be resolved? for d in self._watchers: eventually(d.callback, self._target) del self._watchers def resolvedPromise(resolution): p = Promise() p._resolve(resolution) return p def makePromise(): p = Promise() return p, p._resolve class _MethodGetterWrapper(object): def __init__(self, callback): self.cb = [callback] def __getattr__(self, name): if name.startswith("_"): raise AttributeError("method %s is probably private" % name) cb = self.cb[0] # avoid bound-methodizing def newmethod(*args, **kwargs): return cb(name, args, kwargs) return newmethod def send(o): """Make an eventual-send call on object C{o}. Use this as follows:: p = send(o).foo(args) C{o} can either be a Promise or an immediate value. The arguments can either be promises or immediate values. send() always returns a Promise, and the o.foo(args) method invocation always takes place in a later reactor turn. Many thanks to Mark Miller for suggesting this syntax to me. """ if isinstance(o, Promise): return _MethodGetterWrapper(o._send) p = resolvedPromise(o) return _MethodGetterWrapper(p._send) def sendOnly(o): """Make an eventual-send call on object C{o}, and ignore the results. """ if isinstance(o, Promise): return _MethodGetterWrapper(o._sendOnly) # this is a little bit heavyweight for a simple eventually(), but it # makes the code simpler p = resolvedPromise(o) return _MethodGetterWrapper(p._sendOnly) def when(p): """Turn a Promise into a Deferred that will fire with the enclosed object when it is ready. Use this when you actually need to schedule something to happen in a synchronous fashion. Most of the time, you can just invoke methods on the Promise as if it were immediately available.""" assert isinstance(p, Promise) return p._wait_for_resolution() foolscap-0.13.1/src/foolscap/reconnector.py0000644000076500000240000001503113204160675021317 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test.test_reconnector -*- import random import time from twisted.internet import reactor from twisted.python import log from foolscap.tokens import NegotiationError, RemoteNegotiationError class ReconnectionInfo: def __init__(self): self.state = "unstarted" self.connectionInfo = None self.lastAttempt = None self.nextAttempt = None def _set_state(self, state): self.state = state # unstarted, connecting, connected, waiting def _set_connection_info(self, connectionInfo): self.connectionInfo = connectionInfo def _set_last_attempt(self, when): self.lastAttempt = when def _set_next_attempt(self, when): self.nextAttempt = when class Reconnector(object): """Establish (and maintain) a connection to a given PBURL. I establish a connection to the PBURL and run a callback to inform the caller about the newly-available RemoteReference. If the connection is lost, I schedule a reconnection attempt for the near future. If that one fails, I keep trying at longer and longer intervals (exponential backoff). My constructor accepts a callback which will be fired each time a connection attempt succeeds. This callback is run with the new RemoteReference and any additional args/kwargs provided to me. The callback should then use rref.notifyOnDisconnect() to get a message when the connection goes away. At some point after it goes away, the Reconnector will reconnect. When you no longer want to maintain this connection, call my stopConnecting() method. I promise to not invoke your callback after you've called stopConnecting(), even if there was already a connection attempt in progress. If you had an active connection before calling stopConnecting(), you will still have access to it, until it breaks on its own. (I will not attempt to break existing connections, I will merely stop trying to create new ones). """ # adapted from twisted.internet.protocol.ReconnectingClientFactory maxDelay = 3600 initialDelay = 1.0 # Note: These highly sensitive factors have been precisely measured by # the National Institute of Science and Technology. Take extreme care # in altering them, or you may damage your Internet! factor = 2.7182818284590451 # (math.e) # Phi = 1.6180339887498948 # (Phi is acceptable for use as a # factor if e is too large for your application.) jitter = 0.11962656492 # molar Planck constant times c, Joule meter/mole verbose = False def __init__(self, url, cb, args, kwargs): self._url = url self._active = False self._observer = (cb, args, kwargs) self._delay = self.initialDelay self._timer = None self._tub = None self._last_failure = None self._reconnectionInfo = ReconnectionInfo() def startConnecting(self, tub): self._tub = tub if self.verbose: log.msg("Reconnector starting for %s" % self._url) self._active = True self._connect() def stopConnecting(self): if self.verbose: log.msg("Reconnector stopping for %s" % self._url) self._active = False if self._timer: self._timer.cancel() self._timer = False if self._tub: self._tub._removeReconnector(self) def reset(self): """Reset the connection timer and try again very soon.""" self._delay = self.initialDelay if self._timer: self._timer.reset(1.0) def getDelayUntilNextAttempt(self): if not self._timer: return None return self._timer.getTime() - time.time() def getLastFailure(self): return self._last_failure def getReconnectionInfo(self): return self._reconnectionInfo def _connect(self): self._reconnectionInfo._set_state("connecting") self._reconnectionInfo._set_last_attempt(time.time()) d = self._tub.getReference(self._url) ci = self._tub.getConnectionInfoForFURL(self._url) self._reconnectionInfo._set_connection_info(ci) d.addCallbacks(self._connected, self._failed) def _connected(self, rref): if not self._active: return self._reconnectionInfo._set_state("connected") ci = self._tub.getConnectionInfoForFURL(self._url) self._reconnectionInfo._set_connection_info(ci) self._last_failure = None rref.notifyOnDisconnect(self._disconnected) cb, args, kwargs = self._observer cb(rref, *args, **kwargs) def _failed(self, f): self._last_failure = f ci = getattr(f, "_connectionInfo", None) if ci: self._reconnectionInfo._set_connection_info(ci) # I'd like to quietly handle "normal" problems (basically TCP # failures and NegotiationErrors that result from the target either # not speaking Foolscap or not hosting the Tub that we want), but not # hide coding errors or version mismatches. log_it = self.verbose # log certain unusual errors, even without self.verbose, to help # people figure out why their reconnectors aren't connecting, since # the usual getReference errback chain isn't active. This doesn't # include ConnectError (which is a parent class of # ConnectionRefusedError), so it won't fire if we just had a bad # host/port, since the way we use connection hints will provoke that # all the time. if f.check(RemoteNegotiationError, NegotiationError): log_it = True if log_it: log.msg("Reconnector._failed (furl=%s): %s" % (self._url, f)) if not self._active: return self._delay = min(self._delay * self.factor, self.maxDelay) if self.jitter: self._delay = random.normalvariate(self._delay, self._delay * self.jitter) self._retry() def _disconnected(self): self._delay = self.initialDelay self._retry() def _retry(self): if not self._active: return if self.verbose: log.msg("Reconnector scheduling retry in %ds for %s" % (self._delay, self._url)) self._reconnectionInfo._set_state("waiting") self._reconnectionInfo._set_next_attempt(time.time() + self._delay) self._timer = reactor.callLater(self._delay, self._timer_expired) def _timer_expired(self): self._timer = None self._connect() foolscap-0.13.1/src/foolscap/referenceable.py0000644000076500000240000010157013204160675021564 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test.test_sturdyref -*- # this module is responsible for sending and receiving OnlyReferenceable and # Referenceable (callable) objects. All details of actually invoking methods # live in call.py import weakref from zope.interface import interface from zope.interface import implements from twisted.python.components import registerAdapter Interface = interface.Interface from twisted.internet import defer from twisted.python import failure, log from foolscap import ipb, slicer, tokens, call BananaError = tokens.BananaError Violation = tokens.Violation from foolscap.constraint import IConstraint, ByteStringConstraint from foolscap.remoteinterface import getRemoteInterface, \ getRemoteInterfaceByName, RemoteInterfaceConstraint from foolscap.schema import constraintMap from foolscap.copyable import Copyable, RemoteCopy from foolscap.eventual import eventually, fireEventually from foolscap.furl import decode_furl class OnlyReferenceable(object): implements(ipb.IReferenceable) def processUniqueID(self): return id(self) class Referenceable(OnlyReferenceable): implements(ipb.IReferenceable, ipb.IRemotelyCallable) _interface = None _interfaceName = None # TODO: this code wants to be in an adapter, not a base class. Also, it # would be nice to cache this across the class: if every instance has the # same interfaces, they will have the same values of _interface and # _interfaceName, and it feels silly to store this data separately for # each instance. Perhaps we could compare the instance's interface list # with that of the class and only recompute this stuff if they differ. def getInterface(self): if not self._interface: self._interface = getRemoteInterface(self) if self._interface: self._interfaceName = self._interface.__remote_name__ else: self._interfaceName = None return self._interface def getInterfaceName(self): self.getInterface() return self._interfaceName def doRemoteCall(self, methodname, args, kwargs): meth = getattr(self, "remote_%s" % methodname) res = meth(*args, **kwargs) return res constraintMap[Referenceable] = RemoteInterfaceConstraint(None) class ReferenceableTracker(object): """I hold the data which tracks a local Referenceable that is in used by a remote Broker. @ivar obj: the actual object @ivar refcount: the number of times this reference has been sent to the remote end, minus the number of DECREF messages which it has sent back. When it goes to zero, the remote end has forgotten the RemoteReference, and is prepared to forget the RemoteReferenceData as soon as the DECREF message is acknowledged. @ivar clid: the connection-local ID used to represent this object on the wire. """ def __init__(self, tub, obj, puid, clid): self.tub = tub self.obj = obj self.clid = clid self.puid = puid self.refcount = 0 def send(self): """Increment the refcount. @return: True if this is the first transmission of the reference. """ self.refcount += 1 if self.refcount == 1: return True def getURL(self): if self.tub: return self.tub.getOrCreateURLForReference(self.obj) return None def decref(self, count): """Call this in response to a DECREF message from the other end. @return: True if the refcount went to zero, meaning this clid should be retired. """ assert self.refcount >= count, "decref(%d) but refcount was %d" % (count, self.refcount) self.refcount -= count if self.refcount == 0: return True return False # TODO: rather than subclassing Referenceable, ReferenceableSlicer should be # registered to use for anything which provides any RemoteInterface class ReferenceableSlicer(slicer.BaseSlicer): """I handle pb.Referenceable objects (things with remotely invokable methods, which are copied by reference). """ opentype = ('my-reference',) def slice(self, streamable, protocol): broker = self.requireBroker(protocol) puid = ipb.IReferenceable(self.obj).processUniqueID() tracker = broker.getTrackerForMyReference(puid, self.obj) if broker.remote_broker: # emit a my-reference sequence yield 'my-reference' yield tracker.clid firstTime = tracker.send() if firstTime: # this is the first time the Referenceable has crossed this # wire. In addition to the clid, send the interface name (if # any), and any URL this reference might be known by iname = ipb.IRemotelyCallable(self.obj).getInterfaceName() if iname: yield iname else: yield "" url = tracker.getURL() if url: yield url else: # when we're serializing to data, rather than to a live # connection, all of my Referenceables are turned into # their-reference sequences, to prompt the eventual recipient to # create a new connection for this object. # a big note on object lifetimes: obviously, the data cannot keep # the Referenceable alive. Use tub.registerReference() on any # Referenceable that you want to include in the serialized data, # and take steps to make sure that later incarnations of this Tub # will do the same. yield 'their-reference' yield 0 # giftID==0 tells the recipient to not try to ack it yield tracker.getURL() registerAdapter(ReferenceableSlicer, Referenceable, ipb.ISlicer) class CallableSlicer(slicer.BaseSlicer): """Bound methods are serialized as my-reference sequences with negative clid values.""" opentype = ('my-reference',) def sliceBody(self, streamable, protocol): broker = self.requireBroker(protocol) # TODO: consider this requirement, maybe based upon a Tub flag # assert ipb.ISlicer(self.obj.im_self) # or maybe even isinstance(self.obj.im_self, Referenceable) puid = id(self.obj) tracker = broker.getTrackerForMyCall(puid, self.obj) yield tracker.clid firstTime = tracker.send() if firstTime: # this is the first time the Call has crossed this wire. In # addition to the clid, send the schema name and any URL this # reference might be known by schema = self.getSchema() if schema: yield schema else: yield "" url = tracker.getURL() if url: yield url def getSchema(self): return None # TODO: not quite ready yet # callables which are actually bound methods of a pb.Referenceable # can use the schema from that s = ipb.IReferenceable(self.obj.im_self, None) if s: return s.getSchemaForMethodNamed(self.obj.im_func.__name__) # both bound methods and raw callables can also use a .schema # attribute return getattr(self.obj, "schema", None) # The CallableSlicer is activated through PBRootSlicer.slicerTable, because a # StorageBanana might want to stick with the old MethodSlicer/FunctionSlicer # for these types #registerAdapter(CallableSlicer, types.MethodType, ipb.ISlicer) class ReferenceUnslicer(slicer.BaseUnslicer): """I turn an incoming 'my-reference' sequence into a RemoteReference or a RemoteMethodReference.""" state = 0 clid = None interfaceName = None url = None inameConstraint = ByteStringConstraint() # TODO: only known RI names? urlConstraint = ByteStringConstraint() def checkToken(self, typebyte, size): if self.state == 0: if typebyte not in (tokens.INT, tokens.NEG): raise BananaError("reference ID must be an INT or NEG") elif self.state == 1: self.inameConstraint.checkToken(typebyte, size) elif self.state == 2: self.urlConstraint.checkToken(typebyte, size) else: raise Violation("too many parameters in my-reference") def receiveChild(self, obj, ready_deferred=None): assert not isinstance(obj, defer.Deferred) assert ready_deferred is None if self.state == 0: self.clid = obj self.state = 1 elif self.state == 1: # must be the interface name self.interfaceName = obj if obj == "": self.interfaceName = None self.state = 2 elif self.state == 2: # URL self.url = obj self.state = 3 else: raise BananaError("Too many my-reference parameters") def receiveClose(self): if self.clid is None: raise BananaError("sequence ended too early") tracker = self.broker.getTrackerForYourReference(self.clid, self.interfaceName, self.url) return tracker.getRef(), None def describe(self): if self.clid is None: return "" return "" % self.clid class RemoteReferenceTracker(object): """I hold the data necessary to locate (or create) a RemoteReference. @ivar url: the target Referenceable's global URL @ivar broker: the Broker which holds this RemoteReference @ivar clid: for that Broker, the your-reference CLID for the RemoteReference @ivar interfaceName: the name of a RemoteInterface object that the RemoteReference claims to implement @ivar interface: our version of a RemoteInterface object that corresponds to interfaceName @ivar received_count: the number of times the remote end has send us this object. We must send back decref() calls to match. @ivar ref: a weakref to the RemoteReference itself """ def __init__(self, parent, clid, url, interfaceName): self.broker = parent self.clid = clid # TODO: the remote end sends us a global URL, when really it should # probably send us a per-Tub name, which can can then concatenate to # their TubID if/when we pass it on to others. By accepting a full # URL, we give them the ability to sort-of spoof others. For now, we # check that their URL uses the same tubid as our broker is # expecting, but the Right Way is to just not have them send the base # part in the first place. I haven't yet made this change because I'm # not yet positive it would work.. how exactly does the base url get # sent, anyway? What about Tubs visible through multiple names? self.url = url if url is not None: # unit tests frequently set url=None assert self.broker.remote_tubref expected_tubid = self.broker.remote_tubref.getTubID() url_tubid = SturdyRef(url).getTubRef().getTubID() if expected_tubid != url_tubid: raise BananaError("inbound reference claims bad tubid, %s vs %s" % (expected_tubid, url_tubid)) self.interfaceName = interfaceName self.interface = getRemoteInterfaceByName(interfaceName) self.received_count = 0 self.ref = None def __repr__(self): s = "" % (self.clid, self.url) return s def getURL(self): return self.url def getRef(self): """Return the actual RemoteReference that we hold, creating it if necessary. This is called when we receive a my-reference sequence from the remote end, so we must increment our received_count.""" # self.ref might be None (if we haven't created it yet), or it might # be a dead weakref (if it has been released but our _handleRefLost # hasn't fired yet). In either case we need to make a new # RemoteReference. if self.ref is None or self.ref() is None: ref = RemoteReference(self) self.ref = weakref.ref(ref, self._refLost) self.received_count += 1 return self.ref() def _refLost(self, wref): # don't do anything right now, we could be in the middle of all sorts # of weird code. both __del__ and weakref callbacks can fire at any # time. Almost as bad as threads.. # instead, do stuff later. eventually(self._handleRefLost) def _handleRefLost(self): if self.ref is None or self.ref() is None: count, self.received_count = self.received_count, 0 if count == 0: return self.broker.freeYourReference(self, count) # otherwise our RemoteReference is actually still alive, resurrected # between the call to _refLost and the eventual call to # _handleRefLost. In this case, don't decref anything. class RemoteReferenceOnly(object): implements(ipb.IRemoteReference) def __init__(self, tracker): """@param tracker: the RemoteReferenceTracker which points to us""" self.tracker = tracker def getSturdyRef(self): return SturdyRef(self.tracker.getURL()) def getRemoteTubID(self): rt = self.tracker.broker.remote_tubref assert rt return rt.getTubID() def getPeer(self): """Return an IAddress-providing object that describes the remote peer. If we've connected to ourselves, this will be a foolscap.broker.LoopbackAddress instance. If we've connected to someone else, this will be a twisted.internet.address.IPv4Address instance, with .host and .port attributes.""" transport = self.tracker.broker.transport return transport.getPeer() def isConnected(self): """Return False if this reference is known to be dead.""" return not self.tracker.broker.disconnected def getLocationHints(self): return SturdyRef(self.tracker.url).locationHints def getConnectionInfo(self): return self.tracker.broker.getConnectionInfo() def getDataLastReceivedAt(self): """If keepalives are enabled, this returns seconds-since-epoch when we last received any data from the remote side. This is connection-wide, not specific to this particular object. If keepalives are disabled (the default), it returns None.""" return self.tracker.broker.getDataLastReceivedAt() def notifyOnDisconnect(self, callback, *args, **kwargs): """Register a callback to run when we lose this connection. The callback will be invoked with whatever extra arguments you provide to this function. For example:: def my_callback(name, number): print name, number+4 cookie = rref.notifyOnDisconnect(my_callback, 'bob', number=3) This function returns an opaque cookie. If you want to cancel the notification, pass this same cookie back to dontNotifyOnDisconnect:: rref.dontNotifyOnDisconnect(cookie) Note that if the Tub is shutdown (via stopService), all notifyOnDisconnect handlers are cancelled. """ # return a cookie (really the (cb,args,kwargs) tuple) that they must # use to deregister marker = self.tracker.broker.notifyOnDisconnect(callback, *args, **kwargs) return marker def dontNotifyOnDisconnect(self, marker): self.tracker.broker.dontNotifyOnDisconnect(marker) def __repr__(self): r = "<%s at 0x%x" % (self.__class__.__name__, abs(id(self))) if self.tracker.url: r += " [%s]" % self.tracker.url r += ">" return r class RemoteReference(RemoteReferenceOnly): def callRemote(self, _name, *args, **kwargs): # Note: for consistency, *all* failures are reported asynchronously. return defer.maybeDeferred(self._callRemote, _name, *args, **kwargs) def callRemoteOnly(self, _name, *args, **kwargs): # the remote end will not send us a response. The only error cases # are arguments that don't match the schema, or broken invariants. In # particular, DeadReferenceError will be silently consumed. d = defer.maybeDeferred(self._callRemote, _name, _callOnly=True, *args, **kwargs) del d return None def _callRemote(self, _name, *args, **kwargs): req = None broker = self.tracker.broker # remember that "none" is not a valid constraint, so we use it to # mean "not set by the caller", which means we fall back to whatever # the RemoteInterface says. Using None would mean an AnyConstraint, # which is not the same thing. methodConstraintOverride = kwargs.get("_methodConstraint", "none") resultConstraint = kwargs.get("_resultConstraint", "none") useSchema = kwargs.get("_useSchema", True) callOnly = kwargs.get("_callOnly", False) if "_methodConstraint" in kwargs: del kwargs["_methodConstraint"] if "_resultConstraint" in kwargs: del kwargs["_resultConstraint"] if "_useSchema" in kwargs: del kwargs["_useSchema"] if "_callOnly" in kwargs: del kwargs["_callOnly"] if callOnly: if broker.disconnected: # DeadReferenceError is silently consumed return reqID = 0 else: # newRequestID() could fail with a DeadReferenceError reqID = broker.newRequestID() # in this section, we validate the outbound arguments against our # notion of what the other end will accept (the RemoteInterface) # first, figure out which method they want to invoke (interfaceName, methodName, methodSchema) = self._getMethodInfo(_name) req = call.PendingRequest(reqID, self, interfaceName, methodName) # TODO: consider adding a stringified stack trace to that # PendingRequest creation, so that DeadReferenceError can emit even # more information about the call which failed # for debugging: these are put into the messages emitted when # logRemoteFailures is turned on req.interfaceName = interfaceName req.methodName = methodName if methodConstraintOverride != "none": methodSchema = methodConstraintOverride if useSchema and methodSchema: # check args against the arg constraint. This could fail if # any arguments are of the wrong type try: methodSchema.checkAllArgs(args, kwargs, False) except Violation, v: v.setLocation("%s.%s(%s)" % (interfaceName, methodName, v.getLocation())) raise # the Interface gets to constraint the return value too, so # make a note of it to use later req.setConstraint(methodSchema.getResponseConstraint()) # if the caller specified a _resultConstraint, that overrides # the schema's one if resultConstraint != "none": # overrides schema req.setConstraint(IConstraint(resultConstraint)) clid = self.tracker.clid slicer = call.CallSlicer(reqID, clid, methodName, args, kwargs) # up to this point, we are not committed to sending anything to the # far end. The various phases of commitment are: # 1: once we tell our broker about the PendingRequest, we must # promise to retire it eventually. Specifically, if we encounter an # error before we give responsibility to the connection, we must # retire it ourselves. # 2: once we start sending the CallSlicer to the other end (in # particular, once they receive the reqID), they might send us a # response, so we must be prepared to handle that. Giving the # PendingRequest to the broker arranges for this to happen. # So all failures which occur before these commitment events are # entirely local: stale broker, bad method name, bad arguments. If # anything raises an exception before this point, the PendingRequest # is abandoned, and our maybeDeferred wrapper returns a failing # Deferred. # commitment point 1. We assume that if this call raises an # exception, the broker will be sure to not track the dead # PendingRequest if not callOnly: broker.addRequest(req) # if callOnly, the PendingRequest will never know about the # broker, and will therefore never ask to be removed from it # TODO: there is a decidability problem here: if the reqID made # it through, the other end will send us an answer (possibly an # error if the remaining slices were aborted). If not, we will # not get an answer. To decide whether we should remove our # broker.waitingForAnswers[] entry, we need to know how far the # slicing process made it. try: # commitment point 2 d = broker.send(slicer) # d will fire when the last argument has been serialized. It will # errback if the arguments (or any of their children) could not # be serialized. We need to catch this case and errback the # caller. # if we got here, we have been able to start serializing the # arguments. If serialization fails, the PendingRequest needs to # be flunked (because we aren't guaranteed that the far end will # do it). d.addErrback(req.fail) except: req.fail(failure.Failure()) # the remote end could send back an error response for many reasons: # bad method name # bad argument types (violated their schema) # exception during method execution # method result violated the results schema # something else could occur to cause an errback: # connection lost before response completely received # exception during deserialization of the response # [but only if it occurs after the reqID is received] # method result violated our results schema # if none of those occurred, the callback will be run return req.deferred def _getMethodInfo(self, name): assert type(name) is str interfaceName = None methodName = name methodSchema = None iface = self.tracker.interface if iface: interfaceName = iface.__remote_name__ try: methodSchema = iface[name] except KeyError: raise Violation("%s(%s) does not offer %s" % \ (interfaceName, self, name)) return interfaceName, methodName, methodSchema class RemoteMethodReferenceTracker(RemoteReferenceTracker): def getRef(self): if self.ref is None: ref = RemoteMethodReference(self) self.ref = weakref.ref(ref, self._refLost) self.received_count += 1 return self.ref() class RemoteMethodReference(RemoteReference): def callRemote(self, *args, **kwargs): # TODO: I suspect it would safer to use something other than # 'callRemote' here. # TODO: this probably needs a very different implementation # there is no schema support yet, so we can't convert positional args # into keyword args assert args == () return RemoteReference.callRemote(self, "", *args, **kwargs) def _getMethodInfo(self, name): interfaceName = None methodName = "" methodSchema = None return interfaceName, methodName, methodSchema class LocalReferenceable(object): implements(ipb.IRemoteReference) def __init__(self, original): self.original = original def notifyOnDisconnect(self, callback, *args, **kwargs): # local objects never disconnect return None def dontNotifyOnDisconnect(self, marker): pass def callRemote(self, methname, *args, **kwargs): def _try(ignored): meth = getattr(self.original, "remote_" + methname) return meth(*args, **kwargs) d = fireEventually() d.addCallback(_try) return d def callRemoteOnly(self, methname, *args, **kwargs): d = self.callRemote(methname, *args, **kwargs) d.addErrback(lambda f: None) return None registerAdapter(LocalReferenceable, ipb.IReferenceable, ipb.IRemoteReference) class YourReferenceSlicer(slicer.BaseSlicer): """I handle pb.RemoteReference objects (being sent back home to the original pb.Referenceable-holder) """ def slice(self, streamable, protocol): broker = self.requireBroker(protocol) self.streamable = streamable tracker = self.obj.tracker if tracker.broker == broker: # sending back to home broker yield 'your-reference' yield tracker.clid else: # sending somewhere else furl = tracker.getURL() if furl is None: log.msg("gift has no FURL, host Tub is unreachable, sending ''") furl = "" assert isinstance(furl, str) giftID = broker.makeGift(self.obj) yield 'their-reference' yield giftID yield furl def describe(self): return "" % self.obj.tracker.clid registerAdapter(YourReferenceSlicer, RemoteReference, ipb.ISlicer) class YourReferenceUnslicer(slicer.LeafUnslicer): """I accept incoming (integer) your-reference sequences and try to turn them back into the original Referenceable. I also accept (string) your-reference sequences and try to turn them into a published Referenceable that they did not have access to before.""" clid = None def checkToken(self, typebyte, size): if typebyte != tokens.INT: raise BananaError("your-reference ID must be an INT") def receiveChild(self, obj, ready_deferred=None): assert not isinstance(obj, defer.Deferred) assert ready_deferred is None self.clid = obj def receiveClose(self): if self.clid is None: raise BananaError("sequence ended too early") obj = self.broker.getMyReferenceByCLID(self.clid) if not obj: raise Violation("unknown clid '%s'" % self.clid) return obj, None def describe(self): return "" % self.obj.refID class TheirReferenceUnslicer(slicer.LeafUnslicer): """I accept gifts of third-party references. This is turned into a live reference upon receipt.""" # (their-reference, giftID, URL) state = 0 giftID = None url = None urlConstraint = ByteStringConstraint() def checkToken(self, typebyte, size): if self.state == 0: if typebyte != tokens.INT: raise BananaError("their-reference giftID must be an INT") elif self.state == 1: self.urlConstraint.checkToken(typebyte, size) else: raise Violation("too many parameters in their-reference") def receiveChild(self, obj, ready_deferred=None): assert not isinstance(obj, defer.Deferred) assert ready_deferred is None if self.state == 0: self.giftID = obj self.state = 1 elif self.state == 1: # URL self.url = obj self.state = 2 else: raise BananaError("Too many their-reference parameters") def receiveClose(self): if self.giftID is None or self.url is None: raise BananaError("sequence ended too early") if self.broker.tub.accept_gifts: d = self.broker.tub.getReference(self.url) d.addBoth(self.ackGift) else: d = defer.fail(Violation("gifts are prohibited in this Tub")) # we return a Deferred that will fire with the RemoteReference when # it becomes available. The RemoteReference is not even referenceable # until then. In addition, we provide a ready_deferred, since any # mutable container which holds the gift will be referenceable early # but the message delivery must still wait for the getReference to # complete. See to it that we fire the object deferred before we fire # the ready_deferred. obj_deferred = defer.Deferred() ready_deferred = defer.Deferred() def _ready(rref): obj_deferred.callback(rref) ready_deferred.callback(rref) def _failed(f): # if an error in getReference() occurs, log it locally (with # priority UNUSUAL), because this end might need to diagnose some # connection or networking problems. log.msg("gift (%s) failed to resolve: %s" % (self.url, f)) # deliver a placeholder object to the container, but signal the # ready_deferred that we've failed. This will bubble up to the # enclosing InboundDelivery, and when it gets to the top of the # queue, it will be flunked. obj_deferred.callback("Place holder for a Gift which failed to " "resolve: %s" % f) ready_deferred.errback(f) d.addCallbacks(_ready, _failed) return obj_deferred, ready_deferred def ackGift(self, rref): # giftID==0 means they aren't doing reference counting if self.giftID != 0: rb = self.broker.remote_broker # if we lose the connection, they'll decref the gift anyway rb.callRemoteOnly("decgift", giftID=self.giftID, count=1) return rref def describe(self): if self.giftID is None: return "" return "" % self.giftID class SturdyRef(Copyable, RemoteCopy): """I am a pointer to a Referenceable that lives in some (probably remote) Tub. This pointer is long-lived, however you cannot send messages with it directly. To use it, you must ask your Tub to turn it into a RemoteReference with tub.getReference(sturdyref). The SturdyRef is associated with a URL: you can create a SturdyRef out of a URL that you obtain from some other source, and you can ask the SturdyRef for its URL. SturdyRefs are serialized by copying their URL, and create an identical SturdyRef on the receiving side.""" typeToCopy = copytype = "foolscap.SturdyRef" tubID = None name = None def __init__(self, url=None): self.locationHints = [] # list of strings self.url = url if url: self.tubID, self.locationHints, self.name = decode_furl(url) def getTubRef(self): return TubRef(self.tubID, self.locationHints) def getURL(self): return self.url def __str__(self): return str(self.url) def _distinguishers(self): """Two SturdyRefs are equivalent if they point to the same object. SturdyRefs pay attention only to the TubID and the reference name. This method makes it easier to compare a pair of SturdyRefs.""" return (True, self.tubID, self.name) def __hash__(self): return hash(self._distinguishers()) def __cmp__(self, them): return (cmp(type(self), type(them)) or cmp(self.__class__, them.__class__) or cmp(self._distinguishers(), them._distinguishers())) class TubRef(object): """This is a little helper class which provides a comparable identifier for Tubs. TubRefs can be used as keys in dictionaries that track connections to remote Tubs.""" def __init__(self, tubID, locationHints=None): if locationHints is None: locationHints = [] assert isinstance(locationHints, list), locationHints assert all([isinstance(hint, str) for hint in locationHints]), \ locationHints self.tubID = tubID self.locationHints = locationHints def getLocations(self): return self.locationHints def getTubID(self): return self.tubID def getShortTubID(self): return self.tubID[:4] def __str__(self): return "pb://" + self.tubID def _distinguishers(self): """This serves the same purpose as SturdyRef._distinguishers.""" return (self.tubID,) def __hash__(self): return hash(self._distinguishers()) def __cmp__(self, them): return (cmp(type(self), type(them)) or cmp(self.__class__, them.__class__) or cmp(self._distinguishers(), them._distinguishers())) foolscap-0.13.1/src/foolscap/remoteinterface.py0000644000076500000240000004115212766553111022160 0ustar warnerstaff00000000000000 import types import inspect from zope.interface import interface, providedBy, implements from foolscap.constraint import Constraint, OpenerConstraint, nothingTaster, \ IConstraint, IRemoteMethodConstraint, Optional, Any from foolscap.tokens import Violation, InvalidRemoteInterface from foolscap.schema import addToConstraintTypeMap from foolscap import ipb class RemoteInterfaceClass(interface.InterfaceClass): """This metaclass lets RemoteInterfaces be a lot like Interfaces. The methods are parsed differently (PB needs more information from them than z.i extracts, and the methods can be specified with a RemoteMethodSchema directly). RemoteInterfaces can accept the following additional attribute:: __remote_name__: can be set to a string to specify the globally-unique name for this interface. This should be a URL in a namespace you administer. If not set, defaults to the short classname. RIFoo.names() returns the list of remote method names. RIFoo['bar'] is still used to get information about method 'bar', however it returns a RemoteMethodSchema instead of a z.i Method instance. """ def __init__(self, iname, bases=(), attrs=None, __module__=None): if attrs is None: interface.InterfaceClass.__init__(self, iname, bases, attrs, __module__) return # parse (and remove) the attributes that make this a RemoteInterface try: rname, remote_attrs = self._parseRemoteInterface(iname, attrs) except: raise # now let the normal InterfaceClass do its thing interface.InterfaceClass.__init__(self, iname, bases, attrs, __module__) # now add all the remote methods that InterfaceClass would have # complained about. This is really gross, and it really makes me # question why we're bothing to inherit from z.i.Interface at all. I # will probably stop doing that soon, and just have our own # meta-class, but I want to make sure you can still do # 'implements(RIFoo)' from within a class definition. a = getattr(self, "_InterfaceClass__attrs") # the ickiest part a.update(remote_attrs) self.__remote_name__ = rname # finally, auto-register the interface try: registerRemoteInterface(self, rname) except: raise def _parseRemoteInterface(self, iname, attrs): remote_attrs = {} remote_name = attrs.get("__remote_name__", iname) # and see if there is a __remote_name__ . We delete it because # InterfaceClass doesn't like arbitrary attributes if attrs.has_key("__remote_name__"): del attrs["__remote_name__"] # determine all remotely-callable methods names = [name for name in attrs.keys() if ((type(attrs[name]) == types.FunctionType and not name.startswith("_")) or IConstraint.providedBy(attrs[name]))] # turn them into constraints. Tag each of them with their name and # the RemoteInterface they came from. for name in names: m = attrs[name] if not IConstraint.providedBy(m): m = RemoteMethodSchema(method=m) m.name = name m.interface = self remote_attrs[name] = m # delete the methods, so zope's InterfaceClass doesn't see them. # Particularly necessary for things defined with IConstraints. del attrs[name] return remote_name, remote_attrs RemoteInterface = RemoteInterfaceClass("RemoteInterface", __module__="pb.flavors") def getRemoteInterface(obj): """Get the (one) RemoteInterface supported by the object, or None.""" interfaces = list(providedBy(obj)) # TODO: versioned Interfaces! ilist = [] for i in interfaces: if isinstance(i, RemoteInterfaceClass): if i not in ilist: ilist.append(i) assert len(ilist) <= 1, ("don't use multiple RemoteInterfaces! %s uses %s" % (obj, ilist)) if ilist: return ilist[0] return None class DuplicateRemoteInterfaceError(Exception): pass RemoteInterfaceRegistry = {} def registerRemoteInterface(iface, name=None): if not name: name = iface.__remote_name__ assert isinstance(iface, RemoteInterfaceClass) if RemoteInterfaceRegistry.has_key(name): old = RemoteInterfaceRegistry[name] msg = "remote interface %s was registered with the same name (%s) as %s, please use __remote_name__ to provide a unique name" % (old, name, iface) raise DuplicateRemoteInterfaceError(msg) RemoteInterfaceRegistry[name] = iface def getRemoteInterfaceByName(iname): return RemoteInterfaceRegistry.get(iname) class RemoteMethodSchema(object): """ This is a constraint for a single remotely-invokable method. It gets to require, deny, or impose further constraints upon a set of named arguments. This constraint is created by using keyword arguments with the same names as the target method's arguments. Two special names are used: __ignoreUnknown__: if True, unexpected argument names are silently dropped. (note that this makes the schema unbounded) __acceptUnknown__: if True, unexpected argument names are always accepted without a constraint (which also makes this schema unbounded) The remotely-accesible object's .getMethodSchema() method may return one of these objects. """ implements(IRemoteMethodConstraint) taster = {} # this should not be used as a top-level constraint opentypes = [] # overkill ignoreUnknown = False acceptUnknown = False name = None # method name, set when the RemoteInterface is parsed interface = None # points to the RemoteInterface which defines the method # under development def __init__(self, method=None, _response=None, __options=[], **kwargs): if method: self.initFromMethod(method) return self.argumentNames = [] self.argConstraints = {} self.required = [] self.responseConstraint = None # __response in the argslist gets treated specially, I think it is # mangled into _RemoteMethodSchema__response or something. When I # change it to use _response instead, it works. if _response: self.responseConstraint = IConstraint(_response) self.options = {} # return, wait, reliable, etc if kwargs.has_key("__ignoreUnknown__"): self.ignoreUnknown = kwargs["__ignoreUnknown__"] del kwargs["__ignoreUnknown__"] if kwargs.has_key("__acceptUnknown__"): self.acceptUnknown = kwargs["__acceptUnknown__"] del kwargs["__acceptUnknown__"] for argname, constraint in kwargs.items(): self.argumentNames.append(argname) constraint = IConstraint(constraint) self.argConstraints[argname] = constraint if not isinstance(constraint, Optional): self.required.append(argname) def initFromMethod(self, method): # call this with the Interface's prototype method: the one that has # argument constraints expressed as default arguments, and which # does nothing but returns the appropriate return type names, _, _, typeList = inspect.getargspec(method) if names and names[0] == 'self': why = "RemoteInterface methods should not have 'self' in their argument list" raise InvalidRemoteInterface(why) if not names: typeList = [] # 'def foo(oops)' results in typeList==None if typeList is None or len(names) != len(typeList): # TODO: relax this, use schema=Any for the args that don't have # default values. This would make: # def foo(a, b=int): return None # equivalent to: # def foo(a=Any, b=int): return None why = "RemoteInterface methods must have default values for all their arguments" raise InvalidRemoteInterface(why) self.argumentNames = names self.argConstraints = {} self.required = [] for i in range(len(names)): argname = names[i] constraint = typeList[i] if not isinstance(constraint, Optional): self.required.append(argname) self.argConstraints[argname] = IConstraint(constraint) # call the method, its 'return' value is the return constraint self.responseConstraint = IConstraint(method()) self.options = {} # return, wait, reliable, etc def getPositionalArgConstraint(self, argnum): if argnum >= len(self.argumentNames): raise Violation("too many positional arguments: %d >= %d" % (argnum, len(self.argumentNames))) argname = self.argumentNames[argnum] c = self.argConstraints.get(argname) assert c if isinstance(c, Optional): c = c.constraint return (True, c) def getKeywordArgConstraint(self, argname, num_posargs=0, previous_kwargs=[]): previous_args = self.argumentNames[:num_posargs] for pkw in previous_kwargs: assert pkw not in previous_args previous_args.append(pkw) if argname in previous_args: raise Violation("got multiple values for keyword argument '%s'" % (argname,)) c = self.argConstraints.get(argname) if c: if isinstance(c, Optional): c = c.constraint return (True, c) # what do we do with unknown arguments? if self.ignoreUnknown: return (False, None) if self.acceptUnknown: return (True, None) raise Violation("unknown argument '%s'" % argname) def getResponseConstraint(self): return self.responseConstraint def checkAllArgs(self, args, kwargs, inbound): # first we map the positional arguments allargs = {} if len(args) > len(self.argumentNames): raise Violation("method takes %d positional arguments (%d given)" % (len(self.argumentNames), len(args))) for i,argvalue in enumerate(args): allargs[self.argumentNames[i]] = argvalue for argname,argvalue in kwargs.items(): if argname in allargs: raise Violation("got multiple values for keyword argument '%s'" % (argname,)) allargs[argname] = argvalue for argname, argvalue in allargs.items(): accept, constraint = self.getKeywordArgConstraint(argname) if not accept: # this argument will be ignored by the far end. TODO: emit a # warning pass try: constraint.checkObject(argvalue, inbound) except Violation, v: v.setLocation("%s=" % argname) raise for argname in self.required: if argname not in allargs: raise Violation("missing required argument '%s'" % argname) def checkResults(self, results, inbound): if self.responseConstraint: # this might raise a Violation. The caller will annotate its # location appropriately: they have more information than we do. self.responseConstraint.checkObject(results, inbound) class UnconstrainedMethod(object): """I am a method constraint that accepts any arguments and any return value. To use this, assign it to a method name in a RemoteInterface:: class RIFoo(RemoteInterface): def constrained_method(foo=int, bar=str): # this one is constrained return str not_method = UnconstrainedMethod() # this one is not """ implements(IRemoteMethodConstraint) def getPositionalArgConstraint(self, argnum): return (True, Any()) def getKeywordArgConstraint(self, argname, num_posargs=0, previous_kwargs=[]): return (True, Any()) def checkAllArgs(self, args, kwargs, inbound): pass # accept everything def getResponseConstraint(self): return Any() def checkResults(self, results, inbound): pass # accept everything class LocalInterfaceConstraint(Constraint): """This constraint accepts any (local) instance which implements the given local Interface. """ # TODO: maybe accept RemoteCopy instances # TODO: accept inbound your-references, if the local object they map to # implements the interface # TODO: do we need an string-to-Interface map just like we have a # classname-to-class/factory map? taster = nothingTaster opentypes = [] name = "LocalInterfaceConstraint" def __init__(self, interface): self.interface = interface def checkObject(self, obj, inbound): # TODO: maybe try to get an adapter instead? if not self.interface.providedBy(obj): raise Violation("'%s' does not provide interface %s" % (obj, self.interface)) class RemoteInterfaceConstraint(OpenerConstraint): """This constraint accepts any RemoteReference that claims to be associated with a remote Referenceable that implements the given RemoteInterface. If 'interface' is None, just assert that it is a RemoteReference at all. On the inbound side, this will only accept a suitably-implementing RemoteReference, or a gift that resolves to such a RemoteReference. On the outbound side, this will accept either a Referenceable or a RemoteReference (which might be a your-reference or a their-reference). Sending your-references will result in the recipient getting a local Referenceable, which will not pass the constraint. TODO: think about if we want this behavior or not. """ opentypes = [("my-reference",), ("their-reference",)] name = "RemoteInterfaceConstraint" def __init__(self, interface): self.interface = interface def checkObject(self, obj, inbound): if inbound: # this ought to be a RemoteReference that claims to be associated # with a remote Referenceable that implements the desired # interface. if not ipb.IRemoteReference.providedBy(obj): raise Violation("'%s' does not provide RemoteInterface %s, " "and doesn't even look like a RemoteReference" % (obj, self.interface)) if not self.interface: return iface = obj.tracker.interface # TODO: this test probably doesn't handle subclasses of # RemoteInterface, which might be useful (if it even works) if not iface or iface != self.interface: raise Violation("'%s' does not provide RemoteInterface %s" % (obj, self.interface)) else: # this ought to be a Referenceable which implements the desired # interface. Or, it might be a RemoteReference which points to # one. if ipb.IRemoteReference.providedBy(obj): # it's a RemoteReference if not self.interface: return iface = obj.tracker.interface if not iface or iface != self.interface: raise Violation("'%s' does not provide RemoteInterface %s" % (obj, self.interface)) return if not ipb.IReferenceable.providedBy(obj): # TODO: maybe distinguish between OnlyReferenceable and # Referenceable? which is more useful here? raise Violation("'%s' is not a Referenceable" % (obj,)) if self.interface and not self.interface.providedBy(obj): raise Violation("'%s' does not provide RemoteInterface %s" % (obj, self.interface)) def _makeConstraint(t): # This will be called for both local interfaces (IFoo) and remote # interfaces (RIFoo), so we have to distinguish between them. The late # import is to deal with a circular reference between this module and # remoteinterface.py if isinstance(t, RemoteInterfaceClass): return RemoteInterfaceConstraint(t) return LocalInterfaceConstraint(t) addToConstraintTypeMap(interface.InterfaceClass, _makeConstraint) foolscap-0.13.1/src/foolscap/schema.py0000644000076500000240000001671412766553111020252 0ustar warnerstaff00000000000000 # This module contains all user-visible Constraint subclasses, for # convenience by user code which is defining RemoteInterfaces. The primitive # ones are defined in constraint.py, while the constraints associated with # specific open sequences (list, unicode, etc) are defined in the related # slicer/list.py module, etc. A few are defined here. # It also defines the constraintMap and constraintTypeMap, used when # constructing constraints out of the convenience shorthand. This is used # when processing the methods defined in a RemoteInterface (such that a # default argument like x=int gets turned into an IntegerConstraint). New # slicers that want to add to these mappings can use addToConstraintTypeMap # or manipulate constraintMap directly. # this imports slicers and constraints.py, but is not allowed to import any # other Foolscap modules, to avoid import cycles. """ primitive constraints: - types.StringType: string with maxLength=1k - String(maxLength=1000): string with arbitrary maxLength - types.BooleanType: boolean - types.IntType: integer that fits in s_int32_t - types.LongType: integer with abs(num) < 2**8192 (fits in 1024 bytes) - Int(maxBytes=1024): integer with arbitrary maxValue=2**(8*maxBytes) - types.FloatType: number - Number(maxBytes=1024): float or integer with maxBytes - interface: instance which implements (or adapts to) the Interface - class: instance of the class or a subclass - # unicode? types? none? container constraints: - TupleOf(constraint1, constraint2..): fixed size, per-element constraint - ListOf(constraint, maxLength=30): all elements obey constraint - DictOf(keyconstraint, valueconstraint): keys and values obey constraints - AttributeDict(*attrTuples, ignoreUnknown=False): - attrTuples are (name, constraint) - ignoreUnknown=True means that received attribute names which aren't listed in attrTuples should be ignored instead of raising an UnknownAttrName exception composite constraints: - tuple: alternatives: must obey one of the different constraints modifiers: - Shared(constraint, refLimit=None): object may be referenced multiple times within the serialization domain (question: which domain?). All constraints default to refLimit=1, and a MultiplyReferenced exception is raised as soon as the reference count goes above the limit. refLimit=None means no limit is enforced. - Optional(name, constraint, default=None): key is not required. If not provided and default is None, key/attribute will not be created Only valid inside DictOf and AttributeDict. """ from foolscap.tokens import Violation, UnknownSchemaType, BananaError, \ tokenNames # make constraints available in a single location from foolscap.constraint import Constraint, Any, ByteStringConstraint, \ IntegerConstraint, NumberConstraint, IConstraint, Optional, Shared from foolscap.slicers.unicode import UnicodeConstraint from foolscap.slicers.bool import BooleanConstraint from foolscap.slicers.dict import DictConstraint from foolscap.slicers.list import ListConstraint from foolscap.slicers.set import SetConstraint from foolscap.slicers.tuple import TupleConstraint from foolscap.slicers.none import Nothing # we don't import RemoteMethodSchema from remoteinterface.py, because # remoteinterface.py needs to import us (for addToConstraintTypeMap) ignored = [Constraint, Any, ByteStringConstraint, UnicodeConstraint, IntegerConstraint, NumberConstraint, BooleanConstraint, DictConstraint, ListConstraint, SetConstraint, TupleConstraint, Nothing, Optional, Shared, ] # hush pyflakes # convenience shortcuts TupleOf = TupleConstraint ListOf = ListConstraint DictOf = DictConstraint SetOf = SetConstraint # note: using PolyConstraint (aka ChoiceOf) for inbound tasting is probably # not fully vetted. One of the issues would be with something like # ListOf(ChoiceOf(TupleOf(stuff), SetOf(stuff))). The ListUnslicer, when # handling an inbound Tuple, will do # TupleUnslicer.setConstraint(polyconstraint), since that's all it really # knows about, and the TupleUnslicer will then try to look inside the # polyconstraint for attributes that talk about tuples, and might fail. class PolyConstraint(Constraint): name = "PolyConstraint" def __init__(self, *alternatives): self.alternatives = [IConstraint(a) for a in alternatives] self.alternatives = tuple(self.alternatives) # TODO: taster/opentypes should be a union of the alternatives' def checkToken(self, typebyte, size): ok = False for c in self.alternatives: try: c.checkToken(typebyte, size) ok = True except (Violation, BananaError): pass if not ok: raise Violation("typebyte %s does not satisfy any of %s" % (tokenNames[typebyte], self.alternatives)) def checkObject(self, obj, inbound): ok = False for c in self.alternatives: try: c.checkObject(obj, inbound) ok = True except Violation: pass if not ok: raise Violation("object type %s does not satisfy any of %s" % (type(obj), self.alternatives)) ChoiceOf = PolyConstraint def AnyStringConstraint(*args, **kwargs): return ChoiceOf(ByteStringConstraint(*args, **kwargs), UnicodeConstraint(*args, **kwargs)) # keep the old meaning, for now. Eventually StringConstraint should become an # AnyStringConstraint StringConstraint = ByteStringConstraint constraintMap = { str: ByteStringConstraint(), unicode: UnicodeConstraint(), bool: BooleanConstraint(), int: IntegerConstraint(), long: IntegerConstraint(maxBytes=1024), float: NumberConstraint(), None: Nothing(), } # This module provides a function named addToConstraintTypeMap() which helps # to resolve some import cycles. constraintTypeMap = [] def addToConstraintTypeMap(typ, constraintMaker): constraintTypeMap.insert(0, (typ, constraintMaker)) def _tupleConstraintMaker(t): return TupleConstraint(*t) addToConstraintTypeMap(tuple, _tupleConstraintMaker) # this function transforms the simple syntax (as used in RemoteInterface # method definitions) into Constraint instances. This function is registered # as a zope.interface adapter hook, so that once we've been loaded, other # code can just do IConstraint(stuff) and expect it to work. def adapt_obj_to_iconstraint(iface, t): if iface is not IConstraint: return None assert not IConstraint.providedBy(t) # not sure about this c = constraintMap.get(t, None) if c: return c for (typ, constraintMaker) in constraintTypeMap: if isinstance(t, typ): c = constraintMaker(t) if c: return c # RIFoo means accept either a Referenceable that implements RIFoo, or a # RemoteReference that points to just such a Referenceable. This is # hooked in by remoteinterface.py, when it calls addToConstraintTypeMap # we are the only way to make constraints raise UnknownSchemaType("can't make constraint from '%s' (%s)" % (t, type(t))) from zope.interface.interface import adapter_hooks adapter_hooks.append(adapt_obj_to_iconstraint) # how to accept "([(ref0" ? # X = "TupleOf(ListOf(TupleOf(" * infinity # ok, so you can't write a constraint that accepts it. I'm ok with that. foolscap-0.13.1/src/foolscap/slicer.py0000644000076500000240000002763312766553111020275 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test.test_banana -*- from twisted.python.components import registerAdapter from twisted.python import log from zope.interface import implements from twisted.internet.defer import Deferred import tokens from tokens import Violation, BananaError from foolscap.ipb import IBroker class SlicerClass(type): # auto-register Slicers def __init__(self, name, bases, dict): type.__init__(self, name, bases, dict) typ = dict.get('slices') #reg = dict.get('slicerRegistry') if typ: registerAdapter(self, typ, tokens.ISlicer) class BaseSlicer(object): __metaclass__ = SlicerClass implements(tokens.ISlicer) slices = None parent = None sendOpen = True opentype = () trackReferences = False def __init__(self, obj): # this simplifies Slicers which are adapters self.obj = obj def requireBroker(self, protocol): broker = IBroker(protocol, None) if not broker: msg = "This object can only be serialized by a broker" raise Violation(msg) return broker def registerRefID(self, refid, obj): # optimize: most Slicers will delegate this up to the Root return self.parent.registerRefID(refid, obj) def slicerForObject(self, obj): # optimize: most Slicers will delegate this up to the Root return self.parent.slicerForObject(obj) def slice(self, streamable, banana): # this is what makes us ISlicer self.streamable = streamable assert self.opentype for o in self.opentype: yield o for t in self.sliceBody(streamable, banana): yield t def sliceBody(self, streamable, banana): raise NotImplementedError def childAborted(self, f): return f def describe(self): return "??" class ScopedSlicer(BaseSlicer): """This Slicer provides a containing scope for referenceable things like lists. The same list will not be serialized twice within this scope, but it will not survive outside it.""" def __init__(self, obj): BaseSlicer.__init__(self, obj) self.references = {} # maps id(obj) -> (obj,refid) def registerRefID(self, refid, obj): # keep references here, not in the actual PBRootSlicer # This use of id(obj) requires a bit of explanation. We are making # the assumption that the object graph remains unmodified until # serialization is complete. In particular, we assume that all the # objects in it remain alive, and no new objects are added to it, # until serialization is complete. id(obj) is only unique for live # objects: once the object is garbage-collected, a new object may be # created with the same id(obj) value. # # The concern is that a custom Slicer will call something that # mutates the object graph before it has finished being serialized. # This might be one which calls some user-level function during # Slicing, or one which uses a Deferred to put off serialization for # a while, creating an opportunity for some other code to get # control. # The specific concern is that if, in the middle of serialization, an # object that was already serialized is gc'ed, and a new object is # created and attached to a portion of the object graph that hasn't # been serialized yet, and if the new object gets the same id(obj) as # the dead object, then we could be tricked into sending the # reference number of the old (dead) object. On the receiving end, # this would result in a mangled object graph. # User code isn't supposed to allow the object graph to change during # serialization, so this mangling "should not happen" under normal # circumstances. However, as a reasonably cheap way to mitigate the # worst sort of mangling when user code *does* mess up, # self.references maps from id(obj) to a tuple of (obj,refid) instead # of just the refid. This insures that the object will stay alive # until the ScopedSlicer dies, guaranteeing that we won't get # duplicate id(obj) values. If user code mutates the object graph # during serialization we might still get inconsistent results, but # they'll be the ordinary kind of inconsistent results (snapshots of # different branches of the object graph at different points in time) # rather than the blatantly wrong mangling that would occur with # re-used id(obj) values. self.references[id(obj)] = (obj,refid) def slicerForObject(self, obj): # check for an object which was sent previously or has at least # started sending obj_refid = self.references.get(id(obj), None) if obj_refid is not None: # we've started to send this object already, so just include a # reference to it return ReferenceSlicer(obj_refid[1]) # otherwise go upstream so we can serialize the object completely return self.parent.slicerForObject(obj) UnslicerRegistry = {} BananaUnslicerRegistry = {} def registerUnslicer(opentype, factory, registry=None): if registry is None: registry = UnslicerRegistry assert not registry.has_key(opentype) registry[opentype] = factory class UnslicerClass(type): # auto-register Unslicers def __init__(self, name, bases, dict): type.__init__(self, name, bases, dict) opentype = dict.get('opentype') reg = dict.get('unslicerRegistry') if opentype: registerUnslicer(opentype, self, reg) class BaseUnslicer(object): __metaclass__ = UnslicerClass opentype = None implements(tokens.IUnslicer) def __init__(self): pass def describe(self): return "??" def setConstraint(self, constraint): pass def start(self, count): pass def checkToken(self, typebyte, size): return # no restrictions def openerCheckToken(self, typebyte, size, opentype): return self.parent.openerCheckToken(typebyte, size, opentype) def open(self, opentype): """Return an IUnslicer object based upon the 'opentype' tuple. Subclasses that wish to change the way opentypes are mapped to Unslicers can do so by changing this behavior. This method does not apply constraints, it only serves to map opentype into Unslicer. Most subclasses will implement this by delegating the request to their parent (and thus, eventually, to the RootUnslicer), and will set the new child's .opener attribute so that they can do the same. Subclasses that wish to change the way opentypes are mapped to Unslicers can do so by changing this behavior.""" return self.parent.open(opentype) def doOpen(self, opentype): """Return an IUnslicer object based upon the 'opentype' tuple. This object will receive all tokens destined for the subnode. If you want to enforce a constraint, you must override this method and do two things: make sure your constraint accepts the opentype, and set a per-item constraint on the new child unslicer. This method gets the IUnslicer from our .open() method. That might return None instead of a child unslicer if the they want a multi-token opentype tuple, so be sure to check for Noneness before adding a per-item constraint. """ return self.open(opentype) def receiveChild(self, obj, ready_deferred=None): """Unslicers for containers should accumulate their children's ready_deferreds, then combine them in an AsyncAND when receiveClose() happens, and return the AsyncAND as the ready_deferreds half of the receiveClose() return value. """ pass def reportViolation(self, why): return why def receiveClose(self): raise NotImplementedError def finish(self): pass def setObject(self, counter, obj): """To pass references to previously-sent objects, the [OPEN, 'reference', number, CLOSE] sequence is used. The numbers are generated implicitly by the sending Banana, counting from 0 for the object described by the very first OPEN sent over the wire, incrementing for each subsequent one. The objects themselves are stored in any/all Unslicers who cares to. Generally this is the RootUnslicer, but child slices could do it too if they wished. """ # TODO: examine how abandoned child objects could mess up this # counter pass def getObject(self, counter): """'None' means 'ask our parent instead'. """ return None def explode(self, failure): """If something goes wrong in a Deferred callback, it may be too late to reject the token and to normal error handling. I haven't figured out how to do sensible error-handling in this situation. This method exists to make sure that the exception shows up *somewhere*. If this is called, it is also likely that a placeholder (probably a Deferred) will be left in the unserialized object graph about to be handed to the RootUnslicer. """ # RootUnslicer pays attention to this .exploded attribute and refuses # to deliver anything if it is set. But PBRootUnslicer ignores it. # TODO: clean this up, and write some unit tests to trigger it (by # violating schemas?) log.msg("BaseUnslicer.explode: %s" % failure) self.protocol.exploded = failure class ScopedUnslicer(BaseUnslicer): """This Unslicer provides a containing scope for referenceable things like lists. It corresponds to the ScopedSlicer base class.""" def __init__(self): BaseUnslicer.__init__(self) self.references = {} def setObject(self, counter, obj): if self.protocol.debugReceive: print "setObject(%s): %s{%s}" % (counter, obj, id(obj)) self.references[counter] = obj def getObject(self, counter): obj = self.references.get(counter) if self.protocol.debugReceive: print "getObject(%s) -> %s{%s}" % (counter, obj, id(obj)) return obj class LeafUnslicer(BaseUnslicer): # inherit from this to reject any child nodes # .checkToken in LeafUnslicer subclasses should reject OPEN tokens def doOpen(self, opentype): raise Violation("'%s' does not accept sub-objects" % self) # References are special enough to put here instead of slicers/ class ReferenceSlicer(BaseSlicer): # this is created explicitly, not as an adapter opentype = ('reference',) trackReferences = False def __init__(self, refid): assert type(refid) is int self.refid = refid def sliceBody(self, streamable, banana): yield self.refid class ReferenceUnslicer(LeafUnslicer): opentype = ('reference',) constraint = None finished = False def setConstraint(self, constraint): self.constraint = constraint def checkToken(self, typebyte,size): if typebyte != tokens.INT: raise BananaError("ReferenceUnslicer only accepts INTs") def receiveChild(self, obj, ready_deferred=None): assert not isinstance(obj, Deferred) assert ready_deferred is None if self.finished: raise BananaError("ReferenceUnslicer only accepts one int") self.obj = self.protocol.getObject(obj) self.finished = True # assert that this conforms to the constraint if self.constraint: self.constraint.checkObject(self.obj, True) # TODO: it might be a Deferred, but we should know enough about the # incoming value to check the constraint. This requires a subclass # of Deferred which can give us the metadata. def receiveClose(self): return self.obj, None foolscap-0.13.1/src/foolscap/slicers/0000755000076500000240000000000013204747603020072 5ustar warnerstaff00000000000000foolscap-0.13.1/src/foolscap/slicers/__init__.py0000644000076500000240000000000012766553111022172 0ustar warnerstaff00000000000000foolscap-0.13.1/src/foolscap/slicers/allslicers.py0000644000076500000240000000320212766553111022577 0ustar warnerstaff00000000000000 ######################## Slicers+Unslicers # note that Slicing is always easier than Unslicing, because Unslicing # is the side where you are dealing with the danger from foolscap.slicers.none import NoneSlicer, NoneUnslicer from foolscap.slicers.bool import BooleanSlicer, BooleanUnslicer from foolscap.slicers.unicode import UnicodeSlicer, UnicodeUnslicer from foolscap.slicers.decimal_slicer import DecimalSlicer, DecimalUnslicer from foolscap.slicers.list import ListSlicer, ListUnslicer from foolscap.slicers.tuple import TupleSlicer, TupleUnslicer from foolscap.slicers.set import SetSlicer, SetUnslicer from foolscap.slicers.set import FrozenSetSlicer, FrozenSetUnslicer #from foolscap.slicers.set import BuiltinSetSlicer from foolscap.slicers.dict import DictSlicer, DictUnslicer, OrderedDictSlicer from foolscap.slicers.vocab import ReplaceVocabSlicer, ReplaceVocabUnslicer from foolscap.slicers.vocab import ReplaceVocabularyTable, AddToVocabularyTable from foolscap.slicers.vocab import AddVocabSlicer, AddVocabUnslicer from foolscap.slicers.root import RootSlicer, RootUnslicer # appease pyflakes unused = [ NoneSlicer, NoneUnslicer, BooleanSlicer, BooleanUnslicer, UnicodeSlicer, UnicodeUnslicer, DecimalSlicer, DecimalUnslicer, ListSlicer, ListUnslicer, TupleSlicer, TupleUnslicer, SetSlicer, SetUnslicer, FrozenSetSlicer, FrozenSetUnslicer, #from foolscap.slicers.set import BuiltinSetSlicer DictSlicer, DictUnslicer, OrderedDictSlicer, ReplaceVocabSlicer, ReplaceVocabUnslicer, ReplaceVocabularyTable, AddToVocabularyTable, AddVocabSlicer, AddVocabUnslicer, RootSlicer, RootUnslicer, ] foolscap-0.13.1/src/foolscap/slicers/bool.py0000644000076500000240000000455012766553111021404 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test.test_banana -*- from twisted.python.components import registerAdapter from twisted.internet.defer import Deferred from foolscap import tokens from foolscap.tokens import Violation, BananaError from foolscap.slicer import BaseSlicer, LeafUnslicer from foolscap.constraint import OpenerConstraint, IntegerConstraint, Any class BooleanSlicer(BaseSlicer): opentype = ('boolean',) trackReferences = False def sliceBody(self, streamable, banana): if self.obj: yield 1 else: yield 0 registerAdapter(BooleanSlicer, bool, tokens.ISlicer) class BooleanUnslicer(LeafUnslicer): opentype = ('boolean',) value = None constraint = None def setConstraint(self, constraint): if isinstance(constraint, Any): return assert isinstance(constraint, BooleanConstraint) self.constraint = constraint def checkToken(self, typebyte, size): if typebyte != tokens.INT: raise BananaError("BooleanUnslicer only accepts an INT token") if self.value != None: raise BananaError("BooleanUnslicer only accepts one token") def receiveChild(self, obj, ready_deferred=None): assert not isinstance(obj, Deferred) assert ready_deferred is None assert type(obj) == int if self.constraint: if self.constraint.value != None: if bool(obj) != self.constraint.value: raise Violation("This boolean can only be %s" % \ self.constraint.value) self.value = bool(obj) def receiveClose(self): return self.value, None def describe(self): return "" class BooleanConstraint(OpenerConstraint): strictTaster = True opentypes = [("boolean",)] _myint = IntegerConstraint() name = "BooleanConstraint" def __init__(self, value=None): # self.value is a joke. This allows you to use a schema of # BooleanConstraint(True) which only accepts 'True'. I cannot # imagine a possible use for this, but it made me laugh. self.value = value def checkObject(self, obj, inbound): if type(obj) != bool: raise Violation("not a bool") if self.value != None: if obj != self.value: raise Violation("not %s" % self.value) foolscap-0.13.1/src/foolscap/slicers/decimal_slicer.py0000644000076500000240000000246112766553111023407 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test.test_banana -*- import decimal from twisted.internet.defer import Deferred from foolscap.tokens import BananaError, STRING, VOCAB from foolscap.slicer import BaseSlicer, LeafUnslicer from foolscap.constraint import Any class DecimalSlicer(BaseSlicer): opentype = ("decimal",) slices = decimal.Decimal def sliceBody(self, streamable, banana): yield str(self.obj) class DecimalUnslicer(LeafUnslicer): opentype = ("decimal",) value = None constraint = None def setConstraint(self, constraint): if isinstance(constraint, Any): return assert False, "DecimalUnslicer does not currently accept a constraint" def checkToken(self, typebyte, size): if typebyte not in (STRING, VOCAB): raise BananaError("DecimalUnslicer only accepts strings") #if self.constraint: # self.constraint.checkToken(typebyte, size) def receiveChild(self, obj, ready_deferred=None): assert not isinstance(obj, Deferred) assert ready_deferred is None if self.value != None: raise BananaError("already received a string") self.value = decimal.Decimal(obj) def receiveClose(self): return self.value, None def describe(self): return "" foolscap-0.13.1/src/foolscap/slicers/dict.py0000644000076500000240000001204212766553111021367 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test.test_banana -*- from twisted.python import log from twisted.internet.defer import Deferred from foolscap.tokens import Violation, BananaError from foolscap.slicer import BaseSlicer, BaseUnslicer from foolscap.constraint import OpenerConstraint, Any, IConstraint from foolscap.util import AsyncAND class DictSlicer(BaseSlicer): opentype = ('dict',) trackReferences = True slices = None def sliceBody(self, streamable, banana): for key,value in self.obj.items(): yield key yield value class DictUnslicer(BaseUnslicer): opentype = ('dict',) gettingKey = True keyConstraint = None valueConstraint = None maxKeys = None def setConstraint(self, constraint): if isinstance(constraint, Any): return assert isinstance(constraint, DictConstraint) self.keyConstraint = constraint.keyConstraint self.valueConstraint = constraint.valueConstraint self.maxKeys = constraint.maxKeys def start(self, count): self.d = {} self.protocol.setObject(count, self.d) self.key = None self._ready_deferreds = [] def checkToken(self, typebyte, size): if self.maxKeys != None: if len(self.d) >= self.maxKeys: raise Violation("the dict is full") if self.gettingKey: if self.keyConstraint: self.keyConstraint.checkToken(typebyte, size) else: if self.valueConstraint: self.valueConstraint.checkToken(typebyte, size) def doOpen(self, opentype): if self.maxKeys != None: if len(self.d) >= self.maxKeys: raise Violation("the dict is full") if self.gettingKey: if self.keyConstraint: self.keyConstraint.checkOpentype(opentype) else: if self.valueConstraint: self.valueConstraint.checkOpentype(opentype) unslicer = self.open(opentype) if unslicer: if self.gettingKey: if self.keyConstraint: unslicer.setConstraint(self.keyConstraint) else: if self.valueConstraint: unslicer.setConstraint(self.valueConstraint) return unslicer def update(self, value, key): # this is run as a Deferred callback, hence the backwards arguments self.d[key] = value def receiveChild(self, obj, ready_deferred=None): if ready_deferred: self._ready_deferreds.append(ready_deferred) if self.gettingKey: self.receiveKey(obj) else: self.receiveValue(obj) self.gettingKey = not self.gettingKey def receiveKey(self, key): # I don't think it is legal (in python) to use an incomplete object # as a dictionary key, because you must have all the contents to # hash it. Someone could fake up a token stream to hit this case, # however: OPEN(dict), OPEN(tuple), OPEN(reference), 0, CLOSE, CLOSE, # "value", CLOSE if isinstance(key, Deferred): raise BananaError("incomplete object as dictionary key") try: if self.d.has_key(key): raise BananaError("duplicate key '%s'" % key) except TypeError: raise BananaError("unhashable key '%s'" % key) self.key = key def receiveValue(self, value): if isinstance(value, Deferred): value.addCallback(self.update, self.key) value.addErrback(log.err) self.d[self.key] = value # placeholder def receiveClose(self): ready_deferred = None if self._ready_deferreds: ready_deferred = AsyncAND(self._ready_deferreds) return self.d, ready_deferred def describe(self): if self.gettingKey: return "{}" else: return "{}[%s]" % self.key class OrderedDictSlicer(DictSlicer): slices = dict def sliceBody(self, streamable, banana): keys = self.obj.keys() keys.sort() for key in keys: value = self.obj[key] yield key yield value class DictConstraint(OpenerConstraint): opentypes = [("dict",)] name = "DictConstraint" def __init__(self, keyConstraint, valueConstraint, maxKeys=None): self.keyConstraint = IConstraint(keyConstraint) self.valueConstraint = IConstraint(valueConstraint) self.maxKeys = maxKeys def checkObject(self, obj, inbound): if not isinstance(obj, dict): raise Violation, "'%s' (%s) is not a Dictionary" % (obj, type(obj)) if self.maxKeys != None and len(obj) > self.maxKeys: raise Violation, "Dict keys=%d > maxKeys=%d" % (len(obj), self.maxKeys) for key, value in obj.iteritems(): self.keyConstraint.checkObject(key, inbound) self.valueConstraint.checkObject(value, inbound) foolscap-0.13.1/src/foolscap/slicers/list.py0000644000076500000240000001203712766553111021423 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test.test_banana -*- from twisted.python import log from twisted.internet.defer import Deferred from foolscap.tokens import Violation from foolscap.slicer import BaseSlicer, BaseUnslicer from foolscap.constraint import OpenerConstraint, Any, IConstraint from foolscap.util import AsyncAND class ListSlicer(BaseSlicer): opentype = ("list",) trackReferences = True slices = list def sliceBody(self, streamable, banana): for i in self.obj: yield i class ListUnslicer(BaseUnslicer): opentype = ("list",) maxLength = None itemConstraint = None debug = False def setConstraint(self, constraint): if isinstance(constraint, Any): return assert isinstance(constraint, ListConstraint) self.maxLength = constraint.maxLength self.itemConstraint = constraint.constraint def start(self, count): #self.opener = foo # could replace it if we wanted to self.list = [] self.count = count if self.debug: log.msg("%s[%d].start with %s" % (self, self.count, self.list)) self.protocol.setObject(count, self.list) self._ready_deferreds = [] def checkToken(self, typebyte, size): if self.maxLength != None and len(self.list) >= self.maxLength: # list is full, no more tokens accepted # this is hit if the max+1 item is a primitive type raise Violation("the list is full") if self.itemConstraint: self.itemConstraint.checkToken(typebyte, size) def doOpen(self, opentype): # decide whether the given object type is acceptable here. Raise a # Violation exception if not, otherwise give it to our opener (which # will normally be the RootUnslicer). Apply a constraint to the new # unslicer. if self.maxLength != None and len(self.list) >= self.maxLength: # this is hit if the max+1 item is a non-primitive type raise Violation("the list is full") if self.itemConstraint: self.itemConstraint.checkOpentype(opentype) unslicer = self.open(opentype) if unslicer: if self.itemConstraint: unslicer.setConstraint(self.itemConstraint) return unslicer def update(self, obj, index): # obj has already passed typechecking if self.debug: log.msg("%s[%d].update: [%d]=%s" % (self, self.count, index, obj)) assert isinstance(index, int) self.list[index] = obj return obj def receiveChild(self, obj, ready_deferred=None): if ready_deferred: self._ready_deferreds.append(ready_deferred) if self.debug: log.msg("%s[%d].receiveChild(%s)" % (self, self.count, obj)) # obj could be a primitive type, a Deferred, or a complex type like # those returned from an InstanceUnslicer. However, the individual # object has already been through the schema validation process. The # only remaining question is whether the larger schema will accept # it. if self.maxLength != None and len(self.list) >= self.maxLength: # this is redundant # (if it were a non-primitive one, it would be caught in doOpen) # (if it were a primitive one, it would be caught in checkToken) raise Violation("the list is full") if isinstance(obj, Deferred): if self.debug: log.msg(" adding my update[%d] to %s" % (len(self.list), obj)) obj.addCallback(self.update, len(self.list)) obj.addErrback(self.printErr) placeholder = "list placeholder for arg[%d], rd=%s" % \ (len(self.list), ready_deferred) self.list.append(placeholder) else: self.list.append(obj) def printErr(self, why): print "ERR!" print why.getBriefTraceback() log.err(why) def receiveClose(self): ready_deferred = None if self._ready_deferreds: ready_deferred = AsyncAND(self._ready_deferreds) return self.list, ready_deferred def describe(self): return "[%d]" % len(self.list) class ListConstraint(OpenerConstraint): """The object must be a list of objects, with a given maximum length. To accept lists of any length, use maxLength=None. All member objects must obey the given constraint.""" opentypes = [("list",)] name = "ListConstraint" def __init__(self, constraint, maxLength=None, minLength=0): self.constraint = IConstraint(constraint) self.maxLength = maxLength self.minLength = minLength def checkObject(self, obj, inbound): if not isinstance(obj, list): raise Violation("not a list") if self.maxLength is not None and len(obj) > self.maxLength: raise Violation("list too long") if len(obj) < self.minLength: raise Violation("list too short") for o in obj: self.constraint.checkObject(o, inbound) foolscap-0.13.1/src/foolscap/slicers/none.py0000644000076500000240000000177512766553111021416 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test.test_banana -*- from foolscap.tokens import Violation, BananaError from foolscap.slicer import BaseSlicer, LeafUnslicer from foolscap.constraint import OpenerConstraint class NoneSlicer(BaseSlicer): opentype = ('none',) trackReferences = False slices = type(None) def sliceBody(self, streamable, banana): # hmm, we need an empty generator. I think a sequence is the only way # to accomplish this, other than 'if 0: yield' or something silly return [] class NoneUnslicer(LeafUnslicer): opentype = ('none',) def checkToken(self, typebyte, size): raise BananaError("NoneUnslicer does not accept any tokens") def receiveClose(self): return None, None class Nothing(OpenerConstraint): """Accept only 'None'.""" strictTaster = True opentypes = [("none",)] name = "Nothing" def checkObject(self, obj, inbound): if obj is not None: raise Violation("'%s' is not None" % (obj,)) foolscap-0.13.1/src/foolscap/slicers/root.py0000644000076500000240000002331612766553111021435 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test.test_banana -*- import types from zope.interface import implements from twisted.internet.defer import Deferred from foolscap import tokens from foolscap.tokens import Violation, BananaError from foolscap.slicer import BaseUnslicer, ReferenceSlicer from foolscap.slicer import UnslicerRegistry, BananaUnslicerRegistry from foolscap.slicers.vocab import ReplaceVocabularyTable, AddToVocabularyTable from foolscap import copyable # does this create a cycle? from twisted.python import log class RootSlicer: implements(tokens.ISlicer, tokens.IRootSlicer) streamableInGeneral = True producingDeferred = None objectSentDeferred = None slicerTable = {} debug = False def __init__(self, protocol): self.protocol = protocol self.sendQueue = [] def allowStreaming(self, streamable): self.streamableInGeneral = streamable def registerRefID(self, refid, obj): pass def slicerForObject(self, obj): # could use a table here if you think it'd be faster than an # adapter lookup if self.debug: log.msg("slicerForObject(%s)" % type(obj)) # do the adapter lookup first, so that registered adapters override # UnsafeSlicerTable's InstanceSlicer slicer = tokens.ISlicer(obj, None) if slicer: if self.debug: log.msg("got ISlicer %s" % slicer) return slicer # zope.interface doesn't do transitive adaptation, which is a shame # because we want to let people register ICopyable adapters for # third-party code, and there is an ICopyable->ISlicer adapter # defined in copyable.py, but z.i won't do the transitive # ThirdPartyClass -> ICopyable -> ISlicer # so instead we manually do it here copier = copyable.ICopyable(obj, None) if copier: s = tokens.ISlicer(copier) return s slicerFactory = self.slicerTable.get(type(obj)) if slicerFactory: if self.debug: log.msg(" got slicerFactory %s" % slicerFactory) return slicerFactory(obj) if issubclass(type(obj), types.InstanceType): name = str(obj.__class__) else: name = str(type(obj)) if self.debug: log.msg("cannot serialize %s (%s)" % (obj, name)) raise Violation("cannot serialize %s (%s)" % (obj, name)) def slice(self): return self def __iter__(self): return self # we are our own iterator def next(self): if self.objectSentDeferred: self.objectSentDeferred.callback(None) self.objectSentDeferred = None if self.sendQueue: (obj, self.objectSentDeferred) = self.sendQueue.pop() self.streamable = self.streamableInGeneral return obj if self.protocol.debugSend: print "LAST BAG" self.producingDeferred = Deferred() self.streamable = True return self.producingDeferred def childAborted(self, f): assert self.objectSentDeferred self.objectSentDeferred.errback(f) self.objectSentDeferred = None return None def send(self, obj): # obj can also be a Slicer, say, a CallSlicer. We return a Deferred # which fires when the object has been fully serialized. idle = (len(self.protocol.slicerStack) == 1) and not self.sendQueue objectSentDeferred = Deferred() self.sendQueue.append((obj, objectSentDeferred)) if idle: # wake up if self.protocol.debugSend: print " waking up to send" if self.producingDeferred: d = self.producingDeferred self.producingDeferred = None # TODO: consider reactor.callLater(0, d.callback, None) # I'm not sure it's actually necessary, though d.callback(None) return objectSentDeferred def describe(self): return "" def connectionLost(self, why): # abandon everything we wanted to send if self.objectSentDeferred: self.objectSentDeferred.errback(why) self.objectSentDeferred = None for obj, d in self.sendQueue: d.errback(why) self.sendQueue = [] class ScopedRootSlicer(RootSlicer): # this combines RootSlicer with foolscap.slicer.ScopedSlicer . The funny # self-delegation of slicerForObject() means we can't just inherit from # both. It would be nice to refactor everything to make this cleaner. def __init__(self, obj): RootSlicer.__init__(self, obj) self.references = {} # maps id(obj) -> (obj,refid) def registerRefID(self, refid, obj): self.references[id(obj)] = (obj,refid) def slicerForObject(self, obj): # check for an object which was sent previously or has at least # started sending obj_refid = self.references.get(id(obj), None) if obj_refid is not None: # we've started to send this object already, so just include a # reference to it return ReferenceSlicer(obj_refid[1]) # otherwise go upstream so we can serialize the object completely return RootSlicer.slicerForObject(self, obj) class RootUnslicer(BaseUnslicer): # topRegistries is used for top-level objects topRegistries = [UnslicerRegistry, BananaUnslicerRegistry] # openRegistries is used for everything at lower levels openRegistries = [UnslicerRegistry] constraint = None openCount = None def __init__(self, protocol): self.protocol = protocol self.objects = {} keys = [] for r in self.topRegistries + self.openRegistries: for k in r.keys(): keys.append(len(k[0])) self.maxIndexLength = reduce(max, keys) def start(self, count): pass def setConstraint(self, constraint): # this constraints top-level objects. E.g., if this is an # IntegerConstraint, then only integers will be accepted. self.constraint = constraint def checkToken(self, typebyte, size): if self.constraint: self.constraint.checkToken(typebyte, size) def openerCheckToken(self, typebyte, size, opentype): if typebyte == tokens.STRING: if size > self.maxIndexLength: why = "STRING token is too long, %d>%d" % \ (size, self.maxIndexLength) raise Violation(why) elif typebyte == tokens.VOCAB: return else: # TODO: hack for testing raise Violation("index token 0x%02x not STRING or VOCAB" % \ ord(typebyte)) raise BananaError("index token 0x%02x not STRING or VOCAB" % \ ord(typebyte)) def open(self, opentype): # called (by delegation) by the top Unslicer on the stack, regardless # of what kind of unslicer it is. This is only used for "internal" # objects: non-top-level nodes assert len(self.protocol.receiveStack) > 1 if opentype[0] == 'copyable': if len(opentype) > 1: copyablename = opentype[1] try: factory = copyable.CopyableRegistry[copyablename] except KeyError: raise Violation("unknown RemoteCopy name '%s'" \ % copyablename) child = factory() return child return None # still waiting for copyablename for reg in self.openRegistries: opener = reg.get(opentype) if opener is not None: child = opener() return child raise Violation("unknown OPEN type %s" % (opentype,)) def doOpen(self, opentype): # this is only called for top-level objects assert len(self.protocol.receiveStack) == 1 if self.constraint: self.constraint.checkOpentype(opentype) for reg in self.topRegistries: opener = reg.get(opentype) if opener is not None: child = opener() break else: raise Violation("unknown top-level OPEN type %s" % (opentype,)) if self.constraint: child.setConstraint(self.constraint) return child def receiveChild(self, obj, ready_deferred=None): assert not isinstance(obj, Deferred) assert ready_deferred is None if self.protocol.debugReceive: print "RootUnslicer.receiveChild(%s)" % (obj,) self.objects = {} if obj in (ReplaceVocabularyTable, AddToVocabularyTable): # the unslicer has already changed the vocab table return if self.protocol.exploded: print "protocol exploded, can't deliver object" print self.protocol.exploded self.protocol.receivedObject(self.protocol.exploded) return self.protocol.receivedObject(obj) # give finished object to Banana def receiveClose(self): raise BananaError("top-level should never receive CLOSE tokens") def reportViolation(self, why): return self.protocol.reportViolation(why) def describe(self): return "" def setObject(self, counter, obj): pass def getObject(self, counter): return None class ScopedRootUnslicer(RootUnslicer): # combines RootUnslicer and ScopedUnslicer def __init__(self, protocol): RootUnslicer.__init__(self, protocol) self.references = {} def setObject(self, counter, obj): self.references[counter] = obj def getObject(self, counter): obj = self.references.get(counter) return obj foolscap-0.13.1/src/foolscap/slicers/set.py0000644000076500000240000001515312766553111021245 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test.test_banana -*- from twisted.internet import defer from twisted.python import log from foolscap.slicers.list import ListSlicer from foolscap.slicers.tuple import TupleUnslicer from foolscap.slicer import BaseUnslicer from foolscap.tokens import Violation from foolscap.constraint import OpenerConstraint, Any, IConstraint from foolscap.util import AsyncAND class SetSlicer(ListSlicer): opentype = ("set",) trackReferences = True slices = set def sliceBody(self, streamable, banana): for i in self.obj: yield i class FrozenSetSlicer(SetSlicer): opentype = ("immutable-set",) trackReferences = False slices = frozenset class _Placeholder: pass class SetUnslicer(BaseUnslicer): # this is a lot like a list, but sufficiently different to make it not # worth subclassing opentype = ("set",) debug = False maxLength = None itemConstraint = None def setConstraint(self, constraint): if isinstance(constraint, Any): return assert isinstance(constraint, SetConstraint) self.maxLength = constraint.maxLength self.itemConstraint = constraint.constraint def start(self, count): #self.opener = foo # could replace it if we wanted to self.set = set() self.count = count if self.debug: log.msg("%s[%d].start with %s" % (self, self.count, self.set)) self.protocol.setObject(count, self.set) self._ready_deferreds = [] def checkToken(self, typebyte, size): if self.maxLength != None and len(self.set) >= self.maxLength: # list is full, no more tokens accepted # this is hit if the max+1 item is a primitive type raise Violation("the set is full") if self.itemConstraint: self.itemConstraint.checkToken(typebyte, size) def doOpen(self, opentype): # decide whether the given object type is acceptable here. Raise a # Violation exception if not, otherwise give it to our opener (which # will normally be the RootUnslicer). Apply a constraint to the new # unslicer. if self.maxLength != None and len(self.set) >= self.maxLength: # this is hit if the max+1 item is a non-primitive type raise Violation("the set is full") if self.itemConstraint: self.itemConstraint.checkOpentype(opentype) unslicer = self.open(opentype) if unslicer: if self.itemConstraint: unslicer.setConstraint(self.itemConstraint) return unslicer def update(self, obj, placeholder): # obj has already passed typechecking if self.debug: log.msg("%s[%d].update: [%s]=%s" % (self, self.count, placeholder, obj)) self.set.remove(placeholder) self.set.add(obj) return obj def receiveChild(self, obj, ready_deferred=None): if ready_deferred: self._ready_deferreds.append(ready_deferred) if self.debug: log.msg("%s[%d].receiveChild(%s)" % (self, self.count, obj)) # obj could be a primitive type, a Deferred, or a complex type like # those returned from an InstanceUnslicer. However, the individual # object has already been through the schema validation process. The # only remaining question is whether the larger schema will accept # it. if self.maxLength != None and len(self.set) >= self.maxLength: # this is redundant # (if it were a non-primitive one, it would be caught in doOpen) # (if it were a primitive one, it would be caught in checkToken) raise Violation("the set is full") if isinstance(obj, defer.Deferred): if self.debug: log.msg(" adding my update[%d] to %s" % (len(self.set), obj)) # note: the placeholder isn't strictly necessary, but it will # help debugging to see a _Placeholder sitting in the set when it # shouldn't rather than seeing a set that is smaller than it # ought to be. If a remote method ever sees a _Placeholder, then # something inside Foolscap has broken. placeholder = _Placeholder() obj.addCallback(self.update, placeholder) obj.addErrback(self.printErr) self.set.add(placeholder) else: self.set.add(obj) def printErr(self, why): print "ERR!" print why.getBriefTraceback() log.err(why) def receiveClose(self): ready_deferred = None if self._ready_deferreds: ready_deferred = AsyncAND(self._ready_deferreds) return self.set, ready_deferred class FrozenSetUnslicer(TupleUnslicer): opentype = ("immutable-set",) def receiveClose(self): obj_or_deferred, ready_deferred = TupleUnslicer.receiveClose(self) if isinstance(obj_or_deferred, defer.Deferred): def _convert(the_tuple): return frozenset(the_tuple) obj_or_deferred.addCallback(_convert) else: obj_or_deferred = frozenset(obj_or_deferred) return obj_or_deferred, ready_deferred class SetConstraint(OpenerConstraint): """The object must be a Set of some sort, with a given maximum size. To accept sets of any size, use maxLength=None. All member objects must obey the given constraint. By default this will accept both mutable and immutable sets, if you want to require a particular type, set mutable= to either True or False. """ # TODO: if mutable!=None, we won't throw out the wrong set type soon # enough. We need to override checkOpenType to accomplish this. opentypes = [("set",), ("immutable-set",)] name = "SetConstraint" def __init__(self, constraint, maxLength=None, mutable=None): self.constraint = IConstraint(constraint) self.maxLength = maxLength self.mutable = mutable def checkObject(self, obj, inbound): if not isinstance(obj, (set, frozenset)): raise Violation("not a set") if (self.mutable == True and not isinstance(obj, set)): raise Violation("obj is a set, but not a mutable one") if (self.mutable == False and not isinstance(obj, frozenset)): raise Violation("obj is a set, but not an immutable one") if self.maxLength is not None and len(obj) > self.maxLength: raise Violation("set is too large") if self.constraint: for o in obj: self.constraint.checkObject(o, inbound) foolscap-0.13.1/src/foolscap/slicers/tuple.py0000644000076500000240000001104012766553111021572 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test.test_banana -*- from twisted.internet.defer import Deferred from foolscap.tokens import Violation from foolscap.slicer import BaseUnslicer from foolscap.slicers.list import ListSlicer from foolscap.constraint import OpenerConstraint, Any, IConstraint from foolscap.util import AsyncAND class TupleSlicer(ListSlicer): opentype = ("tuple",) slices = tuple class TupleUnslicer(BaseUnslicer): opentype = ("tuple",) debug = False constraints = None def setConstraint(self, constraint): if isinstance(constraint, Any): return assert isinstance(constraint, TupleConstraint) self.constraints = constraint.constraints def start(self, count): self.list = [] # indices of .list which are unfilled because of children that could # not yet be referenced self.num_unreferenceable_children = 0 self.count = count if self.debug: print "%s[%d].start with %s" % (self, self.count, self.list) self.finished = False self.deferred = Deferred() self.protocol.setObject(count, self.deferred) self._ready_deferreds = [] def checkToken(self, typebyte, size): if self.constraints == None: return if len(self.list) >= len(self.constraints): raise Violation("the tuple is full") self.constraints[len(self.list)].checkToken(typebyte, size) def doOpen(self, opentype): where = len(self.list) if self.constraints != None: if where >= len(self.constraints): raise Violation("the tuple is full") self.constraints[where].checkOpentype(opentype) unslicer = self.open(opentype) if unslicer: if self.constraints != None: unslicer.setConstraint(self.constraints[where]) return unslicer def update(self, obj, index): if self.debug: print "%s[%d].update: [%d]=%s" % (self, self.count, index, obj) self.list[index] = obj self.num_unreferenceable_children -= 1 if self.finished: self.checkComplete() return obj def receiveChild(self, obj, ready_deferred=None): if ready_deferred: self._ready_deferreds.append(ready_deferred) if isinstance(obj, Deferred): obj.addCallback(self.update, len(self.list)) obj.addErrback(self.explode) self.num_unreferenceable_children += 1 self.list.append("placeholder") else: self.list.append(obj) def checkComplete(self): if self.debug: print "%s[%d].checkComplete: %d pending" % \ (self, self.count, self.num_unreferenceable_children) if self.num_unreferenceable_children: # not finished yet, we'll fire our Deferred when we are if self.debug: print " not finished yet" return # list is now complete. We can finish. return self.complete() def complete(self): ready_deferred = None if self._ready_deferreds: ready_deferred = AsyncAND(self._ready_deferreds) t = tuple(self.list) if self.debug: print " finished! tuple:%s{%s}" % (t, id(t)) self.protocol.setObject(self.count, t) self.deferred.callback(t) return t, ready_deferred def receiveClose(self): if self.debug: print "%s[%d].receiveClose" % (self, self.count) self.finished = 1 if self.num_unreferenceable_children: # not finished yet, we'll fire our Deferred when we are if self.debug: print " not finished yet" ready_deferred = None if self._ready_deferreds: ready_deferred = AsyncAND(self._ready_deferreds) return self.deferred, ready_deferred # the list is already complete return self.complete() def describe(self): return "[%d]" % len(self.list) class TupleConstraint(OpenerConstraint): opentypes = [("tuple",)] name = "TupleConstraint" def __init__(self, *elemConstraints): self.constraints = [IConstraint(e) for e in elemConstraints] def checkObject(self, obj, inbound): if not isinstance(obj, tuple): raise Violation("not a tuple") if len(obj) != len(self.constraints): raise Violation("wrong size tuple") for i in range(len(self.constraints)): self.constraints[i].checkObject(obj[i], inbound) foolscap-0.13.1/src/foolscap/slicers/unicode.py0000644000076500000240000000637313204503345022074 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test.test_banana -*- import re from twisted.internet.defer import Deferred from foolscap.tokens import BananaError, STRING, VOCAB, Violation from foolscap.slicer import BaseSlicer, LeafUnslicer from foolscap.constraint import OpenerConstraint, Any class UnicodeSlicer(BaseSlicer): opentype = ("unicode",) slices = unicode def sliceBody(self, streamable, banana): yield self.obj.encode("UTF-8") class UnicodeUnslicer(LeafUnslicer): # accept a UTF-8 encoded string opentype = ("unicode",) string = None constraint = None def setConstraint(self, constraint): if isinstance(constraint, Any): return assert isinstance(constraint, UnicodeConstraint) self.constraint = constraint def checkToken(self, typebyte, size): if typebyte not in (STRING, VOCAB): raise BananaError("UnicodeUnslicer only accepts strings") #if self.constraint: # self.constraint.checkToken(typebyte, size) def receiveChild(self, obj, ready_deferred=None): assert not isinstance(obj, Deferred) assert ready_deferred is None if self.string != None: raise BananaError("already received a string") self.string = unicode(obj, "UTF-8") def receiveClose(self): return self.string, None def describe(self): return "" class UnicodeConstraint(OpenerConstraint): """The object must be a unicode object. The maxLength and minLength parameters restrict the number of characters (code points, *not* bytes) that may be present in the object, which means that the on-wire (UTF-8) representation may take up to 6 times as many bytes as characters. """ strictTaster = True opentypes = [("unicode",)] name = "UnicodeConstraint" def __init__(self, maxLength=None, minLength=0, regexp=None): self.maxLength = maxLength self.minLength = minLength # allow VOCAB in case the Banana-level tokenizer decides to tokenize # the UTF-8 encoded body of a unicode object, since this is just as # likely as tokenizing regular bytestrings. TODO: this is disabled # because it doesn't currently work.. once I remember how Constraints # work, I'll fix this. The current version is too permissive of # tokens. #self.taster = {STRING: 6*self.maxLength, # VOCAB: None} # regexp can either be a string or a compiled SRE_Match object.. # re.compile appears to notice SRE_Match objects and pass them # through unchanged. self.regexp = None if regexp: self.regexp = re.compile(regexp) def checkObject(self, obj, inbound): if not isinstance(obj, unicode): raise Violation("not a unicode object") if self.maxLength != None and len(obj) > self.maxLength: raise Violation("string too long (%d > %d)" % (len(obj), self.maxLength)) if len(obj) < self.minLength: raise Violation("string too short (%d < %d)" % (len(obj), self.minLength)) if self.regexp: if not self.regexp.search(obj): raise Violation("regexp failed to match") foolscap-0.13.1/src/foolscap/slicers/vocab.py0000644000076500000240000001520212766553111021537 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test.test_banana -*- from twisted.internet.defer import Deferred from foolscap.constraint import Any, ByteStringConstraint from foolscap.tokens import Violation, BananaError, INT, STRING from foolscap.slicer import BaseSlicer, BaseUnslicer, LeafUnslicer from foolscap.slicer import BananaUnslicerRegistry class ReplaceVocabularyTable: pass class AddToVocabularyTable: pass class ReplaceVocabSlicer(BaseSlicer): # this works somewhat like a dictionary opentype = ('set-vocab',) trackReferences = False def slice(self, streamable, banana): # we need to implement slice() (instead of merely sliceBody) so we # can get control at the beginning and end of serialization. It also # gives us access to the Banana protocol object, so we can manipulate # their outgoingVocabulary table. self.streamable = streamable self.start(banana) for o in self.opentype: yield o # the vocabDict maps strings to index numbers. The far end needs the # opposite mapping, from index numbers to strings. We perform the # flip here at the sending end. stringToIndex = self.obj indexToString = dict([(stringToIndex[s],s) for s in stringToIndex]) assert len(stringToIndex) == len(indexToString) # catch duplicates indices = indexToString.keys() indices.sort() for index in indices: string = indexToString[index] yield index yield string self.finish(banana) def start(self, banana): # this marks the transition point between the old vocabulary dict and # the new one, so now is the time we should empty the dict. banana.outgoingVocabTableWasReplaced({}) def finish(self, banana): # now we replace the vocab dict banana.outgoingVocabTableWasReplaced(self.obj) class ReplaceVocabUnslicer(LeafUnslicer): """Much like DictUnslicer, but keys must be numbers, and values must be strings. This is used to set the entire vocab table at once. To add individual tokens, use AddVocabUnslicer by sending an (add-vocab num string) sequence.""" opentype = ('set-vocab',) unslicerRegistry = BananaUnslicerRegistry maxKeys = None valueConstraint = ByteStringConstraint(100) def setConstraint(self, constraint): if isinstance(constraint, Any): return assert isinstance(constraint, ByteStringConstraint) self.valueConstraint = constraint def start(self, count): self.d = {} self.key = None def checkToken(self, typebyte, size): if self.maxKeys is not None and len(self.d) >= self.maxKeys: raise Violation("the table is full") if self.key is None: if typebyte != INT: raise BananaError("VocabUnslicer only accepts INT keys") else: if typebyte != STRING: raise BananaError("VocabUnslicer only accepts STRING values") if self.valueConstraint: self.valueConstraint.checkToken(typebyte, size) def receiveChild(self, token, ready_deferred=None): assert not isinstance(token, Deferred) assert ready_deferred is None if self.key is None: if self.d.has_key(token): raise BananaError("duplicate key '%s'" % token) self.key = token else: self.d[self.key] = token self.key = None def receiveClose(self): if self.key is not None: raise BananaError("sequence ended early: got key but not value") # now is the time we replace our protocol's vocab table self.protocol.replaceIncomingVocabulary(self.d) return ReplaceVocabularyTable, None def describe(self): if self.key is not None: return "[%s]" % self.key else: return "" class AddVocabSlicer(BaseSlicer): opentype = ('add-vocab',) trackReferences = False def __init__(self, value): assert isinstance(value, str) self.value = value def slice(self, streamable, banana): # we need to implement slice() (instead of merely sliceBody) so we # can get control at the beginning and end of serialization. It also # gives us access to the Banana protocol object, so we can manipulate # their outgoingVocabulary table. self.streamable = streamable self.start(banana) for o in self.opentype: yield o yield self.index yield self.value self.finish(banana) def start(self, banana): # this marks the transition point between the old vocabulary dict and # the new one, so now is the time we should decide upon the key. It # is important that we *do not* add it to the dict yet, otherwise # we'll send (add-vocab NN [VOCAB#NN]), which is kind of pointless. index = banana.allocateEntryInOutgoingVocabTable(self.value) self.index = index def finish(self, banana): banana.outgoingVocabTableWasAmended(self.index, self.value) class AddVocabUnslicer(BaseUnslicer): # (add-vocab num string): self.vocab[num] = string opentype = ('add-vocab',) unslicerRegistry = BananaUnslicerRegistry index = None value = None valueConstraint = ByteStringConstraint(100) def setConstraint(self, constraint): if isinstance(constraint, Any): return assert isinstance(constraint, ByteStringConstraint) self.valueConstraint = constraint def checkToken(self, typebyte, size): if self.index is None: if typebyte != INT: raise BananaError("Vocab key must be an INT") elif self.value is None: if typebyte != STRING: raise BananaError("Vocab value must be a STRING") if self.valueConstraint: self.valueConstraint.checkToken(typebyte, size) else: raise Violation("add-vocab only accepts two values") def receiveChild(self, obj, ready_deferred=None): assert not isinstance(obj, Deferred) assert ready_deferred is None if self.index is None: self.index = obj else: self.value = obj def receiveClose(self): if self.index is None or self.value is None: raise BananaError("sequence ended too early") self.protocol.addIncomingVocabulary(self.index, self.value) return AddToVocabularyTable, None def describe(self): if self.index is not None: return "[%d]" % self.index return "" foolscap-0.13.1/src/foolscap/storage.py0000644000076500000240000003616012766553111020453 0ustar warnerstaff00000000000000 """ storage.py: support for using Banana as if it were pickle This includes functions for serializing to and from strings, instead of a network socket. It also has support for serializing 'unsafe' objects, specifically classes, modules, functions, and instances of arbitrary classes. These are 'unsafe' because to recreate the object on the deserializing end, we must be willing to execute code of the sender's choosing (i.e. the constructor of whatever package.module.class names they send us). It is unwise to do this unless you are willing to allow your internal state to be compromised by the author of the serialized data you're unpacking. This functionality is isolated here because it is never used for data coming over network connections. """ from cStringIO import StringIO import types from new import instance, instancemethod from pickle import whichmodule # used by FunctionSlicer from foolscap import slicer, banana, tokens from foolscap.tokens import BananaError from twisted.internet.defer import Deferred from twisted.python import reflect from foolscap.slicers.dict import OrderedDictSlicer from foolscap.slicers.root import ScopedRootSlicer, ScopedRootUnslicer ################## Slicers for "unsafe" things # Extended types, not generally safe. The UnsafeRootSlicer checks for these # with a separate table. def getInstanceState(inst): """Utility function to default to 'normal' state rules in serialization. """ if hasattr(inst, "__getstate__"): state = inst.__getstate__() else: state = inst.__dict__ return state class InstanceSlicer(OrderedDictSlicer): opentype = ('instance',) trackReferences = True def sliceBody(self, streamable, banana): yield reflect.qual(self.obj.__class__) # really a second index token self.obj = getInstanceState(self.obj) for t in OrderedDictSlicer.sliceBody(self, streamable, banana): yield t class ModuleSlicer(slicer.BaseSlicer): opentype = ('module',) trackReferences = True def sliceBody(self, streamable, banana): yield self.obj.__name__ class ClassSlicer(slicer.BaseSlicer): opentype = ('class',) trackReferences = True def sliceBody(self, streamable, banana): yield reflect.qual(self.obj) class MethodSlicer(slicer.BaseSlicer): opentype = ('method',) trackReferences = True def sliceBody(self, streamable, banana): yield self.obj.im_func.__name__ yield self.obj.im_self yield self.obj.im_class class FunctionSlicer(slicer.BaseSlicer): opentype = ('function',) trackReferences = True def sliceBody(self, streamable, banana): name = self.obj.__name__ fullname = str(whichmodule(self.obj, self.obj.__name__)) + '.' + name yield fullname UnsafeSlicerTable = {} UnsafeSlicerTable.update({ types.InstanceType: InstanceSlicer, types.ModuleType: ModuleSlicer, types.ClassType: ClassSlicer, types.MethodType: MethodSlicer, types.FunctionType: FunctionSlicer, #types.TypeType: NewstyleClassSlicer, # ???: NewstyleInstanceSlicer, # pickle uses obj.__reduce__ to help # http://docs.python.org/lib/node68.html }) # the root slicer for storage is exactly like the regular root slicer class StorageRootSlicer(ScopedRootSlicer): pass # but the "unsafe" one (which handles instances and stuff) uses its own table class UnsafeStorageRootSlicer(StorageRootSlicer): slicerTable = UnsafeSlicerTable ################## Unslicers for "unsafe" things def setInstanceState(inst, state): """Utility function to default to 'normal' state rules in unserialization. """ if hasattr(inst, "__setstate__"): inst.__setstate__(state) else: inst.__dict__ = state return inst class Dummy: def __repr__(self): return "" % self.__dict__ def __cmp__(self, other): if not type(other) == type(self): return -1 return cmp(self.__dict__, other.__dict__) UnsafeUnslicerRegistry = {} class InstanceUnslicer(slicer.BaseUnslicer): # this is an unsafe unslicer: an attacker could induce you to create # instances of arbitrary classes with arbitrary attributes: VERY # DANGEROUS! opentype = ('instance',) unslicerRegistry = UnsafeUnslicerRegistry # danger: instances are mutable containers. If an attribute value is not # yet available, __dict__ will hold a Deferred until it is. Other # objects might be created and use our object before this is fixed. # TODO: address this. Note that InstanceUnslicers aren't used in PB # (where we have pb.Referenceable and pb.Copyable which have schema # constraints and could have different restrictions like not being # allowed to participate in reference loops). def start(self, count): self.d = {} self.count = count self.classname = None self.attrname = None self.deferred = Deferred() self.protocol.setObject(count, self.deferred) def checkToken(self, typebyte, size): if self.classname is None: if typebyte not in (tokens.STRING, tokens.VOCAB): raise BananaError("InstanceUnslicer classname must be string") elif self.attrname is None: if typebyte not in (tokens.STRING, tokens.VOCAB): raise BananaError("InstanceUnslicer keys must be STRINGs") def receiveChild(self, obj, ready_deferred=None): assert ready_deferred is None if self.classname is None: self.classname = obj self.attrname = None elif self.attrname is None: self.attrname = obj else: if isinstance(obj, Deferred): # TODO: this is an artificial restriction, and it might # be possible to remove it, but I need to think through # it carefully first raise BananaError("unreferenceable object in attribute") if self.d.has_key(self.attrname): raise BananaError("duplicate attribute name '%s'" % self.attrname) self.setAttribute(self.attrname, obj) self.attrname = None def setAttribute(self, name, value): self.d[name] = value def receiveClose(self): # you could attempt to do some value-checking here, but there would # probably still be holes #obj = Dummy() klass = reflect.namedObject(self.classname) assert type(klass) == types.ClassType # TODO: new-style classes obj = instance(klass, {}) setInstanceState(obj, self.d) self.protocol.setObject(self.count, obj) self.deferred.callback(obj) return obj, None def describe(self): if self.classname is None: return "" me = "<%s>" % self.classname if self.attrname is None: return "%s.attrname??" % me else: return "%s.%s" % (me, self.attrname) class ModuleUnslicer(slicer.LeafUnslicer): opentype = ('module',) unslicerRegistry = UnsafeUnslicerRegistry finished = False def checkToken(self, typebyte, size): if typebyte not in (tokens.STRING, tokens.VOCAB): raise BananaError("ModuleUnslicer only accepts strings") def receiveChild(self, obj, ready_deferred=None): assert not isinstance(obj, Deferred) assert ready_deferred is None if self.finished: raise BananaError("ModuleUnslicer only accepts one string") self.finished = True # TODO: taste here! mod = __import__(obj, {}, {}, "x") self.mod = mod def receiveClose(self): if not self.finished: raise BananaError("ModuleUnslicer requires a string") return self.mod, None class ClassUnslicer(slicer.LeafUnslicer): opentype = ('class',) unslicerRegistry = UnsafeUnslicerRegistry finished = False def checkToken(self, typebyte, size): if typebyte not in (tokens.STRING, tokens.VOCAB): raise BananaError("ClassUnslicer only accepts strings") def receiveChild(self, obj, ready_deferred=None): assert not isinstance(obj, Deferred) assert ready_deferred is None if self.finished: raise BananaError("ClassUnslicer only accepts one string") self.finished = True # TODO: taste here! self.klass = reflect.namedObject(obj) def receiveClose(self): if not self.finished: raise BananaError("ClassUnslicer requires a string") return self.klass, None class MethodUnslicer(slicer.BaseUnslicer): opentype = ('method',) unslicerRegistry = UnsafeUnslicerRegistry state = 0 im_func = None im_self = None im_class = None # self.state: # 0: expecting a string with the method name # 1: expecting an instance (or None for unbound methods) # 2: expecting a class def checkToken(self, typebyte, size): if self.state == 0: if typebyte not in (tokens.STRING, tokens.VOCAB): raise BananaError("MethodUnslicer methodname must be a string") elif self.state == 1: if typebyte != tokens.OPEN: raise BananaError("MethodUnslicer instance must be OPEN") elif self.state == 2: if typebyte != tokens.OPEN: raise BananaError("MethodUnslicer class must be an OPEN") def doOpen(self, opentype): # check the opentype if self.state == 1: if opentype[0] not in ("instance", "none"): raise BananaError("MethodUnslicer instance must be " + "instance or None") elif self.state == 2: if opentype[0] != "class": raise BananaError("MethodUnslicer class must be a class") unslicer = self.open(opentype) # TODO: apply constraint return unslicer def receiveChild(self, obj, ready_deferred=None): assert not isinstance(obj, Deferred) assert ready_deferred is None if self.state == 0: self.im_func = obj self.state = 1 elif self.state == 1: assert type(obj) in (types.InstanceType, types.NoneType) self.im_self = obj self.state = 2 elif self.state == 2: assert type(obj) == types.ClassType # TODO: new-style classes? self.im_class = obj self.state = 3 else: raise BananaError("MethodUnslicer only accepts three objects") def receiveClose(self): if self.state != 3: raise BananaError("MethodUnslicer requires three objects") if self.im_self is None: meth = getattr(self.im_class, self.im_func) # getattr gives us an unbound method return meth, None # TODO: late-available instances #if isinstance(self.im_self, NotKnown): # im = _InstanceMethod(self.im_name, self.im_self, self.im_class) # return im meth = self.im_class.__dict__[self.im_func] # whereas __dict__ gives us a function im = instancemethod(meth, self.im_self, self.im_class) return im, None class FunctionUnslicer(slicer.LeafUnslicer): opentype = ('function',) unslicerRegistry = UnsafeUnslicerRegistry finished = False def checkToken(self, typebyte, size): if typebyte not in (tokens.STRING, tokens.VOCAB): raise BananaError("FunctionUnslicer only accepts strings") def receiveChild(self, obj, ready_deferred=None): assert not isinstance(obj, Deferred) assert ready_deferred is None if self.finished: raise BananaError("FunctionUnslicer only accepts one string") self.finished = True # TODO: taste here! self.func = reflect.namedObject(obj) def receiveClose(self): if not self.finished: raise BananaError("FunctionUnslicer requires a string") return self.func, None # the root unslicer for storage is just like the regular one, but hands # received objects to the StorageBanana class StorageRootUnslicer(ScopedRootUnslicer): def receiveChild(self, obj, ready_deferred): self.protocol.receiveChild(obj, ready_deferred) # but the "unsafe" one has its own tables class UnsafeStorageRootUnslicer(StorageRootUnslicer): # This version tracks references for the entire lifetime of the # protocol. It is most appropriate for single-use purposes, such as a # replacement for Pickle. topRegistries = [slicer.UnslicerRegistry, slicer.BananaUnslicerRegistry, UnsafeUnslicerRegistry] openRegistries = [slicer.UnslicerRegistry, UnsafeUnslicerRegistry] class StorageBanana(banana.Banana): object = None violation = None disconnectReason = None slicerClass = StorageRootSlicer unslicerClass = StorageRootUnslicer def prepare(self): self.d = Deferred() return self.d def receiveChild(self, obj, ready_deferred): if ready_deferred: ready_deferred.addBoth(self.d.callback) self.d.addCallback(lambda res: obj) else: self.d.callback(obj) del self.d def receivedObject(self, obj): self.object = obj def sendError(self, msg): pass def reportViolation(self, why): self.violation = why def reportReceiveError(self, f): self.disconnectReason = f f.raiseException() class SerializerTransport: def __init__(self, sio): self.sio = sio def write(self, data): self.sio.write(data) def loseConnection(self, why="ignored"): pass def serialize(obj, outstream=None, root_class=StorageRootSlicer, banana=None): """Serialize an object graph into a sequence of bytes. Returns a Deferred that fires with the sequence of bytes.""" if banana: b = banana else: b = StorageBanana() b.slicerClass = root_class if outstream is None: sio = StringIO() else: sio = outstream b.transport = SerializerTransport(sio) b.connectionMade() d = b.send(obj) def _report_error(res): if b.disconnectReason: return b.disconnectReason if b.violation: return b.violation return res d.addCallback(_report_error) if outstream is None: d.addCallback(lambda res: sio.getvalue()) else: d.addCallback(lambda res: outstream) return d def unserialize(str_or_instream, banana=None, root_class=StorageRootUnslicer): """Unserialize a sequence of bytes back into an object graph.""" if banana: b = banana else: b = StorageBanana() b.unslicerClass = root_class b.connectionMade() d = b.prepare() # this will fire with the unserialized object if isinstance(str_or_instream, str): b.dataReceived(str_or_instream) else: raise RuntimeError("input streams not implemented yet") def _report_error(res): if b.disconnectReason: return b.disconnectReason if b.violation: return b.violation return res # return the unserialized object d.addCallback(_report_error) return d foolscap-0.13.1/src/foolscap/stringchain.py0000644000076500000240000001542212766553111021316 0ustar warnerstaff00000000000000import copy from collections import deque # Note: when changing this class, you should un-comment all the lines that say # "assert self._assert_invariants()". class StringChain(object): def __init__(self): self.d = deque() self.ignored = 0 self.tailignored = 0 self.len = 0 def append(self, s): """ Add s to the end of the chain. """ #assert self._assert_invariants() if not s: return # First trim off any ignored tail bytes. if self.tailignored: self.d[-1] = self.d[-1][:-self.tailignored] self.tailignored = 0 self.d.append(s) self.len += len(s) #assert self._assert_invariants() def appendleft(self, s): """ Add s to the beginning of the chain. """ #assert self._assert_invariants() if not s: return # First trim off any ignored bytes. if self.ignored: self.d[0] = self.d[0][self.ignored:] self.ignored = 0 self.d.appendleft(s) self.len += len(s) #assert self._assert_invariants() def __str__(self): """ Return the entire contents of this chain as a single string. (Obviously this requires copying all of the bytes, so don't do this unless you need to.) This has a side-effect of collecting all the bytes in this StringChain object into a single string which is stored in the first element of its internal deque. """ self._collapse() if self.d: return self.d[0] else: return '' def popleft_new_stringchain(self, bytes): """ Remove some of the leading bytes of the chain and return them as a new StringChain object. (Use str() on it if you want the bytes in a string, or call popleft() instead of popleft_new_stringchain().) """ #assert self._assert_invariants() if not bytes or not self.d: return self.__class__() assert bytes >= 0, bytes # We need to add at least this many bytes to the new StringChain. bytesleft = bytes + self.ignored n = self.__class__() n.ignored = self.ignored while bytesleft > 0 and self.d: s = self.d.popleft() self.len -= (len(s) - self.ignored) n.d.append(s) n.len += (len(s)-self.ignored) self.ignored = 0 bytesleft -= len(s) overrun = - bytesleft if overrun > 0: self.d.appendleft(s) self.len += overrun self.ignored = len(s) - overrun n.len -= overrun n.tailignored = overrun else: self.ignored = 0 # Either you got exactly how many you asked for, or you drained self entirely and you asked for more than you got. #assert (n.len == bytes) or ((not self.d) and (bytes > self.len)), (n.len, bytes, len(self.d)) #assert self._assert_invariants() #assert n._assert_invariants() return n def popleft(self, bytes): """ Remove some of the leading bytes of the chain and return them as a string. """ #assert self._assert_invariants() if not bytes or not self.d: return '' assert bytes >= 0, bytes # We need to add at least this many bytes to the result. bytesleft = bytes resstrs = [] s = self.d.popleft() if self.ignored: s = s[self.ignored:] self.ignored = 0 self.len -= len(s) resstrs.append(s) bytesleft -= len(s) while bytesleft > 0 and self.d: s = self.d.popleft() self.len -= len(s) resstrs.append(s) bytesleft -= len(s) overrun = - bytesleft if overrun > 0: self.d.appendleft(s) self.ignored = (len(s) - overrun) self.len += overrun resstrs[-1] = resstrs[-1][:-overrun] resstr = ''.join(resstrs) # Either you got exactly how many you asked for, or you drained self entirely and you asked for more than you got. #assert (len(resstr) == bytes) or ((not self.d) and (bytes > self.len)), (len(resstr), bytes, len(self.d), overrun) #assert self._assert_invariants() return resstr def __len__(self): #assert self._assert_invariants() return self.len def trim(self, bytes): """ Trim off some of the leading bytes. """ #assert self._assert_invariants() self.ignored += bytes self.len -= bytes while self.d and self.ignored >= len(self.d[0]): s = self.d.popleft() self.ignored -= len(s) if self.len < 0: self.len = 0 if not self.d: self.ignored = 0 #assert self._assert_invariants() def clear(self): """ Empty it out. """ #assert self._assert_invariants() self.d.clear() self.ignored = 0 self.tailignored = 0 self.len = 0 #assert self._assert_invariants() def copy(self): n = self.__class__() n.ignored = self.ignored n.tailignored = self.tailignored n.len = self.len n.d = copy.copy(self.d) #assert n._assert_invariants() return n def _assert_invariants(self): assert self.ignored >= 0, self.ignored assert self.tailignored >= 0, self.tailignored assert self.len >= 0, self.len assert (not self.d) or (self.d[0]), \ ("First element is required to be non-empty.", self.d and self.d[0]) assert (not self.d) or (self.ignored < len(self.d[0])), \ (self.ignored, self.d and len(self.d[0])) assert (not self.d) or (self.tailignored < len(self.d[-1])), \ (self.tailignored, self.d and len(self.d[-1])) assert self.ignored+self.len+self.tailignored == sum([len(x) for x in self.d]), \ (self.ignored, self.len, self.tailignored, sum([len(x) for x in self.d])) return True def _collapse(self): """ Concatenate all of the strings into one string and make that string be the only element of the chain. (Obviously this requires copying all of the bytes, so don't do this unless you need to.) """ #assert self._assert_invariants() # First trim off any leading ignored bytes. if self.ignored: self.d[0] = self.d[0][self.ignored:] self.ignored = 0 # Then any tail ignored bytes. if self.tailignored: self.d[-1] = self.d[-1][:-self.tailignored] self.tailignored = 0 if len(self.d) > 1: newstr = ''.join(self.d) self.d.clear() self.d.append(newstr) #assert self._assert_invariants() foolscap-0.13.1/src/foolscap/test/0000755000076500000240000000000013204747603017405 5ustar warnerstaff00000000000000foolscap-0.13.1/src/foolscap/test/__init__.py0000644000076500000240000000007512766553111021521 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test -*- """foolscap tests""" foolscap-0.13.1/src/foolscap/test/apphelper.py0000644000076500000240000000150312766553111021737 0ustar warnerstaff00000000000000 """ I am the command executed by test_appserver.py when it exercises the 'run-command' server. On a unix box, we'd use /bin/cat and /bin/dd ; this script lets the test work on windows too. """ import sys, os.path if sys.argv[1] == "cat": if not os.path.exists(sys.argv[2]): sys.stderr.write("cat: %s: No such file or directory\n" % sys.argv[2]) sys.exit(1) f = open(sys.argv[2], "rb") data = f.read() f.close() sys.stdout.write(data) sys.exit(0) if sys.argv[1] == "dd": assert sys.argv[2].startswith("of=") fn = sys.argv[2][3:] f = open(fn, "wb") data = sys.stdin.read() f.write(data) f.close() sys.stderr.write("0+1 records in\n") sys.stderr.write("0+1 records out\n") sys.stderr.write("%d bytes transferred in 42 seconds\n" % len(data)) sys.exit(0) foolscap-0.13.1/src/foolscap/test/bench_banana.py0000644000076500000240000000324612766553111022344 0ustar warnerstaff00000000000000import StringIO from foolscap import storage class TestTransport(StringIO.StringIO): disconnectReason = None def loseConnection(self): pass class B(object): def setup_huge_string(self, N): """ This is actually a test for acceptable performance, and it needs to be made more explicit, perhaps by being moved into a separate benchmarking suite instead of living in this test suite. """ self.banana = storage.StorageBanana() self.banana.slicerClass = storage.UnsafeStorageRootSlicer self.banana.unslicerClass = storage.UnsafeStorageRootUnslicer self.banana.transport = TestTransport() self.banana.connectionMade() d = self.banana.send("a"*N) d.addCallback(lambda res: self.banana.transport.getvalue()) def f(o): self._encoded_huge_string = o d.addCallback(f) reactor.runUntilCurrent() def bench_huge_string_decode(self, N): """ This is actually a test for acceptable performance, and it needs to be made more explicit, perhaps by being moved into a separate benchmarking suite instead of living in this test suite. """ o = self._encoded_huge_string # results = [] self.banana.prepare() # d.addCallback(results.append) CHOMP = 4096 for i in range(0, len(o), CHOMP): self.banana.dataReceived(o[i:i+CHOMP]) # print results import sys from twisted.internet import reactor from pyutil import benchutil b = B() for N in 10**3, 10**4, 10**5, 10**6, 10**7: print "%8d" % N, sys.stdout.flush() benchutil.rep_bench(b.bench_huge_string_decode, N, b.setup_huge_string) foolscap-0.13.1/src/foolscap/test/check-connections-client.py0000644000076500000240000000754312766553111024642 0ustar warnerstaff00000000000000#! /usr/bin/python # This is the client side of a manual test for the socks/tor # connection-handler code. To use it, first set up the server as described in # the other file, then copy the hostname, tubid, and .onion address into this # file: HOSTNAME = "foolscap.lothar.com" TUBID = "qy4aezcyd3mppt7arodl4mzaguls6m2o" ONION = "kwmjlhmn5runa4bv.onion" ONIONPORT = 16545 I2P = "???" I2PPORT = 0 LOCALPORT = 7006 # Then run 'check-connections-client.py tcp', then with 'socks', then with # 'tor'. import os, sys, time from twisted.internet import reactor from twisted.internet.defer import inlineCallbacks from twisted.internet.endpoints import HostnameEndpoint, clientFromString from foolscap.api import Referenceable, Tub tub = Tub() which = sys.argv[1] if len(sys.argv) > 1 else None if which == "tcp": furl = "pb://%s@tcp:%s:%d/calculator" % (TUBID, HOSTNAME, LOCALPORT) elif which == "socks": # "slogin -D 8013 HOSTNAME" starts a SOCKS server on localhost 8013, for # which connections will emerge from the other end. Check the server logs # to see the peer address of each addObserver call to verify that it is # coming from 127.0.0.1 rather than the client host. from foolscap.connections import socks h = socks.socks_endpoint(HostnameEndpoint(reactor, "localhost", 8013)) tub.removeAllConnectionHintHandlers() tub.addConnectionHintHandler("tcp", h) furl = "pb://%s@tcp:localhost:%d/calculator" % (TUBID, LOCALPORT) elif which in ("tor-default", "tor-socks", "tor-control", "tor-launch"): from foolscap.connections import tor if which == "tor-default": h = tor.default_socks() elif which == "tor-socks": h = tor.socks_port(int(sys.argv[2])) elif which == "tor-control": control_ep = clientFromString(reactor, sys.argv[2]) h = tor.control_endpoint(control_ep) elif which == "tor-launch": data_directory = None if len(sys.argv) > 2: data_directory = os.path.abspath(sys.argv[2]) h = tor.launch(data_directory) tub.removeAllConnectionHintHandlers() tub.addConnectionHintHandler("tor", h) furl = "pb://%s@tor:%s:%d/calculator" % (TUBID, ONION, ONIONPORT) elif which in ("i2p-default", "i2p-sam"): from foolscap.connections import i2p if which == "i2p-default": h = i2p.default(reactor) else: sam_ep = clientFromString(reactor, sys.argv[2]) h = i2p.sam_endpoint(sam_ep) tub.removeAllConnectionHintHandlers() tub.addConnectionHintHandler("i2p", h) furl = "pb://%s@i2p:%s:%d/calculator" % (TUBID, I2P, I2PPORT) else: print "run as 'check-connections-client.py [tcp|socks|tor-default|tor-socks|tor-control|tor-launch|i2p-default|i2p-sam]'" sys.exit(1) print "using %s: %s" % (which, furl) class Observer(Referenceable): def remote_event(self, msg): pass @inlineCallbacks def go(): tub.startService() start = time.time() rtts = [] remote = yield tub.getReference(furl) t_connect = time.time() - start o = Observer() start = time.time() yield remote.callRemote("addObserver", observer=o) rtts.append(time.time() - start) start = time.time() yield remote.callRemote("removeObserver", observer=o) rtts.append(time.time() - start) start = time.time() yield remote.callRemote("push", num=2) rtts.append(time.time() - start) start = time.time() yield remote.callRemote("push", num=3) rtts.append(time.time() - start) start = time.time() yield remote.callRemote("add") rtts.append(time.time() - start) start = time.time() number = yield remote.callRemote("pop") rtts.append(time.time() - start) print "the result is", number print "t_connect:", t_connect print "avg rtt:", sum(rtts) / len(rtts) d = go() def _oops(f): print "error", f d.addErrback(_oops) d.addCallback(lambda res: reactor.stop()) reactor.run() foolscap-0.13.1/src/foolscap/test/check-connections-server.py0000644000076500000240000000422512766553111024664 0ustar warnerstaff00000000000000#! /usr/bin/python # This is the server side of a manual test for the socks/tor # connection-handler code. On the server host, configure Tor to route a # hidden service to our port with something like: # # HiddenServiceDir /var/lib/tor/foolscap-calc-HS/ # HiddenServicePort 16545 127.0.0.1:7006 # # Then restart Tor, and look in /var/lib/tor/foolscap-calc-HS/hostname to # learn the .onion address that was allocated. Copy that into this file: ONION = "kwmjlhmn5runa4bv.onion" ONIONPORT = 16545 LOCALPORT = 7006 # Then launch this server with "twistd -y check-connections-server.py", and # copy our hostname (and the other values above) into # check-connections-client.py . Then run the client in tcp/socks/tor modes. from twisted.application import service from foolscap.api import Referenceable, Tub class Calculator(Referenceable): def __init__(self): self.stack = [] self.observers = [] def remote_addObserver(self, observer): self.observers.append(observer) print "observer is from", observer.getPeer() def log(self, msg): for o in self.observers: o.callRemote("event", msg=msg) def remote_removeObserver(self, observer): self.observers.remove(observer) def remote_push(self, num): self.log("push(%d)" % num) self.stack.append(num) def remote_add(self): self.log("add") arg1, arg2 = self.stack.pop(), self.stack.pop() self.stack.append(arg1 + arg2) def remote_subtract(self): self.log("subtract") arg1, arg2 = self.stack.pop(), self.stack.pop() self.stack.append(arg2 - arg1) def remote_pop(self): self.log("pop") return self.stack.pop() tub = Tub(certFile="tub.pem") lp = "tcp:%d" % LOCALPORT if 0: lp += ":interface=127.0.0.1" tub.listenOn(lp) tub.setLocation("tor:%s:%d" % (ONION, ONIONPORT)) url = tub.registerReference(Calculator(), "calculator") print "the object is available at:", url application = service.Application("check-connections-server") tub.setServiceParent(application) if __name__ == '__main__': raise RuntimeError("please run this as 'twistd -noy check-connections-server.py'") foolscap-0.13.1/src/foolscap/test/common.py0000644000076500000240000005005413204160675021251 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test.test_pb -*- import re, time from zope.interface import implements, implementsOnly, implementedBy, Interface from twisted.python import log from twisted.internet import defer, reactor, task, protocol from twisted.application import internet from twisted.trial import unittest from foolscap import broker, eventual, negotiate from foolscap.api import Tub, Referenceable, RemoteInterface, \ eventually, fireEventually, flushEventualQueue from foolscap.remoteinterface import getRemoteInterface, RemoteMethodSchema, \ UnconstrainedMethod from foolscap.schema import Any, SetOf, DictOf, ListOf, TupleOf, \ NumberConstraint, ByteStringConstraint, IntegerConstraint, \ UnicodeConstraint, ChoiceOf from foolscap.referenceable import TubRef from foolscap.util import allocate_tcp_port from twisted.python import failure from twisted.internet.main import CONNECTION_DONE def getRemoteInterfaceName(obj): i = getRemoteInterface(obj) return i.__remote_name__ class Loopback: # The transport's promise is that write() can be treated as a # synchronous, isolated function call: specifically, the Protocol's # dataReceived() and connectionLost() methods shall not be called during # a call to write(). connected = True def write(self, data): eventually(self._write, data) def _write(self, data): if not self.connected: return try: # isolate exceptions: if one occurred on a regular TCP transport, # they would hang up, so duplicate that here. self.peer.dataReceived(data) except: f = failure.Failure() log.err(f) print "Loopback.write exception:", f self.loseConnection(f) def loseConnection(self, why=failure.Failure(CONNECTION_DONE)): assert isinstance(why, failure.Failure), why if self.connected: self.connected = False # this one is slightly weird because 'why' is a Failure eventually(self._loseConnection, why) def _loseConnection(self, why): assert isinstance(why, failure.Failure), why self.protocol.connectionLost(why) self.peer.connectionLost(why) def flush(self): self.connected = False return fireEventually() def getPeer(self): return broker.LoopbackAddress() def getHost(self): return broker.LoopbackAddress() Digits = re.compile("\d*") MegaSchema1 = DictOf(str, ListOf(TupleOf(SetOf(int, maxLength=10, mutable=True), str, bool, int, long, float, None, UnicodeConstraint(), ByteStringConstraint(), Any(), NumberConstraint(), IntegerConstraint(), ByteStringConstraint(maxLength=100, minLength=90, regexp="\w+"), ByteStringConstraint(regexp=Digits), ), maxLength=20), maxKeys=5) # containers should convert their arguments into schemas MegaSchema2 = TupleOf(SetOf(int), ListOf(int), DictOf(int, str), ) MegaSchema3 = ListOf(TupleOf(int,int)) class RIHelper(RemoteInterface): def set(obj=Any()): return bool def set2(obj1=Any(), obj2=Any()): return bool def append(obj=Any()): return Any() def get(): return Any() def echo(obj=Any()): return Any() def defer(obj=Any()): return Any() def hang(): return Any() # test one of everything def megaschema(obj1=MegaSchema1, obj2=MegaSchema2): return None def mega3(obj1=MegaSchema3): return None def choice1(obj1=ChoiceOf(ByteStringConstraint(2000), int)): return None class HelperTarget(Referenceable): implements(RIHelper) d = None def __init__(self, name="unnamed"): self.name = name def __repr__(self): return "" % self.name def waitfor(self): self.d = defer.Deferred() return self.d def remote_set(self, obj): self.obj = obj if self.d: self.d.callback(obj) return True def remote_set2(self, obj1, obj2): self.obj1 = obj1 self.obj2 = obj2 return True def remote_append(self, obj): self.calls.append(obj) def remote_get(self): return self.obj def remote_echo(self, obj): self.obj = obj return obj def remote_defer(self, obj): return fireEventually(obj) def remote_hang(self): self.d = defer.Deferred() return self.d def remote_megaschema(self, obj1, obj2): self.obj1 = obj1 self.obj2 = obj2 return None def remote_mega3(self, obj): self.obj = obj return None def remote_choice1(self, obj): self.obj = obj return None class TimeoutError(Exception): pass class PollComplete(Exception): pass class PollMixin: def poll(self, check_f, pollinterval=0.01, timeout=None): # Return a Deferred, then call check_f periodically until it returns # True, at which point the Deferred will fire.. If check_f raises an # exception, the Deferred will errback. If the check_f does not # indicate success within timeout= seconds, the Deferred will # errback. If timeout=None, no timeout will be enforced, and the loop # will poll forever (or really until Trial times out). cutoff = None if timeout is not None: cutoff = time.time() + timeout lc = task.LoopingCall(self._poll, check_f, cutoff) d = lc.start(pollinterval) def _convert_done(f): f.trap(PollComplete) return None d.addErrback(_convert_done) return d def _poll(self, check_f, cutoff): if cutoff is not None and time.time() > cutoff: raise TimeoutError() if check_f(): raise PollComplete() class StallMixin: def stall(self, res, timeout): d = defer.Deferred() reactor.callLater(timeout, d.callback, res) return d class TargetMixin(PollMixin, StallMixin): def setUp(self): self.loopbacks = [] def setupBrokers(self): self.targetBroker = broker.Broker(TubRef("targetBroker")) self.callingBroker = broker.Broker(TubRef("callingBroker")) t1 = Loopback() t1.peer = self.callingBroker t1.protocol = self.targetBroker self.targetBroker.transport = t1 self.loopbacks.append(t1) t2 = Loopback() t2.peer = self.targetBroker t2.protocol = self.callingBroker self.callingBroker.transport = t2 self.loopbacks.append(t2) self.targetBroker.connectionMade() self.callingBroker.connectionMade() def tearDown(self): # returns a Deferred which fires when the Loopbacks are drained dl = [l.flush() for l in self.loopbacks] d = defer.DeferredList(dl) d.addCallback(flushEventualQueue) return d def setupTarget(self, target, txInterfaces=False): # txInterfaces controls what interfaces the sender uses # False: sender doesn't know about any interfaces # True: sender gets the actual interface list from the target # (list): sender uses an artificial interface list puid = target.processUniqueID() tracker = self.targetBroker.getTrackerForMyReference(puid, target) tracker.send() clid = tracker.clid if txInterfaces: iname = getRemoteInterfaceName(target) else: iname = None rtracker = self.callingBroker.getTrackerForYourReference(clid, iname) rr = rtracker.getRef() return rr, target class RIMyTarget(RemoteInterface): # method constraints can be declared directly: add1 = RemoteMethodSchema(_response=int, a=int, b=int) free = UnconstrainedMethod() # or through their function definitions: def add(a=int, b=int): return int #add = schema.callable(add) # the metaclass makes this unnecessary # but it could be used for adding options or something def join(a=str, b=str, c=int): return str def getName(): return str disputed = RemoteMethodSchema(_response=int, a=int) def fail(): return str # actually raises an exception def failstring(): return str # raises a string exception class RIMyTarget2(RemoteInterface): __remote_name__ = "RIMyTargetInterface2" sub = RemoteMethodSchema(_response=int, a=int, b=int) # For some tests, we want the two sides of the connection to disagree about # the contents of the RemoteInterface they are using. This is remarkably # difficult to accomplish within a single process. We do it by creating # something that behaves just barely enough like a RemoteInterface to work. class FakeTarget(dict): pass RIMyTarget3 = FakeTarget() RIMyTarget3.__remote_name__ = RIMyTarget.__remote_name__ RIMyTarget3['disputed'] = RemoteMethodSchema(_response=int, a=str) RIMyTarget3['disputed'].name = "disputed" RIMyTarget3['disputed'].interface = RIMyTarget3 RIMyTarget3['disputed2'] = RemoteMethodSchema(_response=str, a=int) RIMyTarget3['disputed2'].name = "disputed" RIMyTarget3['disputed2'].interface = RIMyTarget3 RIMyTarget3['sub'] = RemoteMethodSchema(_response=int, a=int, b=int) RIMyTarget3['sub'].name = "sub" RIMyTarget3['sub'].interface = RIMyTarget3 class Target(Referenceable): implements(RIMyTarget) def __init__(self, name=None): self.calls = [] self.name = name def getMethodSchema(self, methodname): return None def remote_add(self, a, b): self.calls.append((a,b)) return a+b remote_add1 = remote_add def remote_free(self, *args, **kwargs): self.calls.append((args, kwargs)) return "bird" def remote_getName(self): return self.name def remote_disputed(self, a): return 24 def remote_fail(self): raise ValueError("you asked me to fail") def remote_fail_remotely(self, target): return target.callRemote("fail") def remote_failstring(self): raise "string exceptions are annoying" class TargetWithoutInterfaces(Target): # undeclare the RIMyTarget interface implementsOnly(implementedBy(Referenceable)) class BrokenTarget(Referenceable): implements(RIMyTarget) def remote_add(self, a, b): return "error" class IFoo(Interface): # non-remote Interface pass class Foo(Referenceable): implements(IFoo) class RIDummy(RemoteInterface): pass class RITypes(RemoteInterface): def returns_none(work=bool): return None def takes_remoteinterface(a=RIDummy): return str def returns_remoteinterface(work=int): return RIDummy def takes_interface(a=IFoo): return str def returns_interface(work=bool): return IFoo class DummyTarget(Referenceable): implements(RIDummy) class TypesTarget(Referenceable): implements(RITypes) def remote_returns_none(self, work): if work: return None return "not None" def remote_takes_remoteinterface(self, a): # TODO: really, I want to just be able to say: # if RIDummy.providedBy(a): iface = a.tracker.interface if iface and iface == RIDummy: return "good" raise RuntimeError("my argument (%s) should provide RIDummy, " "but doesn't" % a) def remote_returns_remoteinterface(self, work): if work == 1: return DummyTarget() if work == -1: return TypesTarget() return 15 def remote_takes_interface(self, a): if IFoo.providedBy(a): return "good" raise RuntimeError("my argument (%s) should provide IFoo, but doesn't" % a) def remote_returns_interface(self, work): if work: return Foo() return "not implementor of IFoo" class ShouldFailMixin: def shouldFail(self, expected_failure, which, substring, callable, *args, **kwargs): assert substring is None or isinstance(substring, str) d = defer.maybeDeferred(callable, *args, **kwargs) def done(res): if isinstance(res, failure.Failure): if not res.check(expected_failure): self.fail("got failure %s, was expecting %s" % (res, expected_failure)) if substring: self.failUnless(substring in str(res), "%s: substring '%s' not in '%s'" % (which, substring, str(res))) # make the Failure available to a subsequent callback, but # keep it from triggering an errback return [res] else: self.fail("%s was supposed to raise %s, not get '%s'" % (which, expected_failure, res)) d.addBoth(done) return d tubid_low = "3hemthez7rvgvyhjx2n5kdj7mcyar3yt" certData_low = \ """-----BEGIN CERTIFICATE----- MIIBnjCCAQcCAgCEMA0GCSqGSIb3DQEBBAUAMBcxFTATBgNVBAMUDG5ld3BiX3Ro aW5neTAeFw0wNjExMjYxODUxMTBaFw0wNzExMjYxODUxMTBaMBcxFTATBgNVBAMU DG5ld3BiX3RoaW5neTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEA1DuK9NoF fiSreA8rVqYPAjNiUqFelAAYPgnJR92Jry1J/dPA3ieNcCazbjVeKUFjd6+C30XR APhajsAJFiJdnmgrtVILNrpZDC/vISKQoAmoT9hP/cMqFm8vmUG/+AXO76q63vfH UmabBVDNTlM8FJpbm9M26cFMrH45G840gA0CAwEAATANBgkqhkiG9w0BAQQFAAOB gQBCtjgBbF/s4w/16Y15lkTAO0xt8ZbtrvcsFPGTXeporonejnNaJ/aDbJt8Y6nY ypJ4+LTT3UQwwvqX5xEuJmFhmXGsghRGypbU7Zxw6QZRppBRqz8xMS+y82mMZRQp ezP+BiTvnoWXzDEP1233oYuELVgOVnHsj+rC017Ykfd7fw== -----END CERTIFICATE----- -----BEGIN RSA PRIVATE KEY----- MIICXQIBAAKBgQDUO4r02gV+JKt4DytWpg8CM2JSoV6UABg+CclH3YmvLUn908De J41wJrNuNV4pQWN3r4LfRdEA+FqOwAkWIl2eaCu1Ugs2ulkML+8hIpCgCahP2E/9 wyoWby+ZQb/4Bc7vqrre98dSZpsFUM1OUzwUmlub0zbpwUysfjkbzjSADQIDAQAB AoGBAIvxTykw8dpBt8cMyZjzGoZq93Rg74pLnbCap1x52iXmiRmUHWLfVcYT3tDW 4+X0NfBfjL5IvQ4UtTHXsqYjtvJfXWazYYa4INv5wKDBCd5a7s1YQ8R7mnhlBbRd nqZ6RpGuQbd3gTGZCkUdbHPSqdCPAjryH9mtWoQZIepcIcoJAkEA77gjO+MPID6v K6lf8SuFXHDOpaNOAiMlxVnmyQYQoF0PRVSpKOQf83An7R0S/jN3C7eZ6fPbZcyK SFVktHhYwwJBAOKlgndbSkVzkQCMcuErGZT1AxHNNHSaDo8X3C47UbP3nf60SkxI boqmpuPvEPUB9iPQdiNZGDU04+FUhe5Vtu8CQHDQHXS/hIzOMy2/BfG/Y4F/bSCy W7HRzKK1jlCoVAbEBL3B++HMieTMsV17Q0bx/WI8Q2jAZE3iFmm4Fi6APHUCQCMi 5Yb7cBg0QlaDb4vY0q51DXTFC0zIVVl5qXjBWXk8+hFygdIxqHF2RIkxlr9k/nOu 7aGtPkOBX5KfN+QrBaECQQCltPE9YjFoqPezfyvGZoWAKb8bWzo958U3uVBnCw2f Fs8AQDgI/9gOUXxXno51xQSdCnJLQJ8lThRUa6M7/F1B -----END RSA PRIVATE KEY----- """ tubid_high = "6cxxohyb5ysw6ftpwprbzffxrghbfopm" certData_high = \ """-----BEGIN CERTIFICATE----- MIIBnjCCAQcCAgCEMA0GCSqGSIb3DQEBBAUAMBcxFTATBgNVBAMUDG5ld3BiX3Ro aW5neTAeFw0wNjExMjYxODUxNDFaFw0wNzExMjYxODUxNDFaMBcxFTATBgNVBAMU DG5ld3BiX3RoaW5neTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEArfrebvt3 8FE3kKoscY2J/8A4J6CUUUiM7/gl00UvGvvjfdaWbsj4w0o8W2tE0X8Zce3dScSl D6qVXy6AEc4Flqs0q02w9uNzcdDY6LF3NiK0Lq+JP4OjJeImUBe8wUU0RQxqf/oA GhgHEZhTp6aAdxBXZFOVDloiW6iqrKH/thcCAwEAATANBgkqhkiG9w0BAQQFAAOB gQBXi+edp3iz07wxcRztvXtTAjY/9gUwlfa6qSTg/cGqbF0OPa+sISBOFRnnC8qM ENexlkpiiD4Oyj+UtO5g2CMz0E62cTJTqz6PfexnmKIGwYjq5wZ2tzOrB9AmAzLv TQQ9CdcKBXLd2GCToh8hBvjyyFwj+yTSbq+VKLMFkBY8Rg== -----END CERTIFICATE----- -----BEGIN RSA PRIVATE KEY----- MIICXgIBAAKBgQCt+t5u+3fwUTeQqixxjYn/wDgnoJRRSIzv+CXTRS8a++N91pZu yPjDSjxba0TRfxlx7d1JxKUPqpVfLoARzgWWqzSrTbD243Nx0NjosXc2IrQur4k/ g6Ml4iZQF7zBRTRFDGp/+gAaGAcRmFOnpoB3EFdkU5UOWiJbqKqsof+2FwIDAQAB AoGBAKrU3Vp+Y2u+Y+ARqKgrQai1tq36eAhEQ9dRgtqrYTCOyvcCIR5RCirAFvnx H1bSBUsgNBw+EZGLfzZBs5FICaUjBOQYBYzfxux6+jlGvdl7idfHs7zogyEYBqye 0VkwzZ0mVXM2ujOD/z/ANkdEn2fGj/VwAYDlfvlyNZMckHp5AkEA5sc1VG3snWmG lz4967MMzJ7XNpZcTvLEspjpH7hFbnXUHIQ4wPYOP7dhnVvKX1FiOQ8+zXVYDDGB SK1ABzpc+wJBAMD+imwAhHNBbOb3cPYzOz6XRZaetvep3GfE2wKr1HXP8wchNXWj Ijq6fJinwPlDugHaeNnfb+Dydd+YEiDTSJUCQDGCk2Jlotmyhfl0lPw4EYrkmO9R GsSlOKXIQFtZwSuNg9AKXdKn9y6cPQjxZF1GrHfpWWPixNz40e+xm4bxcnkCQQCs +zkspqYQ/CJVPpHkSnUem83GvAl5IKmp5Nr8oPD0i+fjixN0ljyW8RG+bhXcFaVC BgTuG4QW1ptqRs5w14+lAkEAuAisTPUDsoUczywyoBbcFo3SVpFPNeumEXrj4MD/ uP+TxgBi/hNYaR18mTbKD4mzVSjqyEeRC/emV3xUpUrdqg== -----END RSA PRIVATE KEY----- """ class BaseMixin(ShouldFailMixin): def setUp(self): self.connections = [] self.servers = [] self.services = [] def tearDown(self): for c in self.connections: if c.transport: c.transport.loseConnection() dl = [] for s in self.servers: dl.append(defer.maybeDeferred(s.stopListening)) for s in self.services: dl.append(defer.maybeDeferred(s.stopService)) d = defer.DeferredList(dl) d.addCallback(flushEventualQueue) return d def stall(self, res, timeout): d = defer.Deferred() reactor.callLater(timeout, d.callback, res) return d def insert_turns(self, res, count): d = eventual.fireEventually(res) for i in range(count-1): d.addCallback(eventual.fireEventually) return d def makeServer(self, options={}, listenerOptions={}): self.tub = tub = Tub(_test_options=options) tub.startService() self.services.append(tub) portnum = allocate_tcp_port() tub.listenOn("tcp:%d:interface=127.0.0.1" % portnum, _test_options=listenerOptions) tub.setLocation("127.0.0.1:%d" % portnum) self.target = Target() return tub.registerReference(self.target), portnum def makeSpecificServer(self, certData, negotiationClass=negotiate.Negotiation): self.tub = tub = Tub(certData=certData) tub.negotiationClass = negotiationClass tub.startService() self.services.append(tub) portnum = allocate_tcp_port() tub.listenOn("tcp:%d:interface=127.0.0.1" % portnum) tub.setLocation("127.0.0.1:%d" % portnum) self.target = Target() return tub.registerReference(self.target), portnum def createSpecificServer(self, certData, negotiationClass=negotiate.Negotiation): tub = Tub(certData=certData) tub.negotiationClass = negotiationClass tub.startService() self.services.append(tub) portnum = allocate_tcp_port() tub.listenOn("tcp:%d:interface=127.0.0.1" % portnum) tub.setLocation("127.0.0.1:%d" % portnum) target = Target() return tub, target, tub.registerReference(target), portnum def makeNullServer(self): f = protocol.Factory() f.protocol = protocol.Protocol # discards everything s = internet.TCPServer(0, f) s.startService() self.services.append(s) portnum = s._port.getHost().port return portnum def makeHTTPServer(self): try: from twisted.web import server, resource, static except ImportError: raise unittest.SkipTest('this test needs twisted.web') root = resource.Resource() root.putChild("", static.Data("hello\n", "text/plain")) s = internet.TCPServer(0, server.Site(root)) s.startService() self.services.append(s) portnum = s._port.getHost().port return portnum def connectClient(self, portnum): tub = Tub() tub.startService() self.services.append(tub) d = tub.getReference("pb://127.0.0.1:%d/hello" % portnum) return d class MakeTubsMixin: def makeTubs(self, numTubs, mangleLocation=None, start=True): self.services = [] self.tub_ports = [] for i in range(numTubs): t = Tub() if start: t.startService() self.services.append(t) portnum = allocate_tcp_port() self.tub_ports.append(portnum) t.listenOn("tcp:%d:interface=127.0.0.1" % portnum) location = "tcp:127.0.0.1:%d" % portnum if mangleLocation: location = mangleLocation(portnum) t.setLocation(location) return self.services foolscap-0.13.1/src/foolscap/test/run_trial.py0000644000076500000240000000111012766553111021750 0ustar warnerstaff00000000000000 # This is a tiny helper module, to let "python -m foolscap.test.run_trial # ARGS" does the same thing as running "trial ARGS" (unfortunately # twisted/scripts/trial.py does not have a '__name__=="__main__"' clause). # # This makes it easier to run trial under coverage from tox: # * "coverage run trial ARGS" is how you'd usually do it # * but "trial" must be the one in tox's virtualenv # * "coverage run `which trial` ARGS" works from a shell # * but tox doesn't use a shell # So use: # "coverage run -m foolscap.test.run_trial ARGS" from twisted.scripts.trial import run run() foolscap-0.13.1/src/foolscap/test/test__versions.py0000644000076500000240000000526712766553111023040 0ustar warnerstaff00000000000000 from twisted.trial import unittest import time import platform import twisted from twisted.internet import reactor from twisted.python import log import foolscap from foolscap.api import __version__ import OpenSSL def split_version(version_string): def maybe_int(s): try: return int(s) except ValueError: return s return tuple([maybe_int(piece) for piece in version_string.split(".")]) class Versions(unittest.TestCase): def test_required(self): ssl_ver = split_version(OpenSSL.__version__) tw_ver = split_version(twisted.__version__) # this is gross, but apps aren't supposed to care what sort of # reactor they're using. I use str() instead of isinstance(reactor, # twisted.internet.selectreactor.SelectReactor) because I want to # avoid importing the selectreactor when we aren't already using it. is_select = bool( "select" in str(reactor).lower() ) if ( (ssl_ver >= split_version("0.7")) and (tw_ver <= split_version("8.1.0")) and is_select ): # twisted 8.1.0 bad, 8.0.1 bad, 8.0.0 bad, I think 2.5.0 is too. # twisted 10.1.0 ok. print print "-------------" print "Warning: tests will fail (unclean reactor warnings)" print "when pyOpenSSL >= 0.7 is used in conjunction with" print "Twisted <= 8.1.0 . The workaround is to use the pollreactor" print "instead of the default selectreactor (trial -r poll)." print "This bug is fixed in Twisted trunk, and should appear" print "in the next release of Twisted." print " pyOpenSSL version:", OpenSSL.__version__ print " Twisted version:", twisted.__version__ print " reactor:", str(reactor) print "See http://foolscap.lothar.com/trac/ticket/62 for details." print print "Sleeping for 10 seconds to give you a chance to stop this" print "run and restart with -r poll..." print "-------------" # give them a chance to read it and re-run the tests with -r poll time.sleep(10) # but we don't flunk the test, that would be gratuitous def test_record(self): log.msg("Versions:") log.msg("foolscap-%s" % __version__) log.msg("twisted-%s" % twisted.__version__) log.msg("pyopenssl-%s" % OpenSSL.__version__) log.msg("python-%s" % platform.python_version()) log.msg("platform: %s" % platform.version()) def test_not_unicode(self): self.failUnlessEqual(type(foolscap.__version__), str) self.failUnlessEqual(type(__version__), str) foolscap-0.13.1/src/foolscap/test/test_appserver.py0000644000076500000240000010534212766553111023033 0ustar warnerstaff00000000000000 import os, sys, json from StringIO import StringIO from twisted.trial import unittest from twisted.internet import defer from twisted.application import service from foolscap.api import Tub, eventually from foolscap.appserver import cli, server, client from foolscap.test.common import ShouldFailMixin, StallMixin from foolscap.util import allocate_tcp_port orig_service_data = {"version": 1, "services": { "swiss1": {"relative_basedir": "1", "type": "type1", "args": ["args1a", "args1b"], "comment": None, }, "swiss2": {"relative_basedir": "2", "type": "type2", "args": ["args2a", "args2b"], "comment": "comment2", }, }} # copied+trimmed from the old-format appserver/cli.py def old_add_service(basedir, service_type, service_args, comment, swissnum): service_basedir = os.path.join(basedir, "services", swissnum) os.makedirs(service_basedir) f = open(os.path.join(service_basedir, "service_type"), "w") f.write(service_type + "\n") f.close() f = open(os.path.join(service_basedir, "service_args"), "w") f.write(repr(service_args) + "\n") f.close() if comment: f = open(os.path.join(service_basedir, "comment"), "w") f.write(comment + "\n") f.close() furl_prefix = open(os.path.join(basedir, "furl_prefix")).read().strip() furl = furl_prefix + swissnum return furl, service_basedir class ServiceData(unittest.TestCase): def test_parse_json(self): basedir = "appserver/ServiceData/parse_json" os.makedirs(basedir) f = open(os.path.join(basedir, "services.json"), "wb") json.dump(orig_service_data, f) f.close() data = server.load_service_data(basedir) self.failUnlessEqual(orig_service_data, data) def test_parse_files_and_upgrade(self): # create a structure with individual files, and make sure we parse it # correctly. Test the git-foolscap case with slashes in the swissnum. basedir = "appserver/ServiceData/parse_files" os.makedirs(basedir) J = os.path.join f = open(os.path.join(basedir, "furl_prefix"), "wb") f.write("prefix") f.close() old_add_service(basedir, "type1", ("args1a", "args1b"), None, "swiss1") old_add_service(basedir, "type2", ("args2a", "args2b"), "comment2", "swiss2") old_add_service(basedir, "type3", ("args3a", "args3b"), "comment3", "swiss3/3") data = server.load_service_data(basedir) expected = {"version": 1, "services": { "swiss1": {"relative_basedir": J("services","swiss1"), "type": "type1", "args": ["args1a", "args1b"], "comment": None, }, "swiss2": {"relative_basedir": J("services","swiss2"), "type": "type2", "args": ["args2a", "args2b"], "comment": "comment2", }, J("swiss3","3"): {"relative_basedir": J("services","swiss3","3"), "type": "type3", "args": ["args3a", "args3b"], "comment": "comment3", }, }} self.failUnlessEqual(data, expected) s4 = {"relative_basedir": J("services","4"), "type": "type4", "args": ["args4a", "args4b"], "comment": "comment4", } data["services"]["swiss4"] = s4 server.save_service_data(basedir, data) # this upgrades to JSON data2 = server.load_service_data(basedir) # reads JSON, not files expected["services"]["swiss4"] = s4 self.failUnlessEqual(data2, expected) def test_bad_version(self): basedir = "appserver/ServiceData/bad_version" os.makedirs(basedir) orig = {"version": 99} f = open(os.path.join(basedir, "services.json"), "wb") json.dump(orig, f) f.close() e = self.failUnlessRaises(server.UnknownVersion, server.load_service_data, basedir) self.failUnlessIn("unable to handle version 99", str(e)) def test_save(self): basedir = "appserver/ServiceData/save" os.makedirs(basedir) server.save_service_data(basedir, orig_service_data) data = server.load_service_data(basedir) self.failUnlessEqual(orig_service_data, data) class CLI(unittest.TestCase): def run_cli(self, *args): argv = ["flappserver"] + list(args) d = defer.maybeDeferred(cli.run_flappserver, argv=argv, run_by_human=False) return d # fires with (rc,out,err) def test_create(self): basedir = "appserver/CLI/create" os.makedirs(basedir) serverdir = os.path.join(basedir, "fl") d = self.run_cli("create", "--location", "localhost:1234", serverdir) def _check((rc,out,err)): self.failUnlessEqual(rc, 0) self.failUnless(os.path.isdir(serverdir)) # check that the directory is group/world-inaccessible, even on # windows where those concepts are pretty fuzzy. Do this by # making sure the mode doesn't change when we chmod it again. mode1 = os.stat(serverdir).st_mode os.chmod(serverdir, 0700) mode2 = os.stat(serverdir).st_mode self.failUnlessEqual("%o" % mode1, "%o" % mode2) d.addCallback(_check) return d def test_create_no_clobber_dir(self): basedir = "appserver/CLI/create_no_clobber_dir" os.makedirs(basedir) serverdir = os.path.join(basedir, "fl") os.mkdir(serverdir) d = self.run_cli("create", "--location", "localhost:3116", serverdir) def _check((rc,out,err)): self.failUnlessEqual(rc, 1) self.failUnlessIn("Refusing to touch pre-existing directory", err) self.failIf(os.path.exists(os.path.join(serverdir, "port"))) self.failIf(os.path.exists(os.path.join(serverdir, "services"))) d.addCallback(_check) return d def test_create2(self): basedir = "appserver/CLI/create2" os.makedirs(basedir) serverdir = os.path.join(basedir, "fl") portnum = allocate_tcp_port() d = self.run_cli("create", "--location", "localhost:%d" % portnum, "--port", "tcp:%d" % portnum, "--umask", "022", serverdir) def _check((rc,out,err)): self.failUnlessEqual(rc, 0) self.failUnless(os.path.isdir(serverdir)) got_port = open(os.path.join(serverdir, "port"), "r").read().strip() self.failUnlessEqual(got_port, "tcp:%d" % portnum) prefix = open(os.path.join(serverdir, "furl_prefix"), "r").read().strip() self.failUnless(prefix.endswith(":%d/" % portnum), prefix) umask = open(os.path.join(serverdir, "umask")).read().strip() self.failUnlessEqual(umask, "0022") d.addCallback(_check) return d def test_create3(self): basedir = "appserver/CLI/create3" os.makedirs(basedir) serverdir = os.path.join(basedir, "fl") d = self.run_cli("create", "--location", "proxy.example.com:12345", serverdir) def _check((rc,out,err)): self.failUnlessEqual(rc, 0) self.failUnless(os.path.isdir(serverdir)) # pick an arbitrary port, but FURLs should reference the proxy prefix = open(os.path.join(serverdir, "furl_prefix"), "r").read().strip() self.failUnless(prefix.endswith("@proxy.example.com:12345/"), prefix) d.addCallback(_check) return d def test_add(self): basedir = "appserver/CLI/add" os.makedirs(basedir) serverdir = os.path.join(basedir, "fl") incomingdir = os.path.join(basedir, "incoming") os.mkdir(incomingdir) d = self.run_cli("create", "--location", "localhost:3116", serverdir) def _check((rc,out,err)): self.failUnlessEqual(rc, 0) self.failUnless(os.path.isdir(serverdir)) d.addCallback(_check) d.addCallback(lambda ign: self.run_cli("add", serverdir, "upload-file", incomingdir)) def _check_add((rc,out,err)): self.failUnlessEqual(rc, 0) lines = out.splitlines() self.failUnless(lines[0].startswith("Service added in ")) servicedir = lines[0].split()[-1] self.failUnless(lines[1].startswith("FURL is pb://")) furl = lines[1].split()[-1] swiss = furl[furl.rfind("/")+1:] data = server.load_service_data(serverdir) servicedir2 = os.path.join(serverdir, data["services"][swiss]["relative_basedir"]) self.failUnlessEqual(os.path.abspath(servicedir), os.path.abspath(servicedir2)) self.failUnlessEqual(data["services"][swiss]["comment"], None) d.addCallback(_check_add) return d def test_add_service(self): basedir = "appserver/CLI/add_service" os.makedirs(basedir) serverdir = os.path.join(basedir, "fl") incomingdir = os.path.join(basedir, "incoming") os.mkdir(incomingdir) d = self.run_cli("create", "--location", "localhost:3116", serverdir) def _check((rc,out,err)): self.failUnlessEqual(rc, 0) self.failUnless(os.path.isdir(serverdir)) d.addCallback(_check) def _check_add(ign): furl1,servicedir1a = cli.add_service(serverdir, "upload-file", (incomingdir,), None) self.failUnless(os.path.isdir(servicedir1a)) asd1 = os.path.abspath(servicedir1a) self.failUnless(asd1.startswith(os.path.abspath(basedir))) swiss1 = furl1[furl1.rfind("/")+1:] data = server.load_service_data(serverdir) servicedir1b = os.path.join(serverdir, data["services"][swiss1]["relative_basedir"]) self.failUnlessEqual(os.path.abspath(servicedir1a), os.path.abspath(servicedir1b)) # add a second service, to make sure the "find the next-highest # available servicedir" logic works from both empty and non-empty # starting points furl2,servicedir2a = cli.add_service(serverdir, "run-command", ("dummy",), None) self.failUnless(os.path.isdir(servicedir2a)) asd2 = os.path.abspath(servicedir2a) self.failUnless(asd2.startswith(os.path.abspath(basedir))) swiss2 = furl2[furl2.rfind("/")+1:] data = server.load_service_data(serverdir) servicedir2b = os.path.join(serverdir, data["services"][swiss2]["relative_basedir"]) self.failUnlessEqual(os.path.abspath(servicedir2a), os.path.abspath(servicedir2b)) d.addCallback(_check_add) return d def test_add_comment(self): basedir = "appserver/CLI/add_comment" os.makedirs(basedir) serverdir = os.path.join(basedir, "fl") incomingdir = os.path.join(basedir, "incoming") os.mkdir(incomingdir) d = self.run_cli("create", "--location", "localhost:3116", serverdir) def _check((rc,out,err)): self.failUnlessEqual(rc, 0) self.failUnless(os.path.isdir(serverdir)) d.addCallback(_check) d.addCallback(lambda ign: self.run_cli("add", "--comment", "commentary here", serverdir, "upload-file", incomingdir)) def _check_add((rc,out,err)): self.failUnlessEqual(rc, 0) lines = out.splitlines() self.failUnless(lines[0].startswith("Service added in ")) servicedir = lines[0].split()[-1] self.failUnless(lines[1].startswith("FURL is pb://")) furl = lines[1].split()[-1] swiss = furl[furl.rfind("/")+1:] data = server.load_service_data(serverdir) servicedir2 = os.path.join(serverdir, data["services"][swiss]["relative_basedir"]) self.failUnlessEqual(os.path.abspath(servicedir), os.path.abspath(servicedir2)) self.failUnlessEqual(data["services"][swiss]["comment"], "commentary here") d.addCallback(_check_add) return d def test_add_badargs(self): basedir = "appserver/CLI/add_badargs" os.makedirs(basedir) serverdir = os.path.join(basedir, "fl") servicesdir = os.path.join(serverdir, "services") incomingdir = os.path.join(basedir, "incoming") os.mkdir(incomingdir) d = self.run_cli("create", "--location", "localhost:3116", serverdir) def _check((rc,out,err)): self.failUnlessEqual(rc, 0) self.failUnless(os.path.isdir(serverdir)) d.addCallback(_check) d.addCallback(lambda ign: self.run_cli("add", serverdir, "upload-file", # missing targetdir )) def _check_add((rc,out,err)): self.failIfEqual(rc, 0) self.failUnlessIn("Error", err) self.failUnlessIn("Wrong number of arguments", err) self.failUnlessEqual(os.listdir(servicesdir), []) d.addCallback(_check_add) d.addCallback(lambda ign: self.run_cli("add", serverdir, "upload-file", "nonexistent-targetdir", )) def _check_add2((rc,out,err)): self.failIfEqual(rc, 0) self.failUnlessIn("Error", err) self.failUnlessIn("targetdir ", err) self.failUnlessIn(" must already exist", err) self.failUnlessEqual(os.listdir(servicesdir), []) d.addCallback(_check_add2) return d def test_list(self): basedir = "appserver/CLI/list" os.makedirs(basedir) serverdir = os.path.join(basedir, "fl") incomingdir = os.path.join(basedir, "incoming") os.mkdir(incomingdir) d = self.run_cli("create", "--location", "localhost:3116", serverdir) def _check((rc,out,err)): self.failUnlessEqual(rc, 0) self.failUnless(os.path.isdir(serverdir)) d.addCallback(_check) d.addCallback(lambda ign: self.run_cli("add", serverdir, "upload-file", incomingdir)) def _check_add((rc,out,err)): self.failUnlessEqual(rc, 0) d.addCallback(_check_add) def _check_list_services(ign): services = cli.list_services(serverdir) self.failUnlessEqual(len(services), 1) s = services[0] self.failUnlessEqual(s.service_type, "upload-file") self.failUnlessEqual(s.service_args, [incomingdir] ) d.addCallback(_check_list_services) d.addCallback(lambda ign: self.run_cli("list", serverdir)) def _check_list((rc,out,err)): self.failUnlessEqual(rc, 0) s = cli.list_services(serverdir)[0] lines = out.splitlines() self.failUnlessEqual(lines[0], "") self.failUnlessEqual(lines[1], s.swissnum+":") self.failUnlessEqual(lines[2], " upload-file %s" % incomingdir) self.failUnlessEqual(lines[3], " " + s.furl) self.failUnlessEqual(lines[4], " " + s.service_basedir) d.addCallback(_check_list) return d def test_list_comment(self): basedir = "appserver/CLI/list_comment" os.makedirs(basedir) serverdir = os.path.join(basedir, "fl") incomingdir = os.path.join(basedir, "incoming") os.mkdir(incomingdir) d = self.run_cli("create", "--location", "localhost:3116", serverdir) def _check((rc,out,err)): self.failUnlessEqual(rc, 0) self.failUnless(os.path.isdir(serverdir)) d.addCallback(_check) d.addCallback(lambda ign: self.run_cli("add", "--comment", "commentary here", serverdir, "upload-file", incomingdir)) def _check_add((rc,out,err)): self.failUnlessEqual(rc, 0) d.addCallback(_check_add) d.addCallback(lambda ign: self.run_cli("list", serverdir)) def _check_list((rc,out,err)): self.failUnlessEqual(rc, 0) s = cli.list_services(serverdir)[0] lines = out.splitlines() self.failUnlessEqual(lines[0], "") self.failUnlessEqual(lines[1], s.swissnum+":") self.failUnlessEqual(lines[2], " upload-file %s" % incomingdir) self.failUnlessEqual(lines[3], " # commentary here") self.failUnlessEqual(lines[4], " " + s.furl) self.failUnlessEqual(lines[5], " " + s.service_basedir) d.addCallback(_check_list) return d class Server(unittest.TestCase, ShouldFailMixin): def setUp(self): self.s = service.MultiService() self.s.startService() def tearDown(self): return self.s.stopService() def run_cli(self, *args): argv = ["flappserver"] + list(args) d = defer.maybeDeferred(cli.run_flappserver, argv=argv, run_by_human=False) return d # fires with (rc,out,err) def test_run(self): basedir = "appserver/Server/run" os.makedirs(basedir) serverdir = os.path.join(basedir, "fl") incomingdir = os.path.join(basedir, "incoming") os.mkdir(incomingdir) self.tub = Tub() self.tub.setServiceParent(self.s) portnum = allocate_tcp_port() d = self.run_cli("create", "--location", "localhost:%d" % portnum, "--port", "tcp:%d" % portnum, serverdir) def _check((rc,out,err)): self.failUnlessEqual(rc, 0) self.failUnless(os.path.isdir(serverdir)) d.addCallback(_check) d.addCallback(lambda ign: self.run_cli("add", serverdir, "upload-file", incomingdir)) def _check_add((rc,out,err)): self.failUnlessEqual(rc, 0) lines = out.splitlines() self.failUnless(lines[1].startswith("FURL is pb://")) self.furl = lines[1].split()[-1] d.addCallback(_check_add) stdout = StringIO() def _start_server(ign): ap = server.AppServer(serverdir, stdout) ap.setServiceParent(self.s) d.addCallback(_start_server) # make sure the server can actually instantiate a service d.addCallback(lambda _ign: self.tub.getReference(self.furl)) def _got_rref(rref): # great! pass d.addCallback(_got_rref) d.addCallback(lambda ign: self.shouldFail(KeyError, "getReference(bogus)", "unable to find reference for name ", self.tub.getReference, self.furl+".bogus")) return d class Upload(unittest.TestCase, ShouldFailMixin): def setUp(self): self.s = service.MultiService() self.s.startService() def tearDown(self): return self.s.stopService() def run_cli(self, *args): argv = ["flappserver"] + list(args) d = defer.maybeDeferred(cli.run_flappserver, argv=argv, run_by_human=False) return d # fires with (rc,out,err) def run_client(self, *args): argv = ["flappclient"] + list(args) d = defer.maybeDeferred(client.run_flappclient, argv=argv, run_by_human=False) return d # fires with (rc,out,err) def test_run(self): basedir = "appserver/Upload/run" os.makedirs(basedir) serverdir = os.path.join(basedir, "fl") incomingdir = os.path.join(basedir, "incoming") os.mkdir(incomingdir) furlfile = os.path.join(basedir, "furlfile") portnum = allocate_tcp_port() d = self.run_cli("create", "--location", "localhost:%d" % portnum, "--port", "tcp:%d" % portnum, serverdir) def _check((rc,out,err)): self.failUnlessEqual(rc, 0) self.failUnless(os.path.isdir(serverdir)) d.addCallback(_check) d.addCallback(lambda ign: self.run_cli("add", serverdir, "upload-file", incomingdir)) def _check_add((rc,out,err)): self.failUnlessEqual(rc, 0) lines = out.splitlines() self.failUnless(lines[1].startswith("FURL is pb://")) self.furl = lines[1].split()[-1] f = open(furlfile,"w") f.write("\n") # it should ignore blank lines f.write("# it should ignore comments like this\n") f.write(self.furl+"\n") f.write("# and it should only pay attention to the first FURL\n") f.write(self.furl+".bogus\n") f.close() d.addCallback(_check_add) stdout = StringIO() def _start_server(ign): ap = server.AppServer(serverdir, stdout) ap.setServiceParent(self.s) d.addCallback(_start_server) sourcefile = os.path.join(basedir, "foo.txt") f = open(sourcefile, "wb") DATA = "This is some source text.\n" f.write(DATA) f.close() d.addCallback(lambda _ign: self.run_client("--furl", self.furl, "upload-file", sourcefile)) def _check_client((rc,out,err)): self.failUnlessEqual(rc, 0) self.failUnlessEqual(out.strip(), "foo.txt: uploaded") self.failUnlessEqual(err.strip(), "") fn = os.path.join(incomingdir, "foo.txt") self.failUnless(os.path.exists(fn)) contents = open(fn,"rb").read() self.failUnlessEqual(contents, DATA) d.addCallback(_check_client) sourcefile2 = os.path.join(basedir, "bar.txt") f = open(sourcefile2, "wb") DATA2 = "This is also some source text.\n" f.write(DATA2) f.close() d.addCallback(lambda _ign: self.run_client("--furlfile", furlfile, "upload-file", sourcefile2)) def _check_client2((rc,out,err)): self.failUnlessEqual(rc, 0) self.failUnlessEqual(out.strip(), "bar.txt: uploaded") self.failUnlessEqual(err.strip(), "") fn = os.path.join(incomingdir, "bar.txt") self.failUnless(os.path.exists(fn)) contents = open(fn,"rb").read() self.failUnlessEqual(contents, DATA2) d.addCallback(_check_client2) empty_furlfile = furlfile + ".empty" open(empty_furlfile, "wb").close() d.addCallback(lambda _ign: self.run_client("--furlfile", empty_furlfile, "upload-file", sourcefile2)) def _check_client3((rc,out,err)): self.failIfEqual(rc, 0) self.failUnlessIn("must provide --furl or --furlfile", err.strip()) d.addCallback(_check_client3) sourcefile3 = os.path.join(basedir, "file3.txt") f = open(sourcefile3, "wb") DATA3 = "file number 3\n" f.write(DATA3) f.close() sourcefile4 = os.path.join(basedir, "file4.txt") f = open(sourcefile4, "wb") DATA4 = "file number 4\n" f.write(DATA4) f.close() sourcefile5 = os.path.join(basedir, "file5.txt") f = open(sourcefile5, "wb") DATA5 = "file number 5\n" f.write(DATA5) f.close() d.addCallback(lambda _ign: self.run_client("--furl", self.furl, "upload-file", sourcefile3, sourcefile4, sourcefile5)) def _check_client4((rc,out,err)): self.failUnlessEqual(rc, 0) self.failUnlessIn("file3.txt: uploaded", out) self.failUnlessIn("file4.txt: uploaded", out) self.failUnlessIn("file5.txt: uploaded", out) self.failUnlessEqual(err.strip(), "") fn = os.path.join(incomingdir, "file3.txt") self.failUnless(os.path.exists(fn)) contents = open(fn,"rb").read() self.failUnlessEqual(contents, DATA3) fn = os.path.join(incomingdir, "file4.txt") self.failUnless(os.path.exists(fn)) contents = open(fn,"rb").read() self.failUnlessEqual(contents, DATA4) fn = os.path.join(incomingdir, "file5.txt") self.failUnless(os.path.exists(fn)) contents = open(fn,"rb").read() self.failUnlessEqual(contents, DATA5) d.addCallback(_check_client4) return d class Client(unittest.TestCase): def run_client(self, *args): argv = ["flappclient"] + list(args) d = defer.maybeDeferred(client.run_flappclient, argv=argv, run_by_human=False) return d # fires with (rc,out,err) def test_no_command(self): d = self.run_client() def _check_client1((rc,out,err)): self.failIfEqual(rc, 0) self.failUnlessIn("must provide --furl or --furlfile", err) d.addCallback(_check_client1) d.addCallback(lambda _ign: self.run_client("--furl", "foo")) def _check_client2((rc,out,err)): self.failIfEqual(rc, 0) self.failUnlessIn("must specify a command", err) d.addCallback(_check_client2) return d def test_help(self): d = self.run_client("--help") def _check_client((rc,out,err)): self.failUnlessEqual(rc, 0) self.failUnlessIn("Usage: flappclient [--furl=|--furlfile=] ", out) self.failUnlessEqual("", err.strip()) d.addCallback(_check_client) return d def test_version(self): d = self.run_client("--version") def _check_client((rc,out,err)): self.failUnlessEqual(rc, 0) self.failUnlessIn("Foolscap version:", out) self.failUnlessEqual("", err.strip()) d.addCallback(_check_client) return d class RunCommand(unittest.TestCase, StallMixin): def setUp(self): self.s = service.MultiService() self.s.startService() def tearDown(self): return self.s.stopService() def run_cli(self, *args): argv = ["flappserver"] + list(args) d = defer.maybeDeferred(cli.run_flappserver, argv=argv, run_by_human=False) return d # fires with (rc,out,err) def run_client(self, *args): argv = ["flappclient"] + list(args) d = defer.maybeDeferred(client.run_flappclient, argv=argv, run_by_human=False, stdio=None) return d # fires with (rc,out,err) def run_client_with_stdin(self, stdin, *args): argv = ["flappclient"] + list(args) def my_stdio(proto): eventually(proto.connectionMade) eventually(proto.dataReceived, stdin) eventually(proto.connectionLost, None) d = defer.maybeDeferred(client.run_flappclient, argv=argv, run_by_human=False, stdio=my_stdio) return d # fires with (rc,out,err) def add(self, serverdir, *args): d = self.run_cli("add", serverdir, *args) def _get_furl((rc,out,err)): self.failUnlessEqual(rc, 0) lines = out.splitlines() self.failUnless(lines[1].startswith("FURL is pb://")) furl = lines[1].split()[-1] return furl d.addCallback(_get_furl) return d def stash_furl(self, furl, which): self.furls[which] = furl def test_run(self): basedir = "appserver/RunCommand/run" os.makedirs(basedir) serverdir = os.path.join(basedir, "fl") incomingdir = os.path.join(basedir, "incoming") os.mkdir(incomingdir) self.furls = {} portnum = allocate_tcp_port() d = self.run_cli("create", "--location", "localhost:%d" % portnum, "--port", "tcp:%d" % portnum, serverdir) def _check((rc,out,err)): self.failUnlessEqual(rc, 0) self.failUnless(os.path.isdir(serverdir)) d.addCallback(_check) targetfile = os.path.join(incomingdir, "foo.txt") DATA = "Contents of foo.txt.\n" def _populate_foo(ign): f = open(targetfile, "wb") f.write(DATA) f.close() d.addCallback(_populate_foo) helper = os.path.join(os.path.dirname(__file__), "apphelper.py") d.addCallback(lambda ign: self.add(serverdir, "run-command", "--no-log-stdin", "--log-stdout", "--no-log-stderr", incomingdir, sys.executable, helper, "cat", "foo.txt")) d.addCallback(self.stash_furl, 0) stdout = StringIO() def _start_server(ign): ap = server.AppServer(serverdir, stdout) ap.setServiceParent(self.s) d.addCallback(_start_server) d.addCallback(lambda _ign: self.run_client("--furl", self.furls[0], "run-command")) def _check_client((rc,out,err)): self.failUnlessEqual(rc, 0) self.failUnlessEqual(out.strip(), DATA.strip()) self.failUnlessEqual(err.strip(), "") d.addCallback(_check_client) def _delete_foo(ign): os.unlink(targetfile) d.addCallback(_delete_foo) d.addCallback(lambda _ign: self.run_client("--furl", self.furls[0], "run-command")) def _check_client2((rc,out,err)): self.failIfEqual(rc, 0) self.failUnlessEqual(out, "") self.failUnlessEqual(err.strip(), "cat: foo.txt: No such file or directory") d.addCallback(_check_client2) d.addCallback(lambda ign: self.add(serverdir, "run-command", "--accept-stdin", "--log-stdin", "--no-log-stdout", "--log-stderr", incomingdir, sys.executable, helper, "dd", "of=bar.txt")) d.addCallback(self.stash_furl, 1) barfile = os.path.join(incomingdir, "bar.txt") DATA2 = "Pass this\ninto stdin\n" d.addCallback(lambda _ign: self.run_client_with_stdin(DATA2, "--furl", self.furls[1], "run-command")) def _check_client3((rc,out,err)): self.failUnlessEqual(rc, 0) bardata = open(barfile,"rb").read() self.failUnlessEqual(bardata, DATA2) # we use a script instead of the real dd; we know how it behaves self.failUnlessEqual(out, "") self.failUnlessIn("records in", err.strip()) d.addCallback(_check_client3) # exercise some more options d.addCallback(lambda ign: self.add(serverdir, "run-command", "--no-stdin", "--send-stdout", "--no-stderr", incomingdir, sys.executable, helper, "cat", "foo.txt")) d.addCallback(self.stash_furl, 2) d.addCallback(lambda ign: self.add(serverdir, "run-command", "--no-stdin", "--no-stdout", "--send-stderr", incomingdir, sys.executable, helper, "cat", "foo.txt")) d.addCallback(self.stash_furl, 3) d.addCallback(_populate_foo) d.addCallback(lambda _ign: self.run_client("--furl", self.furls[2], "run-command")) def _check_client4((rc,out,err)): self.failUnlessEqual(rc, 0) self.failUnlessEqual(out.strip(), DATA.strip()) self.failUnlessEqual(err, "") d.addCallback(_check_client4) d.addCallback(lambda _ign: self.run_client("--furl", self.furls[3], "run-command")) def _check_client5((rc,out,err)): self.failUnlessEqual(rc, 0) self.failUnlessEqual(out, "") # --no-stdout self.failUnlessEqual(err, "") d.addCallback(_check_client5) d.addCallback(_delete_foo) d.addCallback(lambda _ign: self.run_client("--furl", self.furls[2], "run-command")) def _check_client6((rc,out,err)): self.failIfEqual(rc, 0) self.failUnlessEqual(out, "") self.failUnlessEqual(err, "") # --no-stderr d.addCallback(_check_client6) d.addCallback(lambda _ign: self.run_client("--furl", self.furls[3], "run-command")) def _check_client7((rc,out,err)): self.failIfEqual(rc, 0) self.failUnlessEqual(out, "") # --no-stdout self.failUnlessEqual(err.strip(), "cat: foo.txt: No such file or directory") d.addCallback(_check_client7) return d foolscap-0.13.1/src/foolscap/test/test_banana.py0000644000076500000240000022156212766553111022247 0ustar warnerstaff00000000000000 from twisted.trial import unittest from twisted.python import reflect from twisted.python.failure import Failure from twisted.python.components import registerAdapter from twisted.internet import defer from foolscap.tokens import ISlicer, Violation, BananaError from foolscap.tokens import BananaFailure, tokenNames, \ OPEN, CLOSE, ABORT, INT, LONGINT, NEG, LONGNEG, FLOAT, STRING from foolscap import slicer, schema, storage, banana, vocab from foolscap.eventual import fireEventually, flushEventualQueue from foolscap.slicers.allslicers import RootSlicer, DictUnslicer, TupleUnslicer from foolscap.constraint import IConstraint from foolscap.banana import int2b128, long_to_bytes import StringIO import struct from decimal import Decimal #log.startLogging(sys.stderr) # some utility functions to manually assemble bytestreams def bOPEN(opentype, count): assert count < 128 return chr(count) + "\x88" + chr(len(opentype)) + "\x82" + opentype def bCLOSE(count): assert count < 128 return chr(count) + "\x89" def bINT(num): if num >=0: assert num < 128 return chr(num) + "\x81" num = -num assert num < 128 return chr(num) + "\x83" def bSTR(str): assert len(str) < 128 return chr(len(str)) + "\x82" + str def bERROR(str): assert len(str) < 128 return chr(len(str)) + "\x8d" + str def bABORT(count): assert count < 128 return chr(count) + "\x8A" # DecodeTest (24): turns tokens into objects, tests objects and UFs # EncodeTest (13): turns objects/instance into tokens, tests tokens # FailedInstanceTests (2): 1:turn instances into tokens and fail, 2:reverse # ByteStream (3): turn object into bytestream, test bytestream # InboundByteStream (14): turn bytestream into object, check object # with or without constraints # ThereAndBackAgain (20): encode then decode object, check object # VocabTest1 (2): test setOutgoingVocabulary and an inbound Vocab sequence # VocabTest2 (1): send object, test bytestream w/vocab-encoding # Sliceable (2): turn instance into tokens (with ISliceable, test tokens def tOPEN(count): return ("OPEN", count) def tCLOSE(count): return ("CLOSE", count) tABORT = ("ABORT",) class TokenBanana(banana.Banana): """this Banana formats tokens as strings, numbers, and ('OPEN',) tuples instead of bytes. Used for testing purposes.""" def sendOpen(self): openID = self.openCount self.openCount += 1 self.sendToken(("OPEN", openID)) return openID def sendToken(self, token): #print token self.tokens.append(token) def sendClose(self, openID): self.sendToken(("CLOSE", openID)) def sendAbort(self, count=0): self.sendToken(("ABORT",)) def sendError(self, msg): #print "TokenBanana.sendError(%s)" % msg pass def testSlice(self, obj): assert len(self.slicerStack) == 1 assert isinstance(self.slicerStack[0][0], RootSlicer) self.tokens = [] d = self.send(obj) d.addCallback(self._testSlice_1) return d def _testSlice_1(self, res): assert len(self.slicerStack) == 1 assert not self.rootSlicer.sendQueue assert isinstance(self.slicerStack[0][0], RootSlicer) return self.tokens def __del__(self): assert not self.rootSlicer.sendQueue def untokenize(tokens): data = [] for t in tokens: if isinstance(t, tuple): if t[0] == "OPEN": int2b128(t[1], data.append) data.append(OPEN) elif t[0] == "CLOSE": int2b128(t[1], data.append) data.append(CLOSE) elif t[0] == "ABORT": data.append(ABORT) else: raise RuntimeError("bad token") else: if isinstance(t, (int, long)): if t >= 2**31: s = long_to_bytes(t) int2b128(len(s), data.append) data.append(LONGINT) data.append(s) elif t >= 0: int2b128(t, data.append) data.append(INT) elif -t > 2**31: # NEG is [-2**31, 0) s = long_to_bytes(-t) int2b128(len(s), data.append) data.append(LONGNEG) data.append(s) else: int2b128(-t, data.append) data.append(NEG) elif isinstance(t, float): data.append(FLOAT) data.append(struct.pack("!d", t)) elif isinstance(t, str): int2b128(len(t), data.append) data.append(STRING) data.append(t) else: raise BananaError, "could not send object: %s" % repr(t) return "".join(data) class UnbananaTestMixin: def setUp(self): self.hangup = False self.banana = storage.StorageBanana() self.banana.slicerClass = storage.UnsafeStorageRootSlicer self.banana.unslicerClass = storage.UnsafeStorageRootUnslicer self.banana.connectionMade() def tearDown(self): if not self.hangup: self.failUnless(len(self.banana.receiveStack) == 1) self.failUnless(isinstance(self.banana.receiveStack[0], storage.UnsafeStorageRootUnslicer)) def do(self, tokens): self.banana.violation = None self.banana.disconnectReason = None self.failUnless(len(self.banana.receiveStack) == 1) self.failUnless(isinstance(self.banana.receiveStack[0], storage.UnsafeStorageRootUnslicer)) data = untokenize(tokens) results = [] d = self.banana.prepare() d.addCallback(results.append) self.banana.dataReceived(data) # we expect everything here to be synchronous if len(results) == 1: return results[0] self.failUnless(self.banana.violation or self.banana.disconnectReason) return None def shouldFail(self, tokens): obj = self.do(tokens) self.failUnless(obj is None, "object was produced: %s" % obj) self.failUnless(self.banana.violation, "didn't fail, ret=%s" % obj) self.failIf(self.banana.disconnectReason, "connection was dropped: %s" % \ self.banana.disconnectReason) return self.banana.violation def shouldDropConnection(self, tokens): self.banana.logReceiveErrors = False try: obj = self.do(tokens) self.fail("connection was supposed to be dropped, got obj=%s" % (obj,)) except BananaError: f = self.banana.disconnectReason if not isinstance(f, Failure): self.fail("disconnectReason wasn't a Failure: %s" % f) if not f.check(BananaError): self.fail("wrong exception type: %s" % f) self.hangup = True # to stop the tearDown check self.failIf(self.banana.violation) return f def failIfBananaFailure(self, res): if isinstance(res, BananaFailure): # something went wrong print "There was a failure while Unbananaing '%s':" % res.where print res.getTraceback() self.fail("BananaFailure") def checkBananaFailure(self, res, where, failtype=None): print res self.failUnless(isinstance(res, BananaFailure)) if failtype: self.failUnless(res.failure, "No Failure object in BananaFailure") if not res.check(failtype): print "Wrong exception (wanted '%s'):" % failtype print res.getTraceback() self.fail("Wrong exception (wanted '%s'):" % failtype) self.failUnlessEqual(res.where, where) self.banana.object = None # to stop the tearDown check TODO ?? class TestTransport(StringIO.StringIO): disconnectReason = None def loseConnection(self): pass class _None: pass class TestBananaMixin: def setUp(self): self.makeBanana() def makeBanana(self): self.banana = storage.StorageBanana() self.banana.slicerClass = storage.UnsafeStorageRootSlicer self.banana.unslicerClass = storage.UnsafeStorageRootUnslicer self.banana.transport = TestTransport() self.banana.connectionMade() def encode(self, obj): d = self.banana.send(obj) d.addCallback(lambda res: self.banana.transport.getvalue()) return d def clearOutput(self): self.banana.transport = TestTransport() def decode(self, stream): self.banana.violation = None results = [] d = self.banana.prepare() d.addCallback(results.append) self.banana.dataReceived(stream) # we expect everything here to be synchronous if len(results) == 1: return results[0] self.failUnless(self.banana.violation or self.banana.disconnectReason) return None def shouldDecode(self, stream): obj = self.decode(stream) self.failIf(self.banana.violation) self.failIf(self.banana.disconnectReason) self.failUnlessEqual(len(self.banana.receiveStack), 1) return obj def shouldFail(self, stream): obj = self.decode(stream) # Violations on a StorageBanana will continue to decode objects, but # will set b.violation, which we can examine afterwards self.failUnlessEqual(obj, None) self.failIf(self.banana.disconnectReason, "connection was dropped: %s" % \ self.banana.disconnectReason) self.failUnlessEqual(len(self.banana.receiveStack), 1) f = self.banana.violation if not f: self.fail("didn't fail") if not isinstance(f, BananaFailure): self.fail("violation wasn't a BananaFailure: %s" % f) if not f.check(Violation): self.fail("wrong exception type: %s" % f) return f def shouldDropConnection(self, stream): self.banana.logReceiveErrors = False # trial hooks log.err try: obj = self.decode(stream) self.fail("decode worked! got '%s', expected dropConnection" \ % (obj,)) except BananaError: # the receiveStack is allowed to be non-empty here, since we've # dropped the connection anyway f = self.banana.disconnectReason if not f: self.fail("didn't fail") if not isinstance(f, Failure): self.fail("disconnectReason wasn't a Failure: %s" % f) if not f.check(BananaError): self.fail("wrong exception type: %s" % f) self.makeBanana() # need a new one, we squished the last one return f def wantEqual(self, got, wanted): if got != wanted: print print "wanted: '%s'" % wanted, repr(wanted) print "got : '%s'" % got, repr(got) self.fail("did not get expected string") def loop(self, obj): self.clearOutput() d = self.encode(obj) d.addCallback(self.shouldDecode) return d def looptest(self, obj, newvalue=_None): if newvalue is _None: newvalue = obj d = self.loop(obj) d.addCallback(self._looptest_1, newvalue) return d def _looptest_1(self, obj2, newvalue): self.failUnlessEqual(obj2, newvalue) self.failUnlessEqual(type(obj2), type(newvalue)) def join(*args): return "".join(args) class BrokenDictUnslicer(DictUnslicer): dieInFinish = 0 def receiveKey(self, key): if key == "die": raise Violation("aaagh") if key == "please_die_in_finish": self.dieInFinish = 1 DictUnslicer.receiveKey(self, key) def receiveValue(self, value): if value == "die": raise Violation("aaaaaaaaargh") DictUnslicer.receiveValue(self, value) def receiveClose(self): if self.dieInFinish: raise Violation("dead in receiveClose()") DictUnslicer.receiveClose(self) return None, None class ReallyBrokenDictUnslicer(DictUnslicer): def start(self, count): raise Violation("dead in start") class DecodeTest(UnbananaTestMixin, unittest.TestCase): def setUp(self): UnbananaTestMixin.setUp(self) self.banana.logReceiveErrors = False d ={ ('dict1',): BrokenDictUnslicer, ('dict2',): ReallyBrokenDictUnslicer, } self.banana.rootUnslicer.topRegistries.insert(0, d) self.banana.rootUnslicer.openRegistries.insert(0, d) def test_simple_list(self): "simple list" res = self.do([tOPEN(0),'list',1,2,3,"a","b",tCLOSE(0)]) self.failUnlessEqual(res, [1,2,3,'a','b']) def test_aborted_list(self): "aborted list" f = self.shouldFail([tOPEN(0),'list', 1, tABORT, tCLOSE(0)]) self.failUnless(isinstance(f, BananaFailure)) self.failUnless(f.check(Violation)) self.failUnlessEqual(f.value.where, ".[1]") self.failUnlessEqual(f.value.args[0], "ABORT received") def test_aborted_list2(self): "aborted list2" f = self.shouldFail([tOPEN(0),'list', 1, tABORT, tOPEN(1),'list', 2, 3, tCLOSE(1), tCLOSE(0)]) self.failUnless(isinstance(f, BananaFailure)) self.failUnless(f.check(Violation)) self.failUnlessEqual(f.value.where, ".[1]") self.failUnlessEqual(f.value.args[0], "ABORT received") def test_aborted_list3(self): "aborted list3" f = self.shouldFail([tOPEN(0),'list', 1, tOPEN(1),'list', 2, 3, 4, tOPEN(2),'list', 5, 6, tABORT, tCLOSE(2), tCLOSE(1), tCLOSE(0)]) self.failUnless(isinstance(f, BananaFailure)) self.failUnless(f.check(Violation)) self.failUnlessEqual(f.value.where, ".[1].[3].[2]") self.failUnlessEqual(f.value.args[0], "ABORT received") def test_nested_list(self): "nested list" res = self.do([tOPEN(0),'list',1,2, tOPEN(1),'list',3,4,tCLOSE(1), tCLOSE(0)]) self.failUnlessEqual(res, [1,2,[3,4]]) def test_list_with_tuple(self): "list with tuple" res = self.do([tOPEN(0),'list',1,2, tOPEN(1),'tuple',3,4,tCLOSE(1), tCLOSE(0)]) self.failUnlessEqual(res, [1,2,(3,4)]) def test_dict(self): "dict" res = self.do([tOPEN(0),'dict',"a",1,"b",2,tCLOSE(0)]) self.failUnlessEqual(res, {'a':1, 'b':2}) def test_dict_with_duplicate_keys(self): "dict with duplicate keys" f = self.shouldDropConnection([tOPEN(0),'dict', "a",1,"a",2, tCLOSE(0)]) self.failUnlessEqual(f.value.where, ".{}") self.failUnlessEqual(f.value.args[0], "duplicate key 'a'") def test_dict_with_list(self): "dict with list" res = self.do([tOPEN(0),'dict', "a",1, "b", tOPEN(1),'list', 2, 3, tCLOSE(1), tCLOSE(0)]) self.failUnlessEqual(res, {'a':1, 'b':[2,3]}) def test_dict_with_tuple_as_key(self): "dict with tuple as key" res = self.do([tOPEN(0),'dict', tOPEN(1),'tuple', 1, 2, tCLOSE(1), "a", tCLOSE(0)]) self.failUnlessEqual(res, {(1,2):'a'}) def test_dict_with_mutable_key(self): "dict with mutable key" f = self.shouldDropConnection([tOPEN(0),'dict', tOPEN(1),'list', 1, 2, tCLOSE(1), "a", tCLOSE(0)]) self.failUnlessEqual(f.value.where, ".{}") self.failUnlessEqual(f.value.args[0], "unhashable key '[1, 2]'") def test_instance(self): "instance" f1 = Foo(); f1.a = 1; f1.b = [2,3] f2 = Bar(); f2.d = 4; f1.c = f2 res = self.do([tOPEN(0),'instance', "foolscap.test.test_banana.Foo", "a", 1, "b", tOPEN(1),'list', 2, 3, tCLOSE(1), "c", tOPEN(2),'instance', "foolscap.test.test_banana.Bar", "d", 4, tCLOSE(2), tCLOSE(0)]) self.failUnlessEqual(res, f1) def test_instance_bad1(self): "subinstance with numeric classname" tokens = [tOPEN(0),'instance', "Foo", "a", 1, "b", tOPEN(1),'list', 2, 3, tCLOSE(1), "c", tOPEN(2),'instance', 37, "d", 4, tCLOSE(2), tCLOSE(0)] f = self.shouldDropConnection(tokens) self.failUnlessEqual(f.value.where, "..c.") self.failUnlessEqual(f.value.args[0], "InstanceUnslicer classname must be string") def test_instance_bad2(self): "subinstance with numeric attribute name" tokens = [tOPEN(0),'instance', "Foo", "a", 1, "b", tOPEN(1),'list', 2, 3, tCLOSE(1), "c", tOPEN(2),'instance', "Bar", 37, 4, tCLOSE(2), tCLOSE(0)] f = self.shouldDropConnection(tokens) self.failUnlessEqual(f.value.where, "..c..attrname??") self.failUnlessEqual(f.value.args[0], "InstanceUnslicer keys must be STRINGs") def test_instance_unsafe1(self): "instances when instances aren't allowed" self.banana.rootUnslicer.topRegistries = [slicer.UnslicerRegistry] self.banana.rootUnslicer.openRegistries = [slicer.UnslicerRegistry] tokens = [tOPEN(0),'instance', "Foo", "a", 1, "b", tOPEN(1),'list', 2, 3, tCLOSE(1), "c", tOPEN(2),'instance', "Bar", 37, 4, tCLOSE(2), tCLOSE(0)] f = self.shouldFail(tokens) self.failUnlessEqual(f.value.where, "") self.failUnlessEqual(f.value.args[0], "unknown top-level OPEN type ('instance',)") def test_ref1(self): res = self.do([tOPEN(0),'list', tOPEN(1),'list', 1, 2, tCLOSE(1), tOPEN(2),'reference', 1, tCLOSE(2), tCLOSE(0)]) self.failIfBananaFailure(res) self.failUnlessEqual(res, [[1,2], [1,2]]) self.failUnlessIdentical(res[0], res[1]) def test_ref2(self): res = self.do([tOPEN(0),'list', tOPEN(1),'list', 1, 2, tCLOSE(1), tOPEN(2),'reference', 0, tCLOSE(2), tCLOSE(0)]) self.failIfBananaFailure(res) wanted = [[1,2]] wanted.append(wanted) # python2.3 is clever and can do # self.failUnlessEqual(res, wanted) # python2.4 is not, so we do it by hand self.failUnlessEqual(len(res), len(wanted)) self.failUnlessEqual(res[0], wanted[0]) self.failUnlessIdentical(res, res[1]) def test_ref3(self): res = self.do([tOPEN(0),'list', tOPEN(1),'tuple', 1, 2, tCLOSE(1), tOPEN(2),'reference', 1, tCLOSE(2), tCLOSE(0)]) self.failIfBananaFailure(res) wanted = [(1,2)] wanted.append(wanted[0]) self.failUnlessEqual(res, wanted) self.failUnlessIdentical(res[0], res[1]) def test_ref4(self): res = self.do([tOPEN(0),'list', tOPEN(1),'dict', "a", 1, tCLOSE(1), tOPEN(2),'reference', 1, tCLOSE(2), tCLOSE(0)]) self.failIfBananaFailure(res) wanted = [{"a":1}] wanted.append(wanted[0]) self.failUnlessEqual(res, wanted) self.failUnlessIdentical(res[0], res[1]) def test_ref5(self): # The Droste Effect: a list that contains itself res = self.do([tOPEN(0),'list', 5, 6, tOPEN(1),'reference', 0, tCLOSE(1), 7, tCLOSE(0)]) self.failIfBananaFailure(res) wanted = [5,6] wanted.append(wanted) wanted.append(7) #self.failUnlessEqual(res, wanted) self.failUnlessEqual(len(res), len(wanted)) self.failUnlessEqual(res[0:2], wanted[0:2]) self.failUnlessIdentical(res[2], res) self.failUnlessEqual(res[3], wanted[3]) def test_ref6(self): # everybody's favorite "([(ref0" test case. A tuple of a list of a # tuple of the original tuple. Such cycles must always have a # mutable container in them somewhere, or they couldn't be # constructed, but the resulting object involves a lot of deferred # results because the mutable list is the *only* object that can # be created without dependencies res = self.do([tOPEN(0),'tuple', tOPEN(1),'list', tOPEN(2),'tuple', tOPEN(3),'reference', 0, tCLOSE(3), tCLOSE(2), tCLOSE(1), tCLOSE(0)]) self.failIfBananaFailure(res) wanted = ([],) wanted[0].append((wanted,)) #self.failUnlessEqual(res, wanted) self.failUnless(type(res) is tuple) self.failUnless(len(res) == 1) self.failUnless(type(res[0]) is list) self.failUnless(len(res[0]) == 1) self.failUnless(type(res[0][0]) is tuple) self.failUnless(len(res[0][0]) == 1) self.failUnlessIdentical(res[0][0][0], res) # TODO: need a test where tuple[0] and [1] are deferred, but # tuple[0] becomes available before tuple[2] is inserted. Not sure # this is possible, but it would improve test coverage in # TupleUnslicer def test_failed_dict1(self): # dies during open because of bad opentype f = self.shouldFail([tOPEN(0),'list', 1, tOPEN(1),"bad", "a", 2, "b", 3, tCLOSE(1), tCLOSE(0)]) self.failUnless(isinstance(f, BananaFailure)) self.failUnless(f.check(Violation)) self.failUnlessEqual(f.value.where, ".[1]") self.failUnlessEqual(f.value.args[0], "unknown OPEN type ('bad',)") def test_failed_dict2(self): # dies during start f = self.shouldFail([tOPEN(0),'list', 1, tOPEN(1),'dict2', "a", 2, "b", 3, tCLOSE(1), tCLOSE(0)]) self.failUnless(isinstance(f, BananaFailure)) self.failUnless(f.check(Violation)) self.failUnlessEqual(f.value.where, ".[1].{}") self.failUnlessEqual(f.value.args[0], "dead in start") def test_failed_dict3(self): # dies during key f = self.shouldFail([tOPEN(0),'list', 1, tOPEN(1),'dict1', "a", 2, "die", tCLOSE(1), tCLOSE(0)]) self.failUnless(isinstance(f, BananaFailure)) self.failUnless(f.check(Violation)) self.failUnlessEqual(f.value.where, ".[1].{}") self.failUnlessEqual(f.value.args[0], "aaagh") res = self.do([tOPEN(2),'list', 3, 4, tCLOSE(2)]) self.failUnlessEqual(res, [3,4]) def test_failed_dict4(self): # dies during value f = self.shouldFail([tOPEN(0),'list', 1, tOPEN(1),'dict1', "a", 2, "b", "die", tCLOSE(1), tCLOSE(0)]) self.failUnless(isinstance(f, BananaFailure)) self.failUnless(f.check(Violation)) self.failUnlessEqual(f.value.where, ".[1].{}[b]") self.failUnlessEqual(f.value.args[0], "aaaaaaaaargh") def test_failed_dict5(self): # dies during finish f = self.shouldFail([tOPEN(0),'list', 1, tOPEN(1),'dict1', "a", 2, "please_die_in_finish", 3, tCLOSE(1), tCLOSE(0)]) self.failUnless(isinstance(f, BananaFailure)) self.failUnless(f.check(Violation)) self.failUnlessEqual(f.value.where, ".[1].{}") self.failUnlessEqual(f.value.args[0], "dead in receiveClose()") class Bar: def __cmp__(self, them): if not type(them) == type(self): return -1 return cmp((self.__class__, self.__dict__), (them.__class__, them.__dict__)) class Foo(Bar): pass class EncodeTest(unittest.TestCase): def setUp(self): self.banana = TokenBanana() self.banana.slicerClass = storage.UnsafeStorageRootSlicer self.banana.unslicerClass = storage.UnsafeStorageRootUnslicer self.banana.connectionMade() def do(self, obj): return self.banana.testSlice(obj) def tearDown(self): self.failUnless(len(self.banana.slicerStack) == 1) self.failUnless(isinstance(self.banana.slicerStack[0][0], RootSlicer)) def testList(self): d = self.do([1,2]) d.addCallback(self.failUnlessEqual, [tOPEN(0),'list', 1, 2, tCLOSE(0)]) return d def testTuple(self): d = self.do((1,2)) d.addCallback(self.failUnlessEqual, [tOPEN(0),'tuple', 1, 2, tCLOSE(0)]) return d def testNestedList(self): d = self.do([1,2,[3,4]]) d.addCallback(self.failUnlessEqual, [tOPEN(0),'list', 1, 2, tOPEN(1),'list', 3, 4, tCLOSE(1), tCLOSE(0)]) return d def testNestedList2(self): d = self.do([1,2,(3,4,[5, "hi"])]) d.addCallback(self.failUnlessEqual, [tOPEN(0),'list', 1, 2, tOPEN(1),'tuple', 3, 4, tOPEN(2),'list', 5, "hi", tCLOSE(2), tCLOSE(1), tCLOSE(0)]) return d def testDict(self): d = self.do({'a': 1, 'b': 2}) d.addCallback(lambda res: self.failUnless( res == [tOPEN(0),'dict', 'a', 1, 'b', 2, tCLOSE(0)] or res == [tOPEN(0),'dict', 'b', 2, 'a', 1, tCLOSE(0)])) return d def test_ref1(self): l = [1,2] obj = [l,l] d = self.do(obj) d.addCallback(self.failUnlessEqual, [tOPEN(0),'list', tOPEN(1),'list', 1, 2, tCLOSE(1), tOPEN(2),'reference', 1, tCLOSE(2), tCLOSE(0)]) return d def test_ref2(self): obj = [[1,2]] obj.append(obj) d = self.do(obj) d.addCallback(self.failUnlessEqual, [tOPEN(0),'list', tOPEN(1),'list', 1, 2, tCLOSE(1), tOPEN(2),'reference', 0, tCLOSE(2), tCLOSE(0)]) return d def test_ref3(self): obj = [(1,2)] obj.append(obj[0]) d = self.do(obj) d.addCallback(self.failUnlessEqual, [tOPEN(0),'list', tOPEN(1),'tuple', 1, 2, tCLOSE(1), tOPEN(2),'reference', 1, tCLOSE(2), tCLOSE(0)]) return d def test_ref4(self): obj = [{"a":1}] obj.append(obj[0]) d = self.do(obj) d.addCallback(self.failUnlessEqual, [tOPEN(0),'list', tOPEN(1),'dict', "a", 1, tCLOSE(1), tOPEN(2),'reference', 1, tCLOSE(2), tCLOSE(0)]) return d def test_ref6(self): # everybody's favorite "([(ref0" test case. obj = ([],) obj[0].append((obj,)) d = self.do(obj) d.addCallback(self.failUnlessEqual, [tOPEN(0),'tuple', tOPEN(1),'list', tOPEN(2),'tuple', tOPEN(3),'reference', 0, tCLOSE(3), tCLOSE(2), tCLOSE(1), tCLOSE(0)]) return d def test_refdict1(self): # a dictionary with a value that isn't available right away d0 = {1: "a"} t = (d0,) d0[2] = t d = self.do(d0) d.addCallback(self.failUnlessEqual, [tOPEN(0),'dict', 1, "a", 2, tOPEN(1),'tuple', tOPEN(2),'reference', 0, tCLOSE(2), tCLOSE(1), tCLOSE(0)]) return d def test_instance_one(self): obj = Bar() obj.a = 1 classname = reflect.qual(Bar) d = self.do(obj) d.addCallback(self.failUnlessEqual, [tOPEN(0),'instance', classname, "a", 1, tCLOSE(0)]) return d def test_instance_two(self): f1 = Foo(); f1.a = 1; f1.b = [2,3] f2 = Bar(); f2.d = 4; f1.c = f2 fooname = reflect.qual(Foo) barname = reflect.qual(Bar) # needs OrderedDictSlicer for the test to work d = self.do(f1) d.addCallback(self.failUnlessEqual, [tOPEN(0),'instance', fooname, "a", 1, "b", tOPEN(1),'list', 2, 3, tCLOSE(1), "c", tOPEN(2),'instance', barname, "d", 4, tCLOSE(2), tCLOSE(0)]) class ErrorfulSlicer(slicer.BaseSlicer): def __init__(self, mode, shouldSucceed, ignoreChildDeath=False): self.mode = mode self.items = [1] self.items.append(mode) self.items.append(3) #if mode not in ('success', 'deferred-good'): if not shouldSucceed: self.items.append("unreached") self.counter = -1 self.childDied = False self.ignoreChildDeath = ignoreChildDeath def __iter__(self): return self def slice(self, streamable, banana): self.streamable = streamable if self.mode == "slice": raise Violation("slice failed") return self def next(self): self.counter += 1 if not self.items: raise StopIteration obj = self.items.pop(0) if obj == "next": raise Violation("next failed") if obj == "deferred-good": return fireEventually(None) if obj == "deferred-bad": d = defer.Deferred() # the Banana should bail, so don't bother with the timer return d if obj == "newSlicerFor": unserializable = open("unserializable.txt", "w") # Hah! Serialize that! return unserializable if obj == "unreached": print "error: slicer.next called after it should have stopped" return obj def childAborted(self, v): self.childDied = True if self.ignoreChildDeath: return None return v def describe(self): return "ErrorfulSlicer[%d]" % self.counter # Slicer creation (schema pre-validation?) # .slice (called in pushSlicer) ? # .slice.next raising Violation # .slice.next returning Deferred when streaming isn't allowed # .sendToken (non-primitive token, can't happen) # .newSlicerFor (no ISlicer adapter) # top.childAborted class EncodeFailureTest(unittest.TestCase): def setUp(self): self.banana = TokenBanana() self.banana.slicerClass = storage.UnsafeStorageRootSlicer self.banana.unslicerClass = storage.UnsafeStorageRootUnslicer self.banana.connectionMade() def tearDown(self): return flushEventualQueue() def send(self, obj): self.banana.tokens = [] d = self.banana.send(obj) d.addCallback(lambda res: self.banana.tokens) return d def testSuccess1(self): # make sure the test slicer works correctly s = ErrorfulSlicer("success", True) d = self.send(s) d.addCallback(self.failUnlessEqual, [('OPEN', 0), 1, 'success', 3, ('CLOSE', 0)]) return d def testSuccessStreaming(self): # success s = ErrorfulSlicer("deferred-good", True) d = self.send(s) d.addCallback(self.failUnlessEqual, [('OPEN', 0), 1, 3, ('CLOSE', 0)]) return d def test1(self): # failure during .slice (called from pushSlicer) s = ErrorfulSlicer("slice", False) d = self.send(s) d.addCallbacks(lambda res: self.fail("this was supposed to fail"), self._test1_1) return d def _test1_1(self, e): e.trap(Violation) self.failUnlessEqual(e.value.where, "") self.failUnlessEqual(e.value.args, ("slice failed",)) self.failUnlessEqual(self.banana.tokens, []) def test2(self): # .slice.next raising Violation s = ErrorfulSlicer("next", False) d = self.send(s) d.addCallbacks(lambda res: self.fail("this was supposed to fail"), self._test2_1) return d def _test2_1(self, e): e.trap(Violation) self.failUnlessEqual(e.value.where, ".ErrorfulSlicer[1]") self.failUnlessEqual(e.value.args, ("next failed",)) self.failUnlessEqual(self.banana.tokens, [('OPEN', 0), 1, ('ABORT',), ('CLOSE', 0)]) def test3(self): # .slice.next returning Deferred when streaming isn't allowed self.banana.rootSlicer.allowStreaming(False) s = ErrorfulSlicer("deferred-bad", False) d = self.send(s) d.addCallbacks(lambda res: self.fail("this was supposed to fail"), self._test3_1) return d def _test3_1(self, e): e.trap(Violation) self.failUnlessEqual(e.value.where, ".ErrorfulSlicer[1]") self.failUnlessEqual(e.value.args, ("parent not streamable",)) self.failUnlessEqual(self.banana.tokens, [('OPEN', 0), 1, ('ABORT',), ('CLOSE', 0)]) def test4(self): # .newSlicerFor (no ISlicer adapter), parent propagates upwards s = ErrorfulSlicer("newSlicerFor", False) d = self.send(s) d.addCallbacks(lambda res: self.fail("this was supposed to fail"), self._test4_1, errbackArgs=(s,)) return d def _test4_1(self, e, s): e.trap(Violation) self.failUnlessEqual(e.value.where, ".ErrorfulSlicer[1]") self.failUnlessSubstring("cannot serialize ") why = e.value.args[0] self.failUnless( why.startswith("cannot serialize " "64 bytes) # checkToken (top.openerCheckToken) # checkToken (top.checkToken) # typebyte == LIST (oldbanana) # bad VOCAB key # TODO # too-long vocab key # bad FLOAT encoding # I don't there is such a thing # top.receiveClose # top.finish # top.reportViolation # oldtop.finish (in from handleViolation) # top.doOpen # top.start # plus all of these when discardCount != 0 class ErrorfulUnslicer(slicer.BaseUnslicer): debug = False def doOpen(self, opentype): if self.mode == "doOpen": raise Violation("boom") return slicer.BaseUnslicer.doOpen(self, opentype) def start(self, count): self.mode = self.protocol.mode self.ignoreChildDeath = self.protocol.ignoreChildDeath if self.debug: print "ErrorfulUnslicer.start, mode=%s" % self.mode self.list = [] if self.mode == "start": raise Violation("boom") def openerCheckToken(self, typebyte, size, opentype): if self.debug: print "ErrorfulUnslicer.openerCheckToken(%s)" % tokenNames[typebyte] if self.mode == "openerCheckToken": raise Violation("boom") return slicer.BaseUnslicer.openerCheckToken(self, typebyte, size, opentype) def checkToken(self, typebyte, size): if self.debug: print "ErrorfulUnslicer.checkToken(%s)" % tokenNames[typebyte] if self.mode == "checkToken": raise Violation("boom") if self.mode == "checkToken-OPEN" and typebyte == OPEN: raise Violation("boom") return slicer.BaseUnslicer.checkToken(self, typebyte, size) def receiveChild(self, obj, ready_deferred=None): if self.debug: print "ErrorfulUnslicer.receiveChild", obj if self.mode == "receiveChild": raise Violation("boom") self.list.append(obj) def reportViolation(self, why): if self.ignoreChildDeath: return None return why def receiveClose(self): if self.debug: print "ErrorfulUnslicer.receiveClose" if self.protocol.mode == "receiveClose": raise Violation("boom") return self.list, None def finish(self): if self.debug: print "ErrorfulUnslicer.receiveClose" if self.protocol.mode == "finish": raise Violation("boom") def describe(self): return "errorful" class FailingUnslicer(TupleUnslicer): def receiveChild(self, obj, ready_deferred=None): if self.protocol.mode != "success": raise Violation("foom") return TupleUnslicer.receiveChild(self, obj, ready_deferred) def describe(self): return "failing" class DecodeFailureTest(TestBananaMixin, unittest.TestCase): listStream = join(bOPEN("errorful", 0), bINT(1), bINT(2), bCLOSE(0)) nestedStream = join(bOPEN("errorful", 0), bINT(1), bOPEN("list", 1), bINT(2), bINT(3), bCLOSE(1), bCLOSE(0)) nestedStream2 = join(bOPEN("failing", 0), bSTR("a"), bOPEN("errorful", 1), bINT(1), bOPEN("list", 2), bINT(2), bINT(3), bCLOSE(2), bCLOSE(1), bSTR("b"), bCLOSE(0), ) abortStream = join(bOPEN("errorful", 0), bINT(1), bOPEN("list", 1), bINT(2), bABORT(1), bINT(3), bCLOSE(1), bCLOSE(0)) def setUp(self): TestBananaMixin.setUp(self) d = {('errorful',): ErrorfulUnslicer, ('failing',): FailingUnslicer, } self.banana.rootUnslicer.topRegistries.insert(0, d) self.banana.rootUnslicer.openRegistries.insert(0, d) self.banana.ignoreChildDeath = False def testSuccess1(self): self.banana.mode = "success" o = self.shouldDecode(self.listStream) self.failUnlessEqual(o, [1,2]) o = self.shouldDecode(self.nestedStream) self.failUnlessEqual(o, [1,[2,3]]) o = self.shouldDecode(self.nestedStream2) self.failUnlessEqual(o, ("a",[1,[2,3]],"b")) def testLongHeader(self): # would be a string but the header is too long s = "\x01" * 66 + "\x82" + "stupidly long string" f = self.shouldDropConnection(s) self.failUnless(f.value.args[0].startswith("token prefix is limited to 64 bytes")) def testLongHeader2(self): # bad string while discarding s = "\x01" * 66 + "\x82" + "stupidly long string" s = bOPEN("errorful",0) + bINT(1) + s + bINT(2) + bCLOSE(0) self.banana.mode = "start" f = self.shouldDropConnection(s) self.failUnless(f.value.args[0].startswith("token prefix is limited to 64 bytes")) def testCheckToken1(self): # violation raised in top.openerCheckToken self.banana.mode = "openerCheckToken" f = self.shouldFail(self.nestedStream) self.failUnlessEqual(f.value.where, ".errorful") self.failUnlessEqual(f.value.args[0], "boom") self.testSuccess1() def testCheckToken2(self): # violation raised in top.openerCheckToken, but the error is # absorbed self.banana.mode = "openerCheckToken" self.banana.ignoreChildDeath = True o = self.shouldDecode(self.nestedStream) self.failUnlessEqual(o, [1]) self.testSuccess1() def testCheckToken3(self): # violation raised in top.checkToken self.banana.mode = "checkToken" f = self.shouldFail(self.listStream) self.failUnlessEqual(f.value.where, ".errorful") self.failUnlessEqual(f.value.args[0], "boom") self.testSuccess1() def testCheckToken4(self): # violation raised in top.checkToken, but only for the OPEN that # starts the nested list. The error is absorbed. self.banana.mode = "checkToken-OPEN" self.banana.ignoreChildDeath = True o = self.shouldDecode(self.nestedStream) self.failUnlessEqual(o, [1]) self.testSuccess1() def testCheckToken5(self): # violation raised in top.checkToken, while discarding self.banana.mode = "checkToken" #self.banana.debugReceive=True f = self.shouldFail(self.nestedStream2) self.failUnlessEqual(f.value.where, ".failing") self.failUnlessEqual(f.value.args[0], "foom") self.testSuccess1() def testReceiveChild1(self): self.banana.mode = "receiveChild" f = self.shouldFail(self.listStream) self.failUnlessEqual(f.value.where, ".errorful") self.failUnlessEqual(f.value.args[0], "boom") self.testSuccess1() def testReceiveChild2(self): self.banana.mode = "receiveChild" f = self.shouldFail(self.nestedStream2) self.failUnlessEqual(f.value.where, ".failing") self.failUnlessEqual(f.value.args[0], "foom") self.testSuccess1() def testReceiveChild3(self): self.banana.mode = "receiveChild" # the ABORT should be ignored, since it is in the middle of a # sequence which is being ignored. One possible bug is that the # ABORT delivers a second Violation. In this test, we only record # the last Violation, so we'll catch that case. f = self.shouldFail(self.abortStream) self.failUnlessEqual(f.value.where, ".errorful") self.failUnlessEqual(f.value.args[0], "boom") # (the other Violation would be at 'root', of type 'ABORT received' self.testSuccess1() def testReceiveClose1(self): self.banana.mode = "receiveClose" f = self.shouldFail(self.listStream) self.failUnlessEqual(f.value.where, ".errorful") self.failUnlessEqual(f.value.args[0], "boom") self.testSuccess1() def testReceiveClose2(self): self.banana.mode = "receiveClose" f = self.shouldFail(self.nestedStream2) self.failUnlessEqual(f.value.where, ".failing") self.failUnlessEqual(f.value.args[0], "foom") self.testSuccess1() def testFinish1(self): self.banana.mode = "finish" f = self.shouldFail(self.listStream) self.failUnlessEqual(f.value.where, ".errorful") self.failUnlessEqual(f.value.args[0], "boom") self.testSuccess1() def testFinish2(self): self.banana.mode = "finish" f = self.shouldFail(self.nestedStream2) self.failUnlessEqual(f.value.where, ".failing") self.failUnlessEqual(f.value.args[0], "foom") self.testSuccess1() def testStart1(self): self.banana.mode = "start" f = self.shouldFail(self.listStream) self.failUnlessEqual(f.value.where, ".errorful") self.failUnlessEqual(f.value.args[0], "boom") self.testSuccess1() def testStart2(self): self.banana.mode = "start" f = self.shouldFail(self.nestedStream2) self.failUnlessEqual(f.value.where, ".failing") self.failUnlessEqual(f.value.args[0], "foom") self.testSuccess1() def testDoOpen1(self): self.banana.mode = "doOpen" f = self.shouldFail(self.nestedStream) self.failUnlessEqual(f.value.where, ".errorful") self.failUnlessEqual(f.value.args[0], "boom") self.testSuccess1() def testDoOpen2(self): self.banana.mode = "doOpen" f = self.shouldFail(self.nestedStream2) self.failUnlessEqual(f.value.where, ".failing") self.failUnlessEqual(f.value.args[0], "foom") self.testSuccess1() class ByteStream(TestBananaMixin, unittest.TestCase): def test_list(self): obj = [1,2] expected = join(bOPEN("list", 0), bINT(1), bINT(2), bCLOSE(0), ) d = self.encode(obj) d.addCallback(self.wantEqual, expected) return d def test_ref6(self): # everybody's favorite "([(ref0" test case. obj = ([],) obj[0].append((obj,)) expected = join(bOPEN("tuple",0), bOPEN("list",1), bOPEN("tuple",2), bOPEN("reference",3), bINT(0), bCLOSE(3), bCLOSE(2), bCLOSE(1), bCLOSE(0)) d = self.encode(obj) d.addCallback(self.wantEqual, expected) return d def test_two(self): f1 = Foo(); f1.a = 1; f1.b = [2,3] f2 = Bar(); f2.d = 4; f1.c = f2 fooname = reflect.qual(Foo) barname = reflect.qual(Bar) # needs OrderedDictSlicer for the test to work expected = join(bOPEN("instance",0), bSTR(fooname), bSTR("a"), bINT(1), bSTR("b"), bOPEN("list",1), bINT(2), bINT(3), bCLOSE(1), bSTR("c"), bOPEN("instance",2), bSTR(barname), bSTR("d"), bINT(4), bCLOSE(2), bCLOSE(0)) d = self.encode(f1) d.addCallback(self.wantEqual, expected) return d class InboundByteStream(TestBananaMixin, unittest.TestCase): def check(self, obj, stream): # use a new Banana for each check self.makeBanana() obj2 = self.shouldDecode(stream) self.failUnlessEqual(obj, obj2) def testInt(self): self.check(1, "\x01\x81") self.check(130, "\x02\x01\x81") self.check(-1, "\x01\x83") self.check(-130, "\x02\x01\x83") self.check(0, bINT(0)) self.check(1, bINT(1)) self.check(127, bINT(127)) self.check(-1, bINT(-1)) self.check(-127, bINT(-127)) def testLong(self): self.check(258L, "\x02\x85\x01\x02") # TODO: 0x85 for LONGINT?? self.check(-258L, "\x02\x86\x01\x02") # TODO: 0x85 for LONGINT?? self.check(0L, "\x85") self.check(0L, "\x00\x85") self.check(0L, "\x86") self.check(0L, "\x00\x86") def testString(self): self.check("", "\x82") self.check("", "\x00\x82") self.check("", "\x00\x00\x82") self.check("", "\x00" * 64 + "\x82") f = self.shouldDropConnection("\x00" * 65) self.failUnlessEqual(f.value.where, "") self.failUnless(f.value.args[0].startswith("token prefix is limited to 64 bytes")) f = self.shouldDropConnection("\x00" * 65 + "\x82") self.failUnlessEqual(f.value.where, "") self.failUnless(f.value.args[0].startswith("token prefix is limited to 64 bytes")) self.check("a", "\x01\x82a") self.check("b"*130, "\x02\x01\x82" + "b"*130 + "extra") self.check("c"*1025, "\x01\x08\x82" + "c" * 1025 + "extra") self.check("fluuber", bSTR("fluuber")) def testList(self): self.check([1,2], join(bOPEN('list',1), bINT(1), bINT(2), bCLOSE(1))) self.check([1,"b"], join(bOPEN('list',1), bINT(1), "\x01\x82b", bCLOSE(1))) self.check([1,2,[3,4]], join(bOPEN('list',1), bINT(1), bINT(2), bOPEN('list',2), bINT(3), bINT(4), bCLOSE(2), bCLOSE(1))) def testTuple(self): self.check((1,2), join(bOPEN('tuple',1), bINT(1), bINT(2), bCLOSE(1))) def testDict(self): self.check({1:"a", 2:["b","c"]}, join(bOPEN('dict',1), bINT(1), bSTR("a"), bINT(2), bOPEN('list',2), bSTR("b"), bSTR("c"), bCLOSE(2), bCLOSE(1))) def TRUE(self): return join(bOPEN("boolean",2), bINT(1), bCLOSE(2)) def FALSE(self): return join(bOPEN("boolean",2), bINT(0), bCLOSE(2)) def testBool(self): self.check(True, self.TRUE()) self.check(False, self.FALSE()) class InboundByteStream2(TestBananaMixin, unittest.TestCase): def setConstraints(self, constraint, childConstraint): if constraint: constraint = IConstraint(constraint) self.banana.receiveStack[-1].constraint = constraint if childConstraint: childConstraint = IConstraint(childConstraint) self.banana.receiveStack[-1].childConstraint = childConstraint def conform2(self, stream, obj, constraint=None, childConstraint=None): self.setConstraints(constraint, childConstraint) obj2 = self.shouldDecode(stream) self.failUnlessEqual(obj, obj2) def violate2(self, stream, where, constraint=None, childConstraint=None): self.setConstraints(constraint, childConstraint) f = self.shouldFail(stream) self.failUnlessEqual(f.value.where, where) self.failUnlessEqual(len(self.banana.receiveStack), 1) def testConstrainedInt(self): pass # TODO: after implementing new LONGINT token def testConstrainedString(self): self.conform2("\x82", "", schema.StringConstraint(10)) self.conform2("\x0a\x82" + "a"*10 + "extra", "a"*10, schema.StringConstraint(10)) self.violate2("\x0b\x82" + "a"*11 + "extra", "", schema.StringConstraint(10)) def NOTtestFoo(self): if 0: a100 = chr(100) + "\x82" + "a"*100 b100 = chr(100) + "\x82" + "b"*100 self.violate2(join(bOPEN('list',1), bOPEN('list',2), a100, b100, bCLOSE(2), bCLOSE(1)), ".[0].[0]", schema.ListOf( schema.ListOf(schema.StringConstraint(99), 2), 2)) def OPENweird(count, weird): return chr(count) + "\x88" + weird self.violate2(join(bOPEN('list',1), bOPEN('list',2), OPENweird(3, bINT(64)), bINT(1), bINT(2), bCLOSE(3), bCLOSE(2), bCLOSE(1)), ".[0].[0]", None) def testConstrainedList(self): self.conform2(join(bOPEN('list',1), bINT(1), bINT(2), bCLOSE(1)), [1,2], schema.ListOf(int)) self.violate2(join(bOPEN('list',1), bINT(1), "\x01\x82b", bCLOSE(1)), ".[1]", schema.ListOf(int)) self.conform2(join(bOPEN('list',1), bINT(1), bINT(2), bINT(3), bCLOSE(1)), [1,2,3], schema.ListOf(int, maxLength=3)) self.violate2(join(bOPEN('list',1), bINT(1), bINT(2), bINT(3), bINT(4), bCLOSE(1)), ".[3]", schema.ListOf(int, maxLength=3)) a100 = chr(100) + "\x82" + "a"*100 b100 = chr(100) + "\x82" + "b"*100 self.conform2(join(bOPEN('list',1), a100, b100, bCLOSE(1)), ["a"*100, "b"*100], schema.ListOf(schema.StringConstraint(100), 2)) self.violate2(join(bOPEN('list',1), a100, b100, bCLOSE(1)), ".[0]", schema.ListOf(schema.StringConstraint(99), 2)) self.violate2(join(bOPEN('list',1), a100, b100, a100, bCLOSE(1)), ".[2]", schema.ListOf(schema.StringConstraint(100), 2)) self.conform2(join(bOPEN('list',1), bOPEN('list',2), bINT(11), bINT(12), bCLOSE(2), bOPEN('list',3), bINT(21), bINT(22), bINT(23), bCLOSE(3), bCLOSE(1)), [[11,12], [21, 22, 23]], schema.ListOf(schema.ListOf(int, maxLength=3))) self.violate2(join(bOPEN('list',1), bOPEN('list',2), bINT(11), bINT(12), bCLOSE(2), bOPEN('list',3), bINT(21), bINT(22), bINT(23), bCLOSE(3), bCLOSE(1)), ".[1].[2]", schema.ListOf(schema.ListOf(int, maxLength=2))) def testConstrainedTuple(self): self.conform2(join(bOPEN('tuple',1), bINT(1), bINT(2), bCLOSE(1)), (1,2), schema.TupleOf(int, int)) self.violate2(join(bOPEN('tuple',1), bINT(1), bINT(2), bINT(3), bCLOSE(1)), ".[2]", schema.TupleOf(int, int)) self.violate2(join(bOPEN('tuple',1), bINT(1), bSTR("not a number"), bCLOSE(1)), ".[1]", schema.TupleOf(int, int)) self.conform2(join(bOPEN('tuple',1), bINT(1), bSTR("twine"), bCLOSE(1)), (1, "twine"), schema.TupleOf(int, str)) self.conform2(join(bOPEN('tuple',1), bINT(1), bOPEN('list',2), bINT(1), bINT(2), bINT(3), bCLOSE(2), bCLOSE(1)), (1, [1,2,3]), schema.TupleOf(int, schema.ListOf(int))) self.conform2(join(bOPEN('tuple',1), bINT(1), bOPEN('list',2), bOPEN('list',3), bINT(2), bCLOSE(3), bOPEN('list',4), bINT(3), bCLOSE(4), bCLOSE(2), bCLOSE(1)), (1, [[2], [3]]), schema.TupleOf(int, schema.ListOf(schema.ListOf(int)))) self.violate2(join(bOPEN('tuple',1), bINT(1), bOPEN('list',2), bOPEN('list',3), bSTR("nan"), bCLOSE(3), bOPEN('list',4), bINT(3), bCLOSE(4), bCLOSE(2), bCLOSE(1)), ".[1].[0].[0]", schema.TupleOf(int, schema.ListOf(schema.ListOf(int)))) def testConstrainedDict(self): self.conform2(join(bOPEN('dict',1), bINT(1), bSTR("a"), bINT(2), bSTR("b"), bINT(3), bSTR("c"), bCLOSE(1)), {1:"a", 2:"b", 3:"c"}, schema.DictOf(int, str)) self.conform2(join(bOPEN('dict',1), bINT(1), bSTR("a"), bINT(2), bSTR("b"), bINT(3), bSTR("c"), bCLOSE(1)), {1:"a", 2:"b", 3:"c"}, schema.DictOf(int, str, maxKeys=3)) self.violate2(join(bOPEN('dict',1), bINT(1), bSTR("a"), bINT(2), bINT(10), bINT(3), bSTR("c"), bCLOSE(1)), ".{}[2]", schema.DictOf(int, str)) self.violate2(join(bOPEN('dict',1), bINT(1), bSTR("a"), bINT(2), bSTR("b"), bINT(3), bSTR("c"), bCLOSE(1)), ".{}", schema.DictOf(int, str, maxKeys=2)) def TRUE(self): return join(bOPEN("boolean",2), bINT(1), bCLOSE(2)) def FALSE(self): return join(bOPEN("boolean",2), bINT(0), bCLOSE(2)) def testConstrainedBool(self): self.conform2(self.TRUE(), True, bool) self.conform2(self.TRUE(), True, schema.BooleanConstraint()) self.conform2(self.FALSE(), False, schema.BooleanConstraint()) # booleans have ints, not strings. To do otherwise is a protocol # error, not a schema Violation. f = self.shouldDropConnection(join(bOPEN("boolean",1), bSTR("vrai"), bCLOSE(1))) self.failUnlessEqual(f.value.args[0], "BooleanUnslicer only accepts an INT token") # but true/false is a constraint, and is reported with Violation self.violate2(self.TRUE(), ".", schema.BooleanConstraint(False)) self.violate2(self.FALSE(), ".", schema.BooleanConstraint(True)) class A: """ dummy class """ def amethod(self): pass def __cmp__(self, other): if not type(other) == type(self): return -1 return cmp(self.__dict__, other.__dict__) class B(object): # new-style class def amethod(self): pass def __cmp__(self, other): if not type(other) == type(self): return -1 return cmp(self.__dict__, other.__dict__) def afunc(self): pass class ThereAndBackAgain(TestBananaMixin, unittest.TestCase): def test_int(self): d = self.looptest(42) d.addCallback(lambda res: self.looptest(-47)) return d def test_bigint(self): # some of these are small enough to fit in an INT d = self.looptest(int(2**31-1)) # most positive representable number d.addCallback(lambda res: self.looptest(long(2**31+0))) d.addCallback(lambda res: self.looptest(long(2**31+1))) d.addCallback(lambda res: self.looptest(long(-2**31-1))) # the following is the most negative representable number d.addCallback(lambda res: self.looptest(int(-2**31+0))) d.addCallback(lambda res: self.looptest(int(-2**31+1))) d.addCallback(lambda res: self.looptest(long(2**100))) d.addCallback(lambda res: self.looptest(long(-2**100))) d.addCallback(lambda res: self.looptest(long(2**1000))) d.addCallback(lambda res: self.looptest(long(-2**1000))) return d def test_decimal(self): d = self.looptest(Decimal(0)) d.addCallback(lambda res: self.looptest(Decimal(123))) d.addCallback(lambda res: self.looptest(Decimal(-123))) d.addCallback(lambda res: self.looptest(Decimal("123"))) d.addCallback(lambda res: self.looptest(Decimal("-123"))) d.addCallback(lambda res: self.looptest(Decimal("123.456"))) d.addCallback(lambda res: self.looptest(Decimal("-123.456"))) d.addCallback(lambda res: self.looptest(Decimal("0.000003"))) d.addCallback(lambda res: self.looptest(Decimal("-0.000003"))) d.addCallback(lambda res: self.looptest(Decimal('Inf'))) d.addCallback(lambda res: self.looptest(Decimal('-Inf'))) # NaN is a bit weird: by definition, NaN != NaN. So we need to make # sure it serializes, and that str(new) == str(old), but we don't # check that new == old. d.addCallback(lambda res: self.loop(Decimal('NaN'))) def _check_NaN(new_NaN): self.failUnlessEqual(str(new_NaN), str(Decimal('NaN'))) d.addCallback(_check_NaN) return d def test_string(self): return self.looptest("biggles") def test_unicode(self): return self.looptest(u"biggles\u1234") def test_list(self): return self.looptest([1,2]) def test_tuple(self): return self.looptest((1,2)) def test_set(self): d = self.looptest(set([1,2])) d.addCallback(lambda res: self.looptest(frozenset([1,2]))) return d def test_bool(self): d = self.looptest(True) d.addCallback(lambda res: self.looptest(False)) return d def test_float(self): return self.looptest(20.3) def test_none(self): d = self.loop(None) d.addCallback(lambda n2: self.failUnless(n2 is None)) return d def test_dict(self): return self.looptest({'a':1}) def test_func(self): return self.looptest(afunc) def test_module(self): return self.looptest(unittest) def test_instance(self): a = A() return self.looptest(a) def test_instance_newstyle(self): raise unittest.SkipTest("new-style classes still broken") b = B() return self.looptest(b) def test_class(self): return self.looptest(A) def test_class_newstyle(self): raise unittest.SkipTest("new-style classes still broken") return self.looptest(B) def test_boundMethod(self): a = A() m1 = a.amethod d = self.loop(m1) d.addCallback(self._test_boundMethod_1, m1) return d def _test_boundMethod_1(self, m2, m1): self.failUnlessEqual(m1.im_class, m2.im_class) self.failUnlessEqual(m1.im_self, m2.im_self) self.failUnlessEqual(m1.im_func, m2.im_func) def test_boundMethod_newstyle(self): raise unittest.SkipTest("new-style classes still broken") b = B() m1 = b.amethod d = self.loop(m1) d.addCallback(self._test_boundMethod_newstyle, m1) return d def _test_boundMethod_newstyle(self, m2, m1): self.failUnlessEqual(m1.im_class, m2.im_class) self.failUnlessEqual(m1.im_self, m2.im_self) self.failUnlessEqual(m1.im_func, m2.im_func) def test_classMethod(self): return self.looptest(A.amethod) def test_classMethod_newstyle(self): raise unittest.SkipTest("new-style classes still broken") return self.looptest(B.amethod) # some stuff from test_newjelly def testIdentity(self): # test to make sure that objects retain identity properly x = [] y = (x,) x.append(y) x.append(y) self.assertIdentical(x[0], x[1]) self.assertIdentical(x[0][0], x) d = self.encode(x) d.addCallback(self.shouldDecode) d.addCallback(self._testIdentity_1) return d def _testIdentity_1(self, z): self.assertIdentical(z[0], z[1]) self.assertIdentical(z[0][0], z) def testUnicode(self): x = [unicode('blah')] d = self.loop(x) d.addCallback(self._testUnicode_1, x) return d def _testUnicode_1(self, y, x): self.assertEquals(x, y) self.assertEquals(type(x[0]), type(y[0])) def testStressReferences(self): reref = [] toplevelTuple = ({'list': reref}, reref) reref.append(toplevelTuple) d = self.loop(toplevelTuple) d.addCallback(self._testStressReferences_1) return d def _testStressReferences_1(self, z): self.assertIdentical(z[0]['list'], z[1]) self.assertIdentical(z[0]['list'][0], z) def test_cycles_1(self): # a list that contains a tuple that can't be referenced yet a = [] t1 = (a,) t2 = (t1,) a.append(t2) d = self.loop(t1) d.addCallback(lambda z: self.assertIdentical(z[0][0][0], z)) return d def test_cycles_2(self): # a dict that contains a tuple that can't be referenced yet. a = {} t1 = (a,) t2 = (t1,) a['foo'] = t2 d = self.loop(t1) d.addCallback(lambda z: self.assertIdentical(z[0]['foo'][0], z)) return d def test_cycles_3(self): # sets seem to be transitively immutable: any mutable contents would # be unhashable, and sets can only contain hashable objects. # Therefore sets cannot participate in cycles the way that tuples # can. # a set that contains a tuple that can't be referenced yet. You can't # actually create this in python, because you can only create a set # out of hashable objects, and sets aren't hashable, and a tuple that # contains a set is not hashable. a = set() t1 = (a,) t2 = (t1,) a.add(t2) d = self.loop(t1) d.addCallback(lambda z: self.assertIdentical(list(z[0])[0][0], z)) # a list that contains a frozenset that can't be referenced yet a = [] t1 = frozenset([a]) t2 = frozenset([t1]) a.append(t2) d = self.loop(t1) d.addCallback(lambda z: self.assertIdentical(list(list(z)[0][0])[0], z)) # a dict that contains a frozenset that can't be referenced yet. a = {} t1 = frozenset([a]) t2 = frozenset([t1]) a['foo'] = t2 d = self.loop(t1) d.addCallback(lambda z: self.assertIdentical(list(list(z)[0]['foo'])[0], z)) # a set that contains a frozenset that can't be referenced yet. a = set() t1 = frozenset([a]) t2 = frozenset([t1]) a.add(t2) d = self.loop(t1) d.addCallback(lambda z: self.assertIdentical(list(list(list(z)[0])[0])[0], z)) return d del test_cycles_3 class VocabTest1(unittest.TestCase): def test_incoming1(self): b = TokenBanana() b.connectionMade() vdict = {1: 'list', 2: 'tuple', 3: 'dict'} keys = vdict.keys() keys.sort() setVdict = [tOPEN(0),'set-vocab'] for k in keys: setVdict.append(k) setVdict.append(vdict[k]) setVdict.append(tCLOSE(0)) b.dataReceived(untokenize(setVdict)) # banana should now know this vocabulary self.failUnlessEqual(b.incomingVocabulary, vdict) def test_outgoing(self): b = TokenBanana() b.connectionMade() b.tokens = [] strings = ["list", "tuple", "dict"] vdict = {0: 'list', 1: 'tuple', 2: 'dict'} keys = vdict.keys() keys.sort() setVdict = [tOPEN(0),'set-vocab'] for k in keys: setVdict.append(k) setVdict.append(vdict[k]) setVdict.append(tCLOSE(0)) b.setOutgoingVocabulary(strings) vocabTokens = b.tokens self.failUnlessEqual(vocabTokens, setVdict) def test_table_hashes(self): # make sure that we don't change any published vocab tables, and that # we don't change the hash algorithm that they use hash_v0 = vocab.hashVocabTable(0) self.failUnlessEqual(hash_v0, "da39") hash_v1 = vocab.hashVocabTable(1) self.failUnlessEqual(hash_v1, "bb33") class VocabTest2(TestBananaMixin, unittest.TestCase): def vbOPEN(self, count, opentype): num = self.invdict[opentype] return chr(count) + "\x88" + chr(num) + "\x87" def test_loop(self): strings = ["list", "tuple", "dict"] vdict = {0: 'list', 1: 'tuple', 2: 'dict'} self.invdict = dict(zip(vdict.values(), vdict.keys())) self.banana.setOutgoingVocabulary(strings) # this next check only happens to work because there is nothing to # keep serialization from completing synchronously. If Banana # acquires some eventual-sends, this test might need to be rewritten. self.failUnlessEqual(self.banana.outgoingVocabulary, self.invdict) self.shouldDecode(self.banana.transport.getvalue()) self.failUnlessEqual(self.banana.incomingVocabulary, vdict) self.clearOutput() vbOPEN = self.vbOPEN expected = "".join([vbOPEN(1,"list"), vbOPEN(2,"tuple"), vbOPEN(3,"dict"), bSTR('a'), bINT(1), bCLOSE(3), bCLOSE(2), bCLOSE(1)]) d = self.encode([({'a':1},)]) d.addCallback(self.wantEqual, expected) return d class SliceableByItself(slicer.BaseSlicer): def __init__(self, value): self.value = value def slice(self, streamable, banana): self.streamable = streamable # this is our "instance state" yield {"value": self.value} class CouldBeSliceable: def __init__(self, value): self.value = value class _AndICanHelp(slicer.BaseSlicer): def slice(self, streamable, banana): self.streamable = streamable yield {"value": self.obj.value} registerAdapter(_AndICanHelp, CouldBeSliceable, ISlicer) class Sliceable(unittest.TestCase): def setUp(self): self.banana = TokenBanana() self.banana.connectionMade() def do(self, obj): return self.banana.testSlice(obj) def tearDown(self): self.failUnless(len(self.banana.slicerStack) == 1) self.failUnless(isinstance(self.banana.slicerStack[0][0], RootSlicer)) def testDirect(self): # the object is its own Slicer i = SliceableByItself(42) d = self.do(i) d.addCallback(self.failUnlessEqual, [tOPEN(0), tOPEN(1), "dict", "value", 42, tCLOSE(1), tCLOSE(0)]) return d def testAdapter(self): # the adapter is the Slicer i = CouldBeSliceable(43) d = self.do(i) d.addCallback(self.failUnlessEqual, [tOPEN(0), tOPEN(1), "dict", "value", 43, tCLOSE(1), tCLOSE(0)]) return d # TODO: vocab test: # send a bunch of strings # send an object that stalls # send some more strings # set the Vocab table to tokenize some of those strings # send yet more strings # unstall serialization, let everything flow through, verify foolscap-0.13.1/src/foolscap/test/test_call.py0000644000076500000240000010735612774376131021752 0ustar warnerstaff00000000000000 import gc import re import sys if False: from twisted.python import log log.startLogging(sys.stderr) from twisted.python import log from twisted.trial import unittest from twisted.internet.main import CONNECTION_LOST, CONNECTION_DONE from twisted.python.failure import Failure from twisted.application import service from foolscap.tokens import Violation from foolscap.eventual import flushEventualQueue from foolscap.test.common import HelperTarget, TargetMixin, ShouldFailMixin from foolscap.test.common import RIMyTarget, Target, TargetWithoutInterfaces, \ BrokenTarget, MakeTubsMixin from foolscap.api import RemoteException, DeadReferenceError from foolscap.call import CopiedFailure from foolscap.logging import log as flog class Unsendable: pass class TestCall(TargetMixin, ShouldFailMixin, unittest.TestCase): def setUp(self): TargetMixin.setUp(self) self.setupBrokers() def testCall1(self): # this is done without interfaces rr, target = self.setupTarget(TargetWithoutInterfaces()) d = rr.callRemote("add", a=1, b=2) d.addCallback(lambda res: self.failUnlessEqual(res, 3)) d.addCallback(lambda res: self.failUnlessEqual(target.calls, [(1,2)])) d.addCallback(self._testCall1_1, rr) return d def _testCall1_1(self, res, rr): # the caller still holds the RemoteReference self.failUnless(self.callingBroker.yourReferenceByCLID.has_key(1)) # release the RemoteReference. This does two things: 1) the # callingBroker will forget about it. 2) they will send a decref to # the targetBroker so *they* can forget about it. del rr # this fires a DecRef gc.collect() # make sure # we need to give it a moment to deliver the DecRef message and act # on it. Poll until the caller has received it. def _check(): if self.callingBroker.yourReferenceByCLID.has_key(1): return False return True d = self.poll(_check) d.addCallback(self._testCall1_2) return d def _testCall1_2(self, res): self.failIf(self.callingBroker.yourReferenceByCLID.has_key(1)) self.failIf(self.targetBroker.myReferenceByCLID.has_key(1)) def testCall1a(self): # no interfaces, but use positional args rr, target = self.setupTarget(TargetWithoutInterfaces()) d = rr.callRemote("add", 1, 2) d.addCallback(lambda res: self.failUnlessEqual(res, 3)) d.addCallback(lambda res: self.failUnlessEqual(target.calls, [(1,2)])) return d testCall1a.timeout = 2 def testCall1b(self): # no interfaces, use both positional and keyword args rr, target = self.setupTarget(TargetWithoutInterfaces()) d = rr.callRemote("add", 1, b=2) d.addCallback(lambda res: self.failUnlessEqual(res, 3)) d.addCallback(lambda res: self.failUnlessEqual(target.calls, [(1,2)])) return d testCall1b.timeout = 2 def testFail1(self): # this is done without interfaces rr, target = self.setupTarget(TargetWithoutInterfaces()) d = rr.callRemote("fail") self.failIf(target.calls) d.addBoth(self._testFail1_1) return d testFail1.timeout = 2 def _testFail1_1(self, f): # f should be a CopiedFailure self.failUnless(isinstance(f, Failure), "Hey, we didn't fail: %s" % f) self.failUnless(isinstance(f, CopiedFailure), "not CopiedFailure: %s" % f) self.failUnless(f.check(ValueError), "wrong exception type: %s" % f) self.failUnlessSubstring("you asked me to fail", f.value) def testFail2(self): # this is done without interfaces rr, target = self.setupTarget(TargetWithoutInterfaces()) d = rr.callRemote("add", a=1, b=2, c=3) # add() does not take a 'c' argument, so we get a TypeError here self.failIf(target.calls) d.addBoth(self._testFail2_1) return d testFail2.timeout = 2 def _testFail2_1(self, f): self.failUnless(isinstance(f, Failure), "Hey, we didn't fail: %s" % f) self.failUnless(f.check(TypeError), "wrong exception type: %s" % f.type) self.failUnlessSubstring("remote_add() got an unexpected keyword " "argument 'c'", f.value) def testFail3(self): # this is done without interfaces rr, target = self.setupTarget(TargetWithoutInterfaces()) d = rr.callRemote("bogus", a=1, b=2) # the target does not have .bogus method, so we get an AttributeError self.failIf(target.calls) d.addBoth(self._testFail3_1) return d testFail3.timeout = 2 def _testFail3_1(self, f): self.failUnless(isinstance(f, Failure), "Hey, we didn't fail: %s" % f) self.failUnless(f.check(AttributeError), "wrong exception type: %s" % f.type) self.failUnlessSubstring("TargetWithoutInterfaces", str(f)) self.failUnlessSubstring(" has no attribute 'remote_bogus'", str(f)) def testFailStringException(self): # make sure we handle string exceptions correctly if sys.version_info >= (2,5): log.msg("skipping test: string exceptions are deprecated in 2.5") return rr, target = self.setupTarget(TargetWithoutInterfaces()) d = rr.callRemote("failstring") self.failIf(target.calls) d.addBoth(self._testFailStringException_1) return d testFailStringException.timeout = 2 def _testFailStringException_1(self, f): # f should be a CopiedFailure self.failUnless(isinstance(f, Failure), "Hey, we didn't fail: %s" % f) self.failUnless(f.check("string exceptions are annoying"), "wrong exception type: %s" % f) def testCopiedFailure(self): # A calls B, who calls C. C fails. B gets a CopiedFailure and reports # it back to A. What does A get? rr, target = self.setupTarget(TargetWithoutInterfaces()) d = rr.callRemote("fail_remotely", target) def _check(f): # f should be a CopiedFailure self.failUnless(isinstance(f, Failure), "Hey, we didn't fail: %s" % f) self.failUnless(f.check(ValueError), "wrong exception type: %s" % f) self.failUnlessSubstring("you asked me to fail", f.value) d.addBoth(_check) return d testCopiedFailure.timeout = 2 def testCall2(self): # server end uses an interface this time, but not the client end rr, target = self.setupTarget(Target(), True) d = rr.callRemote("add", a=3, b=4, _useSchema=False) # the schema is enforced upon receipt d.addCallback(lambda res: self.failUnlessEqual(res, 7)) return d testCall2.timeout = 2 def testCall3(self): # use interface on both sides rr, target = self.setupTarget(Target(), True) d = rr.callRemote('add', 3, 4) # enforces schemas d.addCallback(lambda res: self.failUnlessEqual(res, 7)) return d testCall3.timeout = 2 def testCall4(self): # call through a manually-defined RemoteMethodSchema rr, target = self.setupTarget(Target(), True) d = rr.callRemote("add", 3, 4, _methodConstraint=RIMyTarget['add1']) d.addCallback(lambda res: self.failUnlessEqual(res, 7)) return d testCall4.timeout = 2 def testChoiceOf(self): # this is a really small test case to check specific bugs. We # definitely need more here. # in bug (#13), the ChoiceOf constraint did not override the # checkToken() call to its children, which had the consequence of not # propagating the maxLength= attribute of the StringConstraint to the # children (using the default of 1000 bytes instead). rr, target = self.setupTarget(HelperTarget()) d = rr.callRemote("choice1", 4) d.addCallback(lambda res: self.failUnlessEqual(res, None)) d.addCallback(lambda res: rr.callRemote("choice1", "a"*2000)) d.addCallback(lambda res: self.failUnlessEqual(res, None)) # False does not conform d.addCallback(lambda res: self.shouldFail(Violation, "testChoiceOf", None, rr.callRemote, "choice1", False)) return d def testMegaSchema(self): # try to exercise all our constraints at once rr, target = self.setupTarget(HelperTarget()) t = (set([1, 2, 3]), "str", True, 12, 12L, 19.3, None, u"unicode", "bytestring", "any", 14.3, 15, "a"*95, "1234567890", ) obj1 = {"key": [t]} obj2 = (set([1,2,3]), [1,2,3], {1:"two"}) d = rr.callRemote("megaschema", obj1, obj2) d.addCallback(lambda res: self.failUnlessEqual(res, None)) return d def testMega3(self): # exercise a specific bug: shared references don't pass schemas t = (0,1) obj = [t, t] rr, target = self.setupTarget(HelperTarget()) d = rr.callRemote("mega3", obj) d.addCallback(lambda res: self.failUnlessEqual(res, None)) return d def testUnconstrainedMethod(self): rr, target = self.setupTarget(Target(), True) d = rr.callRemote('free', 3, 4, x="boo") def _check(res): self.failUnlessEqual(res, "bird") self.failUnlessEqual(target.calls, [((3,4), {"x": "boo"})]) d.addCallback(_check) return d def testFailWrongMethodLocal(self): # the caller knows that this method does not really exist rr, target = self.setupTarget(Target(), True) d = rr.callRemote("bogus") # RIMyTarget doesn't implement .bogus() d.addCallbacks(lambda res: self.fail("should have failed"), self._testFailWrongMethodLocal_1) return d testFailWrongMethodLocal.timeout = 2 def _testFailWrongMethodLocal_1(self, f): self.failUnless(f.check(Violation)) self.failUnless(re.search(r'RIMyTarget\(.*\) does not offer bogus', str(f))) def testFailWrongMethodRemote(self): # if the target doesn't specify any remote interfaces, then the # calling side shouldn't try to do any checking. The problem is # caught on the target side. rr, target = self.setupTarget(Target(), False) d = rr.callRemote("bogus") # RIMyTarget doesn't implement .bogus() d.addCallbacks(lambda res: self.fail("should have failed"), self._testFailWrongMethodRemote_1) return d testFailWrongMethodRemote.timeout = 2 def _testFailWrongMethodRemote_1(self, f): self.failUnless(f.check(Violation)) self.failUnlessSubstring("method 'bogus' not defined in RIMyTarget", str(f)) def testFailWrongMethodRemote2(self): # call a method which doesn't actually exist. The sender thinks # they're ok but the recipient catches the violation rr, target = self.setupTarget(Target(), True) d = rr.callRemote("bogus", _useSchema=False) # RIMyTarget2 has a 'sub' method, but RIMyTarget (the real interface) # does not d.addCallbacks(lambda res: self.fail("should have failed"), self._testFailWrongMethodRemote2_1) d.addCallback(lambda res: self.failIf(target.calls)) return d testFailWrongMethodRemote2.timeout = 2 def _testFailWrongMethodRemote2_1(self, f): self.failUnless(f.check(Violation)) self.failUnless(re.search(r'RIMyTarget\(.*\) does not offer bogus', str(f))) def testFailWrongArgsLocal1(self): # we violate the interface (extra arg), and the sender should catch it rr, target = self.setupTarget(Target(), True) d = rr.callRemote("add", a=1, b=2, c=3) d.addCallbacks(lambda res: self.fail("should have failed"), self._testFailWrongArgsLocal1_1) d.addCallback(lambda res: self.failIf(target.calls)) return d testFailWrongArgsLocal1.timeout = 2 def _testFailWrongArgsLocal1_1(self, f): self.failUnless(f.check(Violation)) self.failUnlessSubstring("unknown argument 'c'", str(f.value)) def testFailWrongArgsLocal2(self): # we violate the interface (bad arg), and the sender should catch it rr, target = self.setupTarget(Target(), True) d = rr.callRemote("add", a=1, b="two") d.addCallbacks(lambda res: self.fail("should have failed"), self._testFailWrongArgsLocal2_1) d.addCallback(lambda res: self.failIf(target.calls)) return d testFailWrongArgsLocal2.timeout = 2 def _testFailWrongArgsLocal2_1(self, f): self.failUnless(f.check(Violation)) self.failUnlessSubstring("not a number", str(f.value)) def testFailWrongArgsRemote1(self): # the sender thinks they're ok but the recipient catches the # violation rr, target = self.setupTarget(Target(), True) d = rr.callRemote("add", a=1, b="foo", _useSchema=False) d.addCallbacks(lambda res: self.fail("should have failed"), self._testFailWrongArgsRemote1_1) d.addCallbacks(lambda res: self.failIf(target.calls)) return d testFailWrongArgsRemote1.timeout = 2 def _testFailWrongArgsRemote1_1(self, f): self.failUnless(f.check(Violation)) self.failUnlessSubstring("STRING token rejected by IntegerConstraint", f.value) self.failUnlessSubstring(".", f.value) def testFailWrongReturnRemote(self): rr, target = self.setupTarget(BrokenTarget(), True) d = rr.callRemote("add", 3, 4) # violates return constraint d.addCallbacks(lambda res: self.fail("should have failed"), self._testFailWrongReturnRemote_1) return d testFailWrongReturnRemote.timeout = 2 def _testFailWrongReturnRemote_1(self, f): self.failUnless(f.check(Violation)) self.failUnlessSubstring("in return value of .add", f.value) self.failUnlessSubstring("not a number", f.value) def testFailWrongReturnLocal(self): # the target returns a value which violates our _resultConstraint rr, target = self.setupTarget(Target(), True) d = rr.callRemote("add", a=1, b=2, _resultConstraint=str) # The target returns an int, which matches the schema they're using, # so they think they're ok. We've overridden our expectations to # require a string. d.addCallbacks(lambda res: self.fail("should have failed"), self._testFailWrongReturnLocal_1) # the method should have been run d.addCallback(lambda res: self.failUnless(target.calls)) return d testFailWrongReturnLocal.timeout = 2 def _testFailWrongReturnLocal_1(self, f): self.failUnless(f.check(Violation)) self.failUnlessSubstring("INT token rejected by ByteStringConstraint", str(f)) self.failUnlessSubstring("in inbound method results", str(f)) self.failUnlessSubstring(".Answer(req=1)", str(f)) def testDefer(self): rr, target = self.setupTarget(HelperTarget()) d = rr.callRemote("defer", obj=12) d.addCallback(lambda res: self.failUnlessEqual(res, 12)) return d testDefer.timeout = 2 def testStallOrdering(self): # if the first message hangs (it returns a Deferred that doesn't fire # for a while), that shouldn't stall the second message. rr, target = self.setupTarget(HelperTarget()) d0 = rr.callRemote("hang") d = rr.callRemote("echo", 1) d.addCallback(lambda res: self.failUnlessEqual(res, 1)) def _done(res): target.d.callback(2) return d0 d.addCallback(_done) return d testStallOrdering.timeout = 5 def testDisconnect_during_call(self): rr, target = self.setupTarget(HelperTarget()) d = rr.callRemote("hang") e = RuntimeError("lost connection") rr.tracker.broker.transport.loseConnection(Failure(e)) d.addCallbacks(lambda res: self.fail("should have failed"), lambda why: why.trap(RuntimeError) and None) return d def test_connection_lost_is_deadref(self): rr, target = self.setupTarget(HelperTarget()) d1 = rr.callRemote("hang") def get_d(): return d1 rr.tracker.broker.transport.loseConnection(Failure(CONNECTION_LOST)) d = self.shouldFail(DeadReferenceError, "lost_is_deadref.1", "Connection was lost", get_d) def _examine_error((f,)): # the (to tubid=XXX) part will see "tub=call", which is an # abbreviation of "callingBroker" as created in # TargetMixin.setupBrokers self.failUnlessIn("(to tubid=call)", str(f.value)) self.failUnlessIn("(during method=None:hang)", str(f.value)) d.addCallback(_examine_error) # and once the connection is down, we should get a DeadReferenceError # for new messages d.addCallback(lambda res: self.shouldFail(DeadReferenceError, "lost_is_deadref.2", "Calling Stale Broker", rr.callRemote, "hang")) return d def test_connection_done_is_deadref(self): rr, target = self.setupTarget(HelperTarget()) d = rr.callRemote("hang") rr.tracker.broker.transport.loseConnection(Failure(CONNECTION_DONE)) d.addCallbacks(lambda res: self.fail("should have failed"), lambda why: why.trap(DeadReferenceError) and None) return d def disconnected(self, *args, **kwargs): self.lost = 1 self.lost_args = (args, kwargs) def testNotifyOnDisconnect(self): rr, target = self.setupTarget(HelperTarget()) self.lost = 0 self.failUnlessEqual(rr.isConnected(), True) rr.notifyOnDisconnect(self.disconnected) rr.tracker.broker.transport.loseConnection(Failure(CONNECTION_LOST)) d = flushEventualQueue() def _check(res): self.failUnlessEqual(rr.isConnected(), False) self.failUnless(self.lost) self.failUnlessEqual(self.lost_args, ((),{})) # it should be safe to unregister now, even though the callback # has already fired, since dontNotifyOnDisconnect is tolerant rr.dontNotifyOnDisconnect(self.disconnected) d.addCallback(_check) return d def testNotifyOnDisconnect_unregister(self): rr, target = self.setupTarget(HelperTarget()) self.lost = 0 m = rr.notifyOnDisconnect(self.disconnected) rr.dontNotifyOnDisconnect(m) # dontNotifyOnDisconnect is supposed to be tolerant of duplicate # unregisters, because otherwise it is hard to avoid race conditions. # Validate that we can unregister something multiple times. rr.dontNotifyOnDisconnect(m) rr.tracker.broker.transport.loseConnection(Failure(CONNECTION_LOST)) d = flushEventualQueue() d.addCallback(lambda res: self.failIf(self.lost)) return d def testNotifyOnDisconnect_args(self): rr, target = self.setupTarget(HelperTarget()) self.lost = 0 rr.notifyOnDisconnect(self.disconnected, "arg", foo="kwarg") rr.tracker.broker.transport.loseConnection(Failure(CONNECTION_LOST)) d = flushEventualQueue() def _check(res): self.failUnless(self.lost) self.failUnlessEqual(self.lost_args, (("arg",), {"foo": "kwarg"})) d.addCallback(_check) return d def testNotifyOnDisconnect_already(self): # make sure notifyOnDisconnect works even if the reference was already # broken rr, target = self.setupTarget(HelperTarget()) self.lost = 0 rr.tracker.broker.transport.loseConnection(Failure(CONNECTION_LOST)) d = flushEventualQueue() d.addCallback(lambda res: rr.notifyOnDisconnect(self.disconnected)) d.addCallback(lambda res: flushEventualQueue()) def _check(res): self.failUnless(self.lost, "disconnect handler not run") self.failUnlessEqual(self.lost_args, ((),{})) d.addCallback(_check) return d def testUnsendable(self): rr, target = self.setupTarget(HelperTarget()) d = rr.callRemote("set", obj=Unsendable()) d.addCallbacks(lambda res: self.fail("should have failed"), self._testUnsendable_1) return d testUnsendable.timeout = 2 def _testUnsendable_1(self, why): self.failUnless(why.check(Violation)) self.failUnlessSubstring("cannot serialize", why.value.args[0]) def test_bad_clid(self): rr, target = self.setupTarget(HelperTarget()) clid = rr.tracker.clid del self.targetBroker.myReferenceByCLID[clid] d = rr.callRemote("set", obj=1) d.addCallbacks(lambda res: self.fail("should have failed"), lambda why: [why]) def _check(res): f = res[0] if not f.check(Violation): self.fail("expected Violation, got %s" % f) self.failUnless("unknown CLID %d" % clid in str(f)) d.addCallback(_check) return d class TestCallOnly(TargetMixin, unittest.TestCase): def setUp(self): TargetMixin.setUp(self) self.setupBrokers() def testCallOnly(self): rr, target = self.setupTarget(TargetWithoutInterfaces()) ret = rr.callRemoteOnly("add", a=1, b=2) self.failUnlessIdentical(ret, None) # since we don't have a Deferred to wait upon, we just have to poll # for the call to take place. It should happen pretty quickly. def _check(): if target.calls: self.failUnlessEqual(target.calls, [(1,2)]) return True return False d = self.poll(_check) return d testCallOnly.timeout = 2 class ExamineFailuresMixin: def _examine_raise(self, r, should_be_remote): f = r[0] if should_be_remote: self.failUnless(f.check(RemoteException)) self.failIf(f.check(ValueError)) f2 = f.value.failure else: self.failUnless(f.check(ValueError)) self.failIf(f.check(RemoteException)) f2 = f self.failUnless(f2.check(ValueError)) self.failUnless(isinstance(f2, CopiedFailure)) self.failUnlessSubstring("you asked me to fail", f2.value) self.failIf(f2.check(RemoteException)) l = flog.FoolscapLogger() l.msg("f1", failure=f) l.msg("f2", failure=f2) def _examine_local_violation(self, r): f = r[0] self.failUnless(f.check(Violation)) self.failUnless(re.search(r'RIMyTarget\(.*\) does not offer bogus', str(f))) self.failIf(f.check(RemoteException)) def _examine_remote_violation(self, r, should_be_remote): f = r[0] if should_be_remote: self.failUnless(f.check(RemoteException)) self.failIf(f.check(Violation)) f2 = f.value.failure else: self.failIf(f.check(RemoteException)) self.failUnless(f.check(Violation)) f2 = f self.failUnless(isinstance(f2, CopiedFailure)) self.failUnless(f2.check(Violation)) self.failUnlessSubstring("STRING token rejected by IntegerConstraint", f2.value) self.failUnlessSubstring(".", f2.value) self.failIf(f2.check(RemoteException)) def _examine_remote_attribute_error(self, r, should_be_remote): f = r[0] if should_be_remote: self.failUnless(f.check(RemoteException)) self.failIf(f.check(AttributeError)) f2 = f.value.failure else: self.failUnless(f.check(AttributeError)) self.failIf(f.check(RemoteException)) f2 = f self.failUnless(isinstance(f2, CopiedFailure)) self.failUnless(f2.check(AttributeError)) self.failUnlessSubstring(" has no attribute 'remote_bogus'", str(f2)) self.failIf(f2.check(RemoteException)) def _examine_local_return_violation(self, r): f = r[0] self.failUnless(f.check(Violation)) self.failUnlessSubstring("INT token rejected by ByteStringConstraint", str(f)) self.failUnlessSubstring("in inbound method results", str(f)) self.failUnlessSubstring(".Answer(req=1)", str(f)) self.failIf(f.check(RemoteException)) class Failures(ExamineFailuresMixin, TargetMixin, ShouldFailMixin, unittest.TestCase): def setUp(self): TargetMixin.setUp(self) self.setupBrokers() def _set_expose(self, value): self.callingBroker._expose_remote_exception_types = value def test_raise_not_exposed(self): self._set_expose(False) rr, target = self.setupTarget(TargetWithoutInterfaces()) d = self.shouldFail(RemoteException, "one", None, rr.callRemote, "fail") d.addCallback(self._examine_raise, True) return d def test_raise_yes_exposed(self): self._set_expose(True) rr, target = self.setupTarget(TargetWithoutInterfaces()) d = self.shouldFail(ValueError, "one", None, rr.callRemote, "fail") d.addCallback(self._examine_raise, False) return d def test_raise_default(self): # current default is to expose exceptions. This may change in the # future. rr, target = self.setupTarget(TargetWithoutInterfaces()) d = self.shouldFail(ValueError, "one", None, rr.callRemote, "fail") d.addCallback(self._examine_raise, False) return d def test_local_violation_not_exposed(self): self._set_expose(False) # the caller knows that this method does not really exist, so we # should get a local Violation. Local exceptions are never reported # as RemoteExceptions, so the expose option doesn't affect behavior. rr, target = self.setupTarget(Target(), True) d = self.shouldFail(Violation, "one", None, rr.callRemote, "bogus") d.addCallback(self._examine_local_violation) return d def test_local_violation_yes_exposed(self): self._set_expose(True) rr, target = self.setupTarget(Target(), True) d = self.shouldFail(Violation, "one", None, rr.callRemote, "bogus") d.addCallback(self._examine_local_violation) return d def test_local_violation_default(self): rr, target = self.setupTarget(Target(), True) d = self.shouldFail(Violation, "one", None, rr.callRemote, "bogus") d.addCallback(self._examine_local_violation) return d def test_remote_violation_not_exposed(self): self._set_expose(False) # the sender thinks they're ok, but the recipient catches the # violation. rr, target = self.setupTarget(Target(), True) d = self.shouldFail(RemoteException, "one", None, rr.callRemote, "add", a=1,b="foo", _useSchema=False) d.addCallback(self._examine_remote_violation, True) return d def test_remote_violation_yes_exposed(self): self._set_expose(True) rr, target = self.setupTarget(Target(), True) d = self.shouldFail(Violation, "one", None, rr.callRemote, "add", a=1,b="foo", _useSchema=False) d.addCallback(self._examine_remote_violation, False) return d def test_remote_violation_default(self): rr, target = self.setupTarget(Target(), True) d = self.shouldFail(Violation, "one", None, rr.callRemote, "add", a=1,b="foo", _useSchema=False) d.addCallback(self._examine_remote_violation, False) return d def test_remote_attribute_error_not_exposed(self): self._set_expose(False) # the target doesn't specify an interface, so the sender can't know # that the method is missing rr, target = self.setupTarget(TargetWithoutInterfaces()) d = self.shouldFail(RemoteException, "one", None, rr.callRemote, "bogus") d.addCallback(self._examine_remote_attribute_error, True) return d def test_remote_attribute_error_yes_exposed(self): self._set_expose(True) rr, target = self.setupTarget(TargetWithoutInterfaces()) d = self.shouldFail(AttributeError, "one", None, rr.callRemote, "bogus") d.addCallback(self._examine_remote_attribute_error, False) return d def test_remote_attribute_error_default(self): rr, target = self.setupTarget(TargetWithoutInterfaces()) d = self.shouldFail(AttributeError, "one", None, rr.callRemote, "bogus") d.addCallback(self._examine_remote_attribute_error, False) return d def test_local_return_violation_not_exposed(self): self._set_expose(False) # the target returns a value which violations our _resultConstraint # Local exceptions are never reported as RemoteExceptions, so the # expose option doesn't affect behavior. rr, target = self.setupTarget(Target(), True) d = self.shouldFail(Violation, "one", None, rr.callRemote, "add", a=1, b=2, _resultConstraint=str) d.addCallback(self._examine_local_return_violation) return d def test_local_return_violation_yes_exposed(self): self._set_expose(True) rr, target = self.setupTarget(Target(), True) d = self.shouldFail(Violation, "one", None, rr.callRemote, "add", a=1, b=2, _resultConstraint=str) d.addCallback(self._examine_local_return_violation) return d def test_local_return_violation_default(self): rr, target = self.setupTarget(Target(), True) d = self.shouldFail(Violation, "one", None, rr.callRemote, "add", a=1, b=2, _resultConstraint=str) d.addCallback(self._examine_local_return_violation) return d # TODO: test Tub.setOption("expose-remote-exception-types") # TODO: A calls B. B calls C. C raises an exception. What does A get? class TubFailures(ExamineFailuresMixin, ShouldFailMixin, MakeTubsMixin, unittest.TestCase): def setUp(self): self.s = service.MultiService() self.target_tub, self.source_tub = self.makeTubs(2) self.target_tub.setServiceParent(self.s) self.source_tub.setServiceParent(self.s) def tearDown(self): return self.s.stopService() def setupTarget(self, target): furl = self.target_tub.registerReference(target) d = self.source_tub.getReference(furl) return d def test_raise_not_exposed(self): self.source_tub.setOption("expose-remote-exception-types", False) d = self.setupTarget(TargetWithoutInterfaces()) d.addCallback(lambda rr: self.shouldFail(RemoteException, "one", None, rr.callRemote, "fail")) d.addCallback(self._examine_raise, True) return d def test_raise_yes_exposed(self): self.source_tub.setOption("expose-remote-exception-types", True) d = self.setupTarget(TargetWithoutInterfaces()) d.addCallback(lambda rr: self.shouldFail(ValueError, "one", None, rr.callRemote, "fail")) d.addCallback(self._examine_raise, False) return d def test_raise_default(self): # current default is to expose exceptions. This may change in the # future. d = self.setupTarget(TargetWithoutInterfaces()) d.addCallback(lambda rr: self.shouldFail(ValueError, "one", None, rr.callRemote, "fail")) d.addCallback(self._examine_raise, False) return d class ReferenceCounting(ShouldFailMixin, MakeTubsMixin, unittest.TestCase): def setUp(self): self.s = service.MultiService() self.target_tub, self.source_tub = self.makeTubs(2) self.target_tub.setServiceParent(self.s) self.source_tub.setServiceParent(self.s) def tearDown(self): return self.s.stopService() def setupTarget(self, target): furl = self.target_tub.registerReference(target) d = self.source_tub.getReference(furl) return d def test_reference_counting(self): self.source_tub.setOption("expose-remote-exception-types", True) target = HelperTarget() d = self.setupTarget(target) def _stash(rref): # to exercise bug #104, we need to trigger remote Violations, so # we tell the sending side to not use a RemoteInterface. We do # this by reaching inside the RemoteReference and making it # forget rref.tracker.interfaceName = None rref.tracker.interface = None self.rref = rref d.addCallback(_stash) # the first call causes an error, which discards all remaining # tokens, including the OPEN tokens for the arguments. The #104 bug # is that this causes the open-count to get out of sync, by -2 (one # for the arguments sequence, one for the list inside it). d.addCallback(lambda ign: self.shouldFail(Violation, "one", None, self.rref.callRemote, "bogus", ["one list"])) #d.addCallback(lambda ign: # self.rref.callRemote("set", ["one list"])) # a method call that has no arguments (specifically no REFERENCE # sequences) won't notice the loss of sync d.addCallback(lambda ign: self.rref.callRemote("set", 42)) def _check_42(ign): self.failUnlessEqual(target.obj, 42) d.addCallback(_check_42) # but when the call takes shared arguments, sync matters l = ["list", 1, 2] s = set([3,4]) t = ("tuple", 5, 6) d.addCallback(lambda ign: self.rref.callRemote("set", [t, l, s, t])) def _check_shared(ign): # the off-by-two bug would cause the second tuple shared-ref to # point at the set instead of the first tuple self.failUnlessEqual(type(target.obj), list) one, two, three, four = target.obj self.failUnlessEqual(type(one), tuple) self.failUnlessEqual(one, t) self.failUnlessEqual(type(two), list) self.failUnlessEqual(two, l) self.failUnlessEqual(type(three), set) self.failUnlessEqual(three, s) self.failUnlessEqual(type(four), tuple) # this is where it fails self.failUnlessEqual(four, t) self.failUnlessIdentical(one, four) d.addCallback(_check_shared) return d foolscap-0.13.1/src/foolscap/test/test_connection.py0000644000076500000240000007136013204160675023162 0ustar warnerstaff00000000000000import os import mock from zope.interface import implementer from twisted.trial import unittest from twisted.internet import endpoints, defer, reactor from twisted.internet.endpoints import clientFromString from twisted.internet.defer import inlineCallbacks from twisted.internet.interfaces import IStreamClientEndpoint from twisted.application import service import txtorcon from txsocksx.client import SOCKS5ClientEndpoint from foolscap.api import Tub from foolscap.info import ConnectionInfo from foolscap.connection import get_endpoint from foolscap.connections import tcp, socks, tor, i2p from foolscap.tokens import NoLocationHintsError from foolscap.ipb import InvalidHintError from foolscap.test.common import (certData_low, certData_high, Target, ShouldFailMixin) from foolscap import ipb, util def discard_status(status): pass @implementer(IStreamClientEndpoint) class FakeHostnameEndpoint: def __init__(self, reactor, host, port): self.reactor = reactor self.host = host self.port = port class Convert(unittest.TestCase): def checkTCPEndpoint(self, hint, expected_host, expected_port): with mock.patch("foolscap.connections.tcp.HostnameEndpoint", side_effect=FakeHostnameEndpoint): d = get_endpoint(hint, {"tcp": tcp.default()}, ConnectionInfo()) (ep, host) = self.successResultOf(d) self.failUnless(isinstance(ep, FakeHostnameEndpoint), ep) self.failUnlessIdentical(ep.reactor, reactor) self.failUnlessEqual(ep.host, expected_host) self.failUnlessEqual(ep.port, expected_port) def checkBadTCPEndpoint(self, hint): d = get_endpoint(hint, {"tcp": tcp.default()}, ConnectionInfo()) self.failureResultOf(d, ipb.InvalidHintError) def checkUnknownEndpoint(self, hint): d = get_endpoint(hint, {"tcp": tcp.default()}, ConnectionInfo()) self.failureResultOf(d, ipb.InvalidHintError) def testConvertLegacyHint(self): self.failUnlessEqual(tcp.convert_legacy_hint("127.0.0.1:9900"), "tcp:127.0.0.1:9900") self.failUnlessEqual(tcp.convert_legacy_hint("tcp:127.0.0.1:9900"), "tcp:127.0.0.1:9900") self.failUnlessEqual(tcp.convert_legacy_hint("other:127.0.0.1:9900"), "other:127.0.0.1:9900") # this is unfortunate self.failUnlessEqual(tcp.convert_legacy_hint("unix:1"), "tcp:unix:1") # so new hints should do one of these: self.failUnlessEqual(tcp.convert_legacy_hint("tor:host:1234"), "tor:host:1234") # multiple colons self.failUnlessEqual(tcp.convert_legacy_hint("unix:fd=1"), "unix:fd=1") # equals signs, key=value -style def testTCP(self): self.checkTCPEndpoint("tcp:127.0.0.1:9900", "127.0.0.1", 9900) self.checkTCPEndpoint("tcp:hostname:9900", "hostname", 9900) self.checkBadTCPEndpoint("tcp:hostname:NOTAPORT") def testLegacyTCP(self): self.checkTCPEndpoint("127.0.0.1:9900", "127.0.0.1", 9900) self.checkTCPEndpoint("hostname:9900", "hostname", 9900) self.checkBadTCPEndpoint("hostname:NOTAPORT") def testTCP6(self): self.checkTCPEndpoint("tcp:[2001:0DB8:f00e:eb00::1]:9900", "2001:0DB8:f00e:eb00::1", 9900) self.checkBadTCPEndpoint("tcp:[2001:0DB8:f00e:eb00::1]:NOTAPORT") self.checkBadTCPEndpoint("tcp:2001:0DB8:f00e:eb00::1]:9900") self.checkBadTCPEndpoint("tcp:[2001:0DB8:f00e:eb00::1:9900") self.checkBadTCPEndpoint("tcp:2001:0DB8:f00e:eb00::1:9900") # IPv4-mapped addresses self.checkTCPEndpoint("tcp:[::FFFF:1.2.3.4]:99", "::FFFF:1.2.3.4", 99) self.checkBadTCPEndpoint("tcp:[::FFFF:1.2.3]:99") self.checkBadTCPEndpoint("tcp:[::FFFF:1.2.3.4567]:99") # local-scoped address with good/bad zone-ids (like "123" or "en0") self.checkTCPEndpoint("tcp:[FE8::1%123]:9900", "FE8::1%123", 9900) self.checkTCPEndpoint("tcp:[FE8::1%en1.2]:9900", "FE8::1%en1.2", 9900) self.checkBadTCPEndpoint("tcp:[FE8::1%%]:9900") self.checkBadTCPEndpoint("tcp:[FE8::1%$]:9900") self.checkBadTCPEndpoint("tcp:[FE8::1%]:9900") self.checkBadTCPEndpoint("tcp:[FE8::1%en0%nomultiple]:9900") # not both IPv4-mapped and zone-id self.checkBadTCPEndpoint("tcp:[::FFFF:1.2.3.4%en0]:9900") def testNoColon(self): self.checkBadTCPEndpoint("hostname") def testExtensionsFromFuture(self): self.checkUnknownEndpoint("udp:127.0.0.1:7700") self.checkUnknownEndpoint("127.0.0.1:7700:postextension") @implementer(ipb.IConnectionHintHandler) class NewHandler: def __init__(self): self.asked = 0 self.accepted = 0 def hint_to_endpoint(self, hint, reactor, update_status): self.asked += 1 if "bad" in hint: raise ipb.InvalidHintError self.accepted += 1 pieces = hint.split(":") new_hint = "tcp:%s:%d" % (pieces[1], int(pieces[2])+0) ep = tcp.default().hint_to_endpoint(new_hint, reactor, update_status) if pieces[0] == "slow": update_status("being slow") self._d = defer.Deferred() self._d.addCallback(lambda _: ep) return self._d return ep class ErrorSuffix(unittest.TestCase): def test_context(self): statuses = [] with tor.add_context(statuses.append, "context"): pass self.assertEqual(statuses, ["context"]) statuses = [] def _try(): with tor.add_context(statuses.append, "context"): raise ValueError("foo") e = self.assertRaises(ValueError, _try) self.assertEqual(statuses, ["context"]) self.assert_(hasattr(e, "foolscap_connection_handler_error_suffix")) self.assertEqual(e.foolscap_connection_handler_error_suffix, " (while context)") class Handlers(ShouldFailMixin, unittest.TestCase): def setUp(self): self.s = service.MultiService() self.s.startService() def tearDown(self): return self.s.stopService() def makeTub(self, hint_type): tubA = Tub(certData=certData_low) tubA.setServiceParent(self.s) tubB = Tub(certData=certData_high) tubB.setServiceParent(self.s) portnum = util.allocate_tcp_port() tubA.listenOn("tcp:%d:interface=127.0.0.1" % portnum) tubA.setLocation("%s:127.0.0.1:%d" % (hint_type, portnum)) furl = tubA.registerReference(Target()) return furl, tubB def testNoHandlers(self): furl, tubB = self.makeTub("type2") tubB.removeAllConnectionHintHandlers() d = tubB.getReference(furl) self.failureResultOf(d, NoLocationHintsError) def testNoSuccessfulHandlers(self): furl, tubB = self.makeTub("type2") d = self.shouldFail(NoLocationHintsError, "no handlers", None, tubB.getReference, furl) return d def testExtraHandler(self): furl, tubB = self.makeTub("type2") h = NewHandler() tubB.addConnectionHintHandler("type2", h) d = tubB.getReference(furl) def _got(rref): self.failUnlessEqual(h.asked, 1) self.failUnlessEqual(h.accepted, 1) d.addCallback(_got) return d def testOnlyHandler(self): furl, tubB = self.makeTub("type2") h = NewHandler() tubB.removeAllConnectionHintHandlers() tubB.addConnectionHintHandler("type2", h) d = tubB.getReference(furl) def _got(rref): self.failUnlessEqual(h.asked, 1) self.failUnlessEqual(h.accepted, 1) d.addCallback(_got) return d def testOrdering(self): furl, tubB = self.makeTub("type2") h1 = NewHandler() h2 = NewHandler() tubB.removeAllConnectionHintHandlers() tubB.addConnectionHintHandler("type2", h1) # replaced by h2 tubB.addConnectionHintHandler("type2", h2) d = tubB.getReference(furl) def _got(rref): self.failUnlessEqual(h1.asked, 0) self.failUnlessEqual(h1.accepted, 0) self.failUnlessEqual(h2.asked, 1) self.failUnlessEqual(h2.accepted, 1) d.addCallback(_got) return d def testUnhelpfulHandlers(self): furl, tubB = self.makeTub("type2") h1 = NewHandler() h2 = NewHandler() tubB.removeAllConnectionHintHandlers() tubB.addConnectionHintHandler("type1", h1) # this is ignored tubB.addConnectionHintHandler("type2", h2) # this handles it d = tubB.getReference(furl) def _got(rref): self.failUnlessEqual(h1.asked, 0) self.failUnlessEqual(h1.accepted, 0) self.failUnlessEqual(h2.asked, 1) self.failUnlessEqual(h2.accepted, 1) d.addCallback(_got) return d def testDeferredHandler(self): furl, tubB = self.makeTub("slow") h = NewHandler() tubB.removeAllConnectionHintHandlers() tubB.addConnectionHintHandler("slow", h) d = tubB.getReference(furl) self.assertNoResult(d) h._d.callback(None) def _got(rref): self.failUnlessEqual(h.asked, 1) self.failUnlessEqual(h.accepted, 1) d.addCallback(_got) return d class Socks(unittest.TestCase): @mock.patch("foolscap.connections.socks.SOCKS5ClientEndpoint") def test_ep(self, scep): proxy_ep = FakeHostnameEndpoint(reactor, "localhost", 8080) h = socks.socks_endpoint(proxy_ep) rv = scep.return_value = mock.Mock() ep, host = h.hint_to_endpoint("tor:example.com:1234", reactor, discard_status) self.assertEqual(scep.mock_calls, [mock.call("example.com", 1234, proxy_ep)]) self.assertIdentical(ep, rv) self.assertEqual(host, "example.com") def test_real_ep(self): proxy_ep = FakeHostnameEndpoint(reactor, "localhost", 8080) h = socks.socks_endpoint(proxy_ep) ep, host = h.hint_to_endpoint("tcp:example.com:1234", reactor, discard_status) self.assertIsInstance(ep, SOCKS5ClientEndpoint) self.assertEqual(host, "example.com") def test_bad_hint(self): proxy_ep = FakeHostnameEndpoint(reactor, "localhost", 8080) h = socks.socks_endpoint(proxy_ep) # legacy hints will be upgraded before the connection handler is # invoked, so the handler should not handle them self.assertRaises(ipb.InvalidHintError, h.hint_to_endpoint, "example.com:1234", reactor, discard_status) self.assertRaises(ipb.InvalidHintError, h.hint_to_endpoint, "tcp:example.com:noport", reactor, discard_status) self.assertRaises(ipb.InvalidHintError, h.hint_to_endpoint, "tcp:@:1234", reactor, discard_status) class Empty: pass class Tor(unittest.TestCase): @inlineCallbacks def test_default_socks(self): with mock.patch("foolscap.connections.tor.txtorcon.TorClientEndpoint" ) as tce: tce.return_value = expected_ep = object() h = tor.default_socks() res = yield h.hint_to_endpoint("tcp:example.com:1234", reactor, discard_status) self.assertEqual(tce.mock_calls, [mock.call("example.com", 1234, socks_endpoint=None)]) ep, host = res self.assertIdentical(ep, expected_ep) self.assertEqual(host, "example.com") @inlineCallbacks def test_default_socks_real(self): h = tor.default_socks() res = yield h.hint_to_endpoint("tcp:example.com:1234", reactor, discard_status) ep, host = res self.assertIsInstance(ep, txtorcon.endpoints.TorClientEndpoint) self.assertEqual(host, "example.com") self.assertEqual(h.describe(), "tor") def test_badaddr(self): isnon = tor.is_non_public_numeric_address self.assertTrue(isnon("10.0.0.1")) self.assertTrue(isnon("127.0.0.1")) self.assertTrue(isnon("192.168.78.254")) self.assertTrue(isnon("::1")) self.assertFalse(isnon("8.8.8.8")) self.assertFalse(isnon("example.org")) @inlineCallbacks def test_default_socks_badaddr(self): h = tor.default_socks() d = h.hint_to_endpoint("tcp:10.0.0.1:1234", reactor, discard_status) f = yield self.assertFailure(d, InvalidHintError) self.assertEqual(str(f), "ignoring non-Tor-able ipaddr 10.0.0.1") d = h.hint_to_endpoint("tcp:127.0.0.1:1234", reactor, discard_status) f = yield self.assertFailure(d, InvalidHintError) self.assertEqual(str(f), "ignoring non-Tor-able ipaddr 127.0.0.1") d = h.hint_to_endpoint("tcp:not@a@hint:123", reactor, discard_status) f = yield self.assertFailure(d, InvalidHintError) self.assertEqual(str(f), "unrecognized TCP/Tor hint") @inlineCallbacks def test_socks_endpoint(self): tor_socks_endpoint = clientFromString(reactor, "tcp:socks_host:100") with mock.patch("foolscap.connections.tor.txtorcon.TorClientEndpoint" ) as tce: tce.return_value = expected_ep = object() h = tor.socks_endpoint(tor_socks_endpoint) res = yield h.hint_to_endpoint("tcp:example.com:1234", reactor, discard_status) self.assertEqual(tce.mock_calls, [mock.call("example.com", 1234, socks_endpoint=tor_socks_endpoint)]) ep, host = res self.assertIs(ep, expected_ep) self.assertEqual(host, "example.com") @inlineCallbacks def test_socks_endpoint_real(self): tor_socks_endpoint = clientFromString(reactor, "tcp:socks_host:100") h = tor.socks_endpoint(tor_socks_endpoint) res = yield h.hint_to_endpoint("tcp:example.com:1234", reactor, discard_status) ep, host = res self.assertIsInstance(ep, txtorcon.endpoints.TorClientEndpoint) self.assertEqual(host, "example.com") @inlineCallbacks def test_launch(self): tpp = Empty() tpp.tor_protocol = None h = tor.launch() fake_reactor = object() with mock.patch("txtorcon.launch_tor", return_value=tpp) as lt: res = yield h.hint_to_endpoint("tor:foo.onion:29212", fake_reactor, discard_status) self.assertEqual(len(lt.mock_calls), 1) args,kwargs = lt.mock_calls[0][1:] self.assertIdentical(args[0], h.config) self.assertIdentical(args[1], fake_reactor) self.assertEqual(kwargs, {"tor_binary": None}) ep, host = res self.assertIsInstance(ep, txtorcon.endpoints.TorClientEndpoint) self.assertEqual(host, "foo.onion") # launch_tor will allocate a local TCP port for SOCKS self.assert_(h._socks_desc.startswith("tcp:127.0.0.1:"), h._socks_desc) @inlineCallbacks def test_launch_tor_binary(self): tpp = Empty() tpp.tor_protocol = None h = tor.launch(tor_binary="/bin/tor") fake_reactor = object() with mock.patch("txtorcon.launch_tor", return_value=tpp) as lt: res = yield h.hint_to_endpoint("tor:foo.onion:29212", fake_reactor, discard_status) self.assertEqual(len(lt.mock_calls), 1) args,kwargs = lt.mock_calls[0][1:] self.assertIdentical(args[0], h.config) self.assertIdentical(args[1], fake_reactor) self.assertEqual(kwargs, {"tor_binary": "/bin/tor"}) ep, host = res self.assertIsInstance(ep, txtorcon.endpoints.TorClientEndpoint) self.assertEqual(host, "foo.onion") self.assert_(h._socks_desc.startswith("tcp:127.0.0.1:"), h._socks_desc) @inlineCallbacks def test_launch_data_directory(self): datadir = self.mktemp() tpp = Empty() tpp.tor_protocol = None h = tor.launch(data_directory=datadir) fake_reactor = object() with mock.patch("txtorcon.launch_tor", return_value=tpp) as lt: res = yield h.hint_to_endpoint("tor:foo.onion:29212", fake_reactor, discard_status) self.assertEqual(len(lt.mock_calls), 1) args,kwargs = lt.mock_calls[0][1:] self.assertIdentical(args[0], h.config) self.assertIdentical(args[1], fake_reactor) self.assertEqual(kwargs, {"tor_binary": None}) self.assertEqual(h.config.DataDirectory, datadir) ep, host = res self.assertIsInstance(ep, txtorcon.endpoints.TorClientEndpoint) self.assertEqual(host, "foo.onion") self.assert_(h._socks_desc.startswith("tcp:127.0.0.1:"), h._socks_desc) @inlineCallbacks def test_launch_data_directory_exists(self): datadir = self.mktemp() os.mkdir(datadir) tpp = Empty() tpp.tor_protocol = None h = tor.launch(data_directory=datadir) fake_reactor = object() with mock.patch("txtorcon.launch_tor", return_value=tpp) as lt: res = yield h.hint_to_endpoint("tor:foo.onion:29212", fake_reactor, discard_status) self.assertEqual(len(lt.mock_calls), 1) args,kwargs = lt.mock_calls[0][1:] self.assertIdentical(args[0], h.config) self.assertIdentical(args[1], fake_reactor) self.assertEqual(kwargs, {"tor_binary": None}) self.assertEqual(h.config.DataDirectory, datadir) ep, host = res self.assertIsInstance(ep, txtorcon.endpoints.TorClientEndpoint) self.assertEqual(host, "foo.onion") self.assert_(h._socks_desc.startswith("tcp:127.0.0.1:"), h._socks_desc) @inlineCallbacks def test_control_endpoint(self): control_ep = FakeHostnameEndpoint(reactor, "localhost", 9051) h = tor.control_endpoint(control_ep) # We don't actually care about the generated endpoint, just the state # that the handler builds up internally. But we need to provoke a # connection to build that state, and we need to prevent the handler # from actually talking to a Tor daemon (which probably doesn't exist # on this host). config = Empty() config.SocksPort = ["1234"] with mock.patch("txtorcon.build_tor_connection", return_value=None): with mock.patch("txtorcon.TorConfig.from_protocol", return_value=config): res = yield h.hint_to_endpoint("tor:foo.onion:29212", reactor, discard_status) ep, host = res self.assertIsInstance(ep, txtorcon.endpoints.TorClientEndpoint) self.assertEqual(host, "foo.onion") self.assertEqual(h._socks_desc, "tcp:127.0.0.1:1234") @inlineCallbacks def test_control_endpoint_default(self): control_ep = FakeHostnameEndpoint(reactor, "localhost", 9051) h = tor.control_endpoint(control_ep) config = Empty() config.SocksPort = [txtorcon.DEFAULT_VALUE] with mock.patch("txtorcon.build_tor_connection", return_value=None): with mock.patch("txtorcon.TorConfig.from_protocol", return_value=config): res = yield h.hint_to_endpoint("tor:foo.onion:29212", reactor, discard_status) ep, host = res self.assertIsInstance(ep, txtorcon.endpoints.TorClientEndpoint) self.assertEqual(host, "foo.onion") self.assertEqual(h._socks_desc, "tcp:127.0.0.1:9050") @inlineCallbacks def test_control_endpoint_non_numeric(self): control_ep = FakeHostnameEndpoint(reactor, "localhost", 9051) h = tor.control_endpoint(control_ep) config = Empty() config.SocksPort = ["unix:var/run/tor/socks WorldWritable", "1234"] with mock.patch("txtorcon.build_tor_connection", return_value=None): with mock.patch("txtorcon.TorConfig.from_protocol", return_value=config): res = yield h.hint_to_endpoint("tor:foo.onion:29212", reactor, discard_status) ep, host = res self.assertIsInstance(ep, txtorcon.endpoints.TorClientEndpoint) self.assertEqual(host, "foo.onion") self.assertEqual(h._socks_desc, "tcp:127.0.0.1:1234") @inlineCallbacks def test_control_endpoint_no_port(self): control_ep = FakeHostnameEndpoint(reactor, "localhost", 9051) h = tor.control_endpoint(control_ep) config = Empty() config.SocksPort = ["unparseable"] with mock.patch("txtorcon.build_tor_connection", return_value=None): with mock.patch("txtorcon.TorConfig.from_protocol", return_value=config): d = h.hint_to_endpoint("tor:foo.onion:29212", reactor, discard_status) f = yield self.assertFailure(d, ValueError) self.assertIn("could not use config.SocksPort", str(f)) def test_control_endpoint_maker_immediate(self): return self.do_test_control_endpoint_maker(False) def test_control_endpoint_maker_deferred(self): return self.do_test_control_endpoint_maker(True) def test_control_endpoint_maker_nostatus(self): return self.do_test_control_endpoint_maker(True, takes_status=False) @inlineCallbacks def do_test_control_endpoint_maker(self, use_deferred, takes_status=True): control_ep = FakeHostnameEndpoint(reactor, "localhost", 9051) results = [] def make(arg): results.append(arg) if use_deferred: return defer.succeed(control_ep) else: return control_ep # immediate def make_takes_status(arg, update_status): return make(arg) if takes_status: h = tor.control_endpoint_maker(make_takes_status, takes_status=True) else: h = tor.control_endpoint_maker(make, takes_status=False) self.assertEqual(results, []) # not called yet # We don't actually care about the generated endpoint, just the state # that the handler builds up internally. But we need to provoke a # connection to build that state, and we need to prevent the handler # from actually talking to a Tor daemon (which probably doesn't exist # on this host). config = Empty() config.SocksPort = ["1234"] with mock.patch("txtorcon.build_tor_connection", return_value=None): with mock.patch("txtorcon.TorConfig.from_protocol", return_value=config): res = yield h.hint_to_endpoint("tor:foo.onion:29212", reactor, discard_status) self.assertEqual(results, [reactor]) # called once ep, host = res self.assertIsInstance(ep, txtorcon.endpoints.TorClientEndpoint) self.assertEqual(host, "foo.onion") self.assertEqual(h._socks_desc, "tcp:127.0.0.1:1234") res = yield h.hint_to_endpoint("tor:foo.onion:29213", reactor, discard_status) self.assertEqual(results, [reactor]) # still only called once ep, host = res self.assertIsInstance(ep, txtorcon.endpoints.TorClientEndpoint) self.assertEqual(host, "foo.onion") self.assertEqual(h._socks_desc, "tcp:127.0.0.1:1234") class I2P(unittest.TestCase): @inlineCallbacks def test_default(self): with mock.patch("foolscap.connections.i2p.SAMI2PStreamClientEndpoint") as sep: sep.new = n = mock.Mock() n.return_value = expected_ep = object() h = i2p.default(reactor, misc_kwarg="foo") res = yield h.hint_to_endpoint("i2p:fppym.b32.i2p", reactor, discard_status) self.assertEqual(len(n.mock_calls), 1) args = n.mock_calls[0][1] got_sep, got_host, got_portnum = args self.assertIsInstance(got_sep, endpoints.TCP4ClientEndpoint) self.failUnlessEqual(got_sep._host, "127.0.0.1") # fragile self.failUnlessEqual(got_sep._port, 7656) self.failUnlessEqual(got_host, "fppym.b32.i2p") self.failUnlessEqual(got_portnum, None) kwargs = n.mock_calls[0][2] self.failUnlessEqual(kwargs, {"misc_kwarg": "foo"}) ep, host = res self.assertIdentical(ep, expected_ep) self.assertEqual(host, "fppym.b32.i2p") self.assertEqual(h.describe(), "i2p") @inlineCallbacks def test_default_with_portnum(self): # I2P addresses generally don't use port numbers, but the parser is # supposed to handle them with mock.patch("foolscap.connections.i2p.SAMI2PStreamClientEndpoint") as sep: sep.new = n = mock.Mock() n.return_value = expected_ep = object() h = i2p.default(reactor) res = yield h.hint_to_endpoint("i2p:fppym.b32.i2p:1234", reactor, discard_status) self.assertEqual(len(n.mock_calls), 1) args = n.mock_calls[0][1] got_sep, got_host, got_portnum = args self.assertIsInstance(got_sep, endpoints.TCP4ClientEndpoint) self.failUnlessEqual(got_sep._host, "127.0.0.1") # fragile self.failUnlessEqual(got_sep._port, 7656) self.failUnlessEqual(got_host, "fppym.b32.i2p") self.failUnlessEqual(got_portnum, 1234) ep, host = res self.assertIdentical(ep, expected_ep) self.assertEqual(host, "fppym.b32.i2p") @inlineCallbacks def test_default_with_portnum_kwarg(self): # setting extra kwargs on the handler should provide a default for # the portnum. sequential calls with/without portnums in the hints # should get the right values. h = i2p.default(reactor, port=1234) with mock.patch("foolscap.connections.i2p.SAMI2PStreamClientEndpoint") as sep: sep.new = n = mock.Mock() yield h.hint_to_endpoint("i2p:fppym.b32.i2p", reactor, discard_status) got_portnum = n.mock_calls[0][1][2] self.failUnlessEqual(got_portnum, 1234) with mock.patch("foolscap.connections.i2p.SAMI2PStreamClientEndpoint") as sep: sep.new = n = mock.Mock() yield h.hint_to_endpoint("i2p:fppym.b32.i2p:3456", reactor, discard_status) got_portnum = n.mock_calls[0][1][2] self.failUnlessEqual(got_portnum, 3456) with mock.patch("foolscap.connections.i2p.SAMI2PStreamClientEndpoint") as sep: sep.new = n = mock.Mock() yield h.hint_to_endpoint("i2p:fppym.b32.i2p", reactor, discard_status) got_portnum = n.mock_calls[0][1][2] self.failUnlessEqual(got_portnum, 1234) def test_default_badhint(self): h = i2p.default(reactor) d = defer.maybeDeferred(h.hint_to_endpoint, "i2p:not@a@hint", reactor, discard_status) f = self.failureResultOf(d, InvalidHintError) self.assertEqual(str(f.value), "unrecognized I2P hint") @inlineCallbacks def test_sam_endpoint(self): with mock.patch("foolscap.connections.i2p.SAMI2PStreamClientEndpoint") as sep: sep.new = n = mock.Mock() n.return_value = expected_ep = object() my_ep = FakeHostnameEndpoint(reactor, "localhost", 1234) h = i2p.sam_endpoint(my_ep, misc_kwarg="foo") res = yield h.hint_to_endpoint("i2p:fppym.b32.i2p", reactor, discard_status) self.assertEqual(len(n.mock_calls), 1) args = n.mock_calls[0][1] got_sep, got_host, got_portnum = args self.assertIdentical(got_sep, my_ep) self.failUnlessEqual(got_host, "fppym.b32.i2p") self.failUnlessEqual(got_portnum, None) kwargs = n.mock_calls[0][2] self.failUnlessEqual(kwargs, {"misc_kwarg": "foo"}) ep, host = res self.assertIdentical(ep, expected_ep) self.assertEqual(host, "fppym.b32.i2p") foolscap-0.13.1/src/foolscap/test/test_copyable.py0000644000076500000240000002650113204511065022607 0ustar warnerstaff00000000000000 from twisted.trial import unittest from twisted.python import components, failure, reflect from foolscap.test.common import TargetMixin, HelperTarget from foolscap import copyable, tokens from foolscap.api import Copyable, RemoteCopy from foolscap.tokens import Violation from foolscap.schema import StringConstraint # MyCopyable1 is the basic Copyable/RemoteCopy pair, using auto-registration. class MyCopyable1(Copyable): typeToCopy = "foolscap.test_copyable.MyCopyable1" pass class MyRemoteCopy1(RemoteCopy): copytype = MyCopyable1.typeToCopy pass #registerRemoteCopy(MyCopyable1.typeToCopy, MyRemoteCopy1) # MyCopyable2 overrides the various Copyable/RemoteCopy methods. It # also sets 'copytype' to auto-register with a matching name class MyCopyable2(Copyable): def getTypeToCopy(self): return "MyCopyable2name" def getStateToCopy(self): return {"a": 1, "b": self.b} class MyRemoteCopy2(RemoteCopy): copytype = "MyCopyable2name" def setCopyableState(self, state): self.c = 1 self.d = state["b"] # MyCopyable3 uses a custom Slicer and a custom Unslicer class MyCopyable3: def getAlternateCopyableState(self): return {"e": 2} class MyCopyable3Slicer(copyable.CopyableSlicer): def slice(self, streamable, banana): yield 'copyable' yield "MyCopyable3name" state = self.obj.getAlternateCopyableState() for k,v in state.iteritems(): yield k yield v class MyRemoteCopy3: pass class MyRemoteCopy3Unslicer(copyable.RemoteCopyUnslicer): def __init__(self): self.schema = None def factory(self, state): obj = MyRemoteCopy3() obj.__dict__ = state return obj def receiveClose(self): obj,d = copyable.RemoteCopyUnslicer.receiveClose(self) obj.f = "yes" return obj, d # register MyCopyable3Slicer as an ISlicer adapter for MyCopyable3, so we # can verify that it overrides the inherited CopyableSlicer behavior. We # also register an Unslicer to create the results. components.registerAdapter(MyCopyable3Slicer, MyCopyable3, tokens.ISlicer) copyable.registerRemoteCopyUnslicerFactory("MyCopyable3name", MyRemoteCopy3Unslicer) # MyCopyable4 uses auto-registration, and adds a stateSchema class MyCopyable4(Copyable): typeToCopy = "foolscap.test_copyable.MyCopyable4" pass class MyRemoteCopy4(RemoteCopy): copytype = MyCopyable4.typeToCopy stateSchema = copyable.AttributeDictConstraint( ('foo', int), ('bar', StringConstraint(1000))) pass # MyCopyable5 disables auto-registration class MyRemoteCopy5(RemoteCopy): copytype = None # disable auto-registration class Copyable(TargetMixin, unittest.TestCase): def setUp(self): TargetMixin.setUp(self) self.setupBrokers() if 0: print self.callingBroker.doLog = "TX" self.targetBroker.doLog = " rx" def send(self, arg): rr, target = self.setupTarget(HelperTarget()) d = rr.callRemote("set", obj=arg) d.addCallback(self.failUnless) # some of these tests require that we return a Failure object, so we # have to wrap this in a tuple to survive the Deferred. d.addCallback(lambda res: (target.obj,)) return d def testCopy0(self): d = self.send(1) d.addCallback(self.failUnlessEqual, (1,)) return d def testFailure1(self): self.callingBroker.unsafeTracebacks = True try: raise RuntimeError("message here") except: f0 = failure.Failure() d = self.send(f0) d.addCallback(self._testFailure1_1) return d def _testFailure1_1(self, (f,)): #print "CopiedFailure is:", f #print f.__dict__ self.failUnlessEqual(reflect.qual(f.type), "exceptions.RuntimeError") self.failUnless(f.check, RuntimeError) self.failUnlessEqual(f.value, "message here") self.failUnlessEqual(f.frames, []) self.failUnlessEqual(f.tb, None) self.failUnlessEqual(f.stack, []) # there should be a traceback self.failUnless(f.traceback.find("raise RuntimeError") != -1, "no 'raise RuntimeError' in '%s'" % (f.traceback,)) # older Twisted (before 17.9.0) used a Failure class that could be # pickled, so our derived CopiedFailure class could be round-tripped # through pickle correclty. Twisted-17.9.0 changed that, so we no # longer try that. ## p = pickle.dumps(f) ## f2 = pickle.loads(p) ## self.failUnlessEqual(reflect.qual(f2.type), "exceptions.RuntimeError") ## self.failUnless(f2.check, RuntimeError) ## self.failUnlessEqual(f2.value, "message here") ## self.failUnlessEqual(f2.frames, []) ## self.failUnlessEqual(f2.tb, None) ## self.failUnlessEqual(f2.stack, []) ## self.failUnless(f2.traceback.find("raise RuntimeError") != -1, ## "no 'raise RuntimeError' in '%s'" % (f2.traceback,)) def testFailure2(self): self.callingBroker.unsafeTracebacks = False try: raise RuntimeError("message here") except: f0 = failure.Failure() d = self.send(f0) d.addCallback(self._testFailure2_1) return d def _testFailure2_1(self, (f,)): #print "CopiedFailure is:", f #print f.__dict__ self.failUnlessEqual(reflect.qual(f.type), "exceptions.RuntimeError") self.failUnless(f.check, RuntimeError) self.failUnlessEqual(f.value, "message here") self.failUnlessEqual(f.frames, []) self.failUnlessEqual(f.tb, None) self.failUnlessEqual(f.stack, []) # there should not be a traceback self.failUnlessEqual(f.traceback, "Traceback unavailable\n") ## # we should be able to pickle CopiedFailures, and when we restore ## # them, they should look like the original ## p = pickle.dumps(f) ## f2 = pickle.loads(p) ## self.failUnlessEqual(reflect.qual(f2.type), "exceptions.RuntimeError") ## self.failUnless(f2.check, RuntimeError) ## self.failUnlessEqual(f2.value, "message here") ## self.failUnlessEqual(f2.frames, []) ## self.failUnlessEqual(f2.tb, None) ## self.failUnlessEqual(f2.stack, []) ## self.failUnlessEqual(f2.traceback, "Traceback unavailable\n") def testCopy1(self): obj = MyCopyable1() # just copies the dict obj.a = 12 obj.b = "foo" d = self.send(obj) d.addCallback(self._testCopy1_1) return d def _testCopy1_1(self, (res,)): self.failUnless(isinstance(res, MyRemoteCopy1)) self.failUnlessEqual(res.a, 12) self.failUnlessEqual(res.b, "foo") def testCopy2(self): obj = MyCopyable2() # has a custom getStateToCopy obj.a = 12 # ignored obj.b = "foo" d = self.send(obj) d.addCallback(self._testCopy2_1) return d def _testCopy2_1(self, (res,)): self.failUnless(isinstance(res, MyRemoteCopy2)) self.failUnlessEqual(res.c, 1) self.failUnlessEqual(res.d, "foo") self.failIf(hasattr(res, "a")) def testCopy3(self): obj = MyCopyable3() # has a custom Slicer obj.a = 12 # ignored obj.b = "foo" # ignored d = self.send(obj) d.addCallback(self._testCopy3_1) return d def _testCopy3_1(self, (res,)): self.failUnless(isinstance(res, MyRemoteCopy3)) self.failUnlessEqual(res.e, 2) self.failUnlessEqual(res.f, "yes") self.failIf(hasattr(res, "a")) def testCopy4(self): obj = MyCopyable4() obj.foo = 12 obj.bar = "bar" d = self.send(obj) d.addCallback(self._testCopy4_1, obj) return d def _testCopy4_1(self, (res,), obj): self.failUnless(isinstance(res, MyRemoteCopy4)) self.failUnlessEqual(res.foo, 12) self.failUnlessEqual(res.bar, "bar") obj.bad = "unwanted attribute" d = self.send(obj) d.addCallbacks(lambda res: self.fail("this was supposed to fail"), self._testCopy4_2, errbackArgs=(obj,)) return d def _testCopy4_2(self, why, obj): why.trap(Violation) self.failUnlessSubstring("unknown attribute 'bad'", str(why)) del obj.bad obj.foo = "not a number" d = self.send(obj) d.addCallbacks(lambda res: self.fail("this was supposed to fail"), self._testCopy4_3, errbackArgs=(obj,)) return d def _testCopy4_3(self, why, obj): why.trap(Violation) self.failUnlessSubstring("STRING token rejected by IntegerConstraint", str(why)) obj.foo = 12 obj.bar = "very long " * 1000 # MyRemoteCopy4 says .bar is a String(1000), so reject long strings d = self.send(obj) d.addCallbacks d.addCallbacks(lambda res: self.fail("this was supposed to fail"), self._testCopy4_4) return d def _testCopy4_4(self, why): why.trap(Violation) self.failUnlessSubstring("token too large", str(why)) class Registration(unittest.TestCase): def testRegistration(self): rc_classes = copyable.debug_RemoteCopyClasses copyable_classes = rc_classes.values() self.failUnless(MyRemoteCopy1 in copyable_classes) self.failUnless(MyRemoteCopy2 in copyable_classes) self.failUnlessIdentical(rc_classes["MyCopyable2name"], MyRemoteCopy2) self.failIf(MyRemoteCopy5 in copyable_classes) ############## # verify that ICopyable adapters are actually usable class TheThirdPartyClassThatIWantToCopy: def __init__(self, a, b): self.a = a self.b = b def copy_ThirdPartyClass(orig): return "TheThirdPartyClassThatIWantToCopy_name", orig.__dict__ copyable.registerCopier(TheThirdPartyClassThatIWantToCopy, copy_ThirdPartyClass) def make_ThirdPartyClass(state): # unpack the state into constructor arguments a = state['a']; b = state['b'] # now create the object with the constructor return TheThirdPartyClassThatIWantToCopy(a, b) copyable.registerRemoteCopyFactory("TheThirdPartyClassThatIWantToCopy_name", make_ThirdPartyClass) class Adaptation(TargetMixin, unittest.TestCase): def setUp(self): TargetMixin.setUp(self) self.setupBrokers() if 0: print self.callingBroker.doLog = "TX" self.targetBroker.doLog = " rx" def send(self, arg): rr, target = self.setupTarget(HelperTarget()) d = rr.callRemote("set", obj=arg) d.addCallback(self.failUnless) # some of these tests require that we return a Failure object, so we # have to wrap this in a tuple to survive the Deferred. d.addCallback(lambda res: (target.obj,)) return d def testAdaptation(self): obj = TheThirdPartyClassThatIWantToCopy(45, 91) d = self.send(obj) d.addCallback(self._testAdaptation_1) return d def _testAdaptation_1(self, (res,)): self.failUnless(isinstance(res, TheThirdPartyClassThatIWantToCopy)) self.failUnlessEqual(res.a, 45) self.failUnlessEqual(res.b, 91) foolscap-0.13.1/src/foolscap/test/test_crypto.py0000644000076500000240000001014012766553111022333 0ustar warnerstaff00000000000000 import re from twisted.trial import unittest from zope.interface import implements from twisted.internet import defer from foolscap import pb from foolscap.api import RemoteInterface, Referenceable, Tub, flushEventualQueue from foolscap.remoteinterface import RemoteMethodSchema from foolscap.util import allocate_tcp_port class RIMyCryptoTarget(RemoteInterface): # method constraints can be declared directly: add1 = RemoteMethodSchema(_response=int, a=int, b=int) # or through their function definitions: def add(a=int, b=int): return int #add = schema.callable(add) # the metaclass makes this unnecessary # but it could be used for adding options or something def join(a=str, b=str, c=int): return str def getName(): return str class Target(Referenceable): implements(RIMyCryptoTarget) def __init__(self, name=None): self.calls = [] self.name = name def getMethodSchema(self, methodname): return None def remote_add(self, a, b): self.calls.append((a,b)) return a+b remote_add1 = remote_add def remote_getName(self): return self.name def remote_disputed(self, a): return 24 def remote_fail(self): raise ValueError("you asked me to fail") class UsefulMixin: num_services = 2 def setUp(self): self.services = [] for i in range(self.num_services): s = Tub() s.startService() self.services.append(s) def tearDown(self): d = defer.DeferredList([s.stopService() for s in self.services]) d.addCallback(self._tearDown_1) return d def _tearDown_1(self, res): return flushEventualQueue() class TestPersist(UsefulMixin, unittest.TestCase): num_services = 2 def testPersist(self): t1 = Target() s1,s2 = self.services port = allocate_tcp_port() s1.listenOn("tcp:%d:interface=127.0.0.1" % port) s1.setLocation("127.0.0.1:%d" % port) public_url = s1.registerReference(t1, "name") self.failUnless(public_url.startswith("pb:")) d = defer.maybeDeferred(s1.stopService) d.addCallback(self._testPersist_1, s1, s2, t1, public_url, port) return d testPersist.timeout = 5 def _testPersist_1(self, res, s1, s2, t1, public_url, port): self.services.remove(s1) s3 = Tub(certData=s1.getCertData()) s3.startService() self.services.append(s3) t2 = Target() newport = allocate_tcp_port() s3.listenOn("tcp:%d:interface=127.0.0.1" % newport) s3.setLocation("127.0.0.1:%d" % newport) s3.registerReference(t2, "name") # now patch the URL to replace the port number newurl = re.sub(":%d/" % port, ":%d/" % newport, public_url) d = s2.getReference(newurl) d.addCallback(lambda rr: rr.callRemote("add", a=1, b=2)) d.addCallback(self.failUnlessEqual, 3) d.addCallback(self._testPersist_2, t1, t2) return d def _testPersist_2(self, res, t1, t2): self.failUnlessEqual(t1.calls, []) self.failUnlessEqual(t2.calls, [(1,2)]) class TestListeners(UsefulMixin, unittest.TestCase): num_services = 3 def testListenOn(self): s1 = self.services[0] l = s1.listenOn("tcp:%d:interface=127.0.0.1" % allocate_tcp_port()) self.failUnless(isinstance(l, pb.Listener)) self.failUnlessEqual(len(s1.getListeners()), 1) s1.stopListeningOn(l) self.failUnlessEqual(len(s1.getListeners()), 0) def testGetPort1(self): s1,s2,s3 = self.services s1.listenOn("tcp:%d:interface=127.0.0.1" % allocate_tcp_port()) listeners = s1.getListeners() self.failUnlessEqual(len(listeners), 1) def testGetPort2(self): s1,s2,s3 = self.services s1.listenOn("tcp:%d:interface=127.0.0.1" % allocate_tcp_port()) listeners = s1.getListeners() self.failUnlessEqual(len(listeners), 1) # listen on a second port too s1.listenOn("tcp:%d:interface=127.0.0.1" % allocate_tcp_port()) l2 = s1.getListeners() self.failUnlessEqual(len(l2), 2) foolscap-0.13.1/src/foolscap/test/test_eventual.py0000644000076500000240000000223012766553111022637 0ustar warnerstaff00000000000000 from twisted.trial import unittest from foolscap.eventual import eventually, fireEventually, flushEventualQueue class TestEventual(unittest.TestCase): def tearDown(self): return flushEventualQueue() def testSend(self): results = [] eventually(results.append, 1) self.failIf(results) def _check(): self.failUnlessEqual(results, [1]) eventually(_check) def _check2(): self.failUnlessEqual(results, [1,2]) eventually(results.append, 2) eventually(_check2) def testFlush(self): results = [] eventually(results.append, 1) eventually(results.append, 2) d = flushEventualQueue() def _check(res): self.failUnlessEqual(results, [1,2]) d.addCallback(_check) return d def testFire(self): results = [] fireEventually(1).addCallback(results.append) fireEventually(2).addCallback(results.append) self.failIf(results) def _check(res): self.failUnlessEqual(results, [1,2]) d = flushEventualQueue() d.addCallback(_check) return d foolscap-0.13.1/src/foolscap/test/test_gifts.py0000644000076500000240000006400512766553111022140 0ustar warnerstaff00000000000000 from zope.interface import implements from twisted.trial import unittest from twisted.internet import defer, protocol, reactor from twisted.internet.error import ConnectionRefusedError from foolscap.api import RemoteInterface, Referenceable, flushEventualQueue, \ BananaError, Tub from foolscap.util import allocate_tcp_port from foolscap.referenceable import RemoteReference from foolscap.furl import encode_furl, decode_furl from foolscap.test.common import (HelperTarget, RIHelper, ShouldFailMixin, MakeTubsMixin) from foolscap.tokens import NegotiationError, Violation class RIConstrainedHelper(RemoteInterface): def set(obj=RIHelper): return None class ConstrainedHelper(Referenceable): implements(RIConstrainedHelper) def __init__(self, name="unnamed"): self.name = name def remote_set(self, obj): self.obj = obj class Base(ShouldFailMixin, MakeTubsMixin): debug = False def setUp(self): self.tubA, self.tubB, self.tubC, self.tubD = self.makeTubs(4) def tearDown(self): d = defer.DeferredList([s.stopService() for s in self.services]) d.addCallback(flushEventualQueue) return d def createCharacters(self): self.alice = HelperTarget("alice") self.bob = HelperTarget("bob") self.bob_url = self.tubB.registerReference(self.bob, "bob") self.carol = HelperTarget("carol") self.carol_url = self.tubC.registerReference(self.carol, "carol") # cindy is Carol's little sister. She doesn't have a phone, but # Carol might talk about her anyway. self.cindy = HelperTarget("cindy") # more sisters. Alice knows them, and she introduces Bob to them. self.charlene = HelperTarget("charlene") self.christine = HelperTarget("christine") self.clarisse = HelperTarget("clarisse") self.colette = HelperTarget("colette") self.courtney = HelperTarget("courtney") self.dave = HelperTarget("dave") self.dave_url = self.tubD.registerReference(self.dave, "dave") def createInitialReferences(self): # we must start by giving Alice a reference to both Bob and Carol. if self.debug: print "Alice gets Bob" d = self.tubA.getReference(self.bob_url) def _aliceGotBob(abob): if self.debug: print "Alice got bob" self.abob = abob # Alice's reference to Bob if self.debug: print "Alice gets carol" d = self.tubA.getReference(self.carol_url) return d d.addCallback(_aliceGotBob) def _aliceGotCarol(acarol): if self.debug: print "Alice got carol" self.acarol = acarol # Alice's reference to Carol d = self.tubB.getReference(self.dave_url) return d d.addCallback(_aliceGotCarol) def _bobGotDave(bdave): self.bdave = bdave d.addCallback(_bobGotDave) return d def createMoreReferences(self): # give Alice references to Carol's sisters dl = [] url = self.tubC.registerReference(self.charlene, "charlene") d = self.tubA.getReference(url) def _got_charlene(rref): self.acharlene = rref d.addCallback(_got_charlene) dl.append(d) url = self.tubC.registerReference(self.christine, "christine") d = self.tubA.getReference(url) def _got_christine(rref): self.achristine = rref d.addCallback(_got_christine) dl.append(d) url = self.tubC.registerReference(self.clarisse, "clarisse") d = self.tubA.getReference(url) def _got_clarisse(rref): self.aclarisse = rref d.addCallback(_got_clarisse) dl.append(d) url = self.tubC.registerReference(self.colette, "colette") d = self.tubA.getReference(url) def _got_colette(rref): self.acolette = rref d.addCallback(_got_colette) dl.append(d) url = self.tubC.registerReference(self.courtney, "courtney") d = self.tubA.getReference(url) def _got_courtney(rref): self.acourtney = rref d.addCallback(_got_courtney) dl.append(d) return defer.DeferredList(dl) class Gifts(Base, unittest.TestCase): # Here we test the three-party introduction process as depicted in the # classic Granovetter diagram. Alice has a reference to Bob and another # one to Carol. Alice wants to give her Carol-reference to Bob, by # including it as the argument to a method she invokes on her # Bob-reference. def testGift(self): #defer.setDebugging(True) self.createCharacters() d = self.createInitialReferences() def _introduce(res): d2 = self.bob.waitfor() if self.debug: print "Alice introduces Carol to Bob" # send the gift. This might not get acked by the time the test is # done and everything is torn down, so we use callRemoteOnly self.abob.callRemoteOnly("set", obj=(self.alice, self.acarol)) return d2 # this fires with the gift that bob got d.addCallback(_introduce) def _bobGotCarol((balice,bcarol)): if self.debug: print "Bob got Carol" self.bcarol = bcarol if self.debug: print "Bob says something to Carol" d2 = self.carol.waitfor() # handle ConnectionDone as described before self.bcarol.callRemoteOnly("set", obj=12) return d2 d.addCallback(_bobGotCarol) def _carolCalled(res): if self.debug: print "Carol heard from Bob" self.failUnlessEqual(res, 12) d.addCallback(_carolCalled) return d testGift.timeout = 10 def testImplicitGift(self): # in this test, Carol was registered in her Tub (using # registerReference), but Cindy was not. Alice is given a reference # to Carol, then uses that to get a reference to Cindy. Then Alice # sends a message to Bob and includes a reference to Cindy. The test # here is that we can make gifts out of references that were not # passed to registerReference explicitly. #defer.setDebugging(True) self.createCharacters() # the message from Alice to Bob will include a reference to Cindy d = self.createInitialReferences() def _tell_alice_about_cindy(res): self.carol.obj = self.cindy cindy_d = self.acarol.callRemote("get") return cindy_d d.addCallback(_tell_alice_about_cindy) def _introduce(a_cindy): # alice now has references to carol (self.acarol) and cindy # (a_cindy). She sends both of them (plus a reference to herself) # to bob. d2 = self.bob.waitfor() if self.debug: print "Alice introduces Carol to Bob" # send the gift. This might not get acked by the time the test is # done and everything is torn down, so explicitly silence any # ConnectionDone error that might result. When we get # callRemoteOnly(), use that instead. self.abob.callRemoteOnly("set", obj=(self.alice, self.acarol, a_cindy)) return d2 # this fires with the gift that bob got d.addCallback(_introduce) def _bobGotCarol((b_alice,b_carol,b_cindy)): if self.debug: print "Bob got Carol" self.failUnless(b_alice) self.failUnless(b_carol) self.failUnless(b_cindy) self.bcarol = b_carol if self.debug: print "Bob says something to Carol" d2 = self.carol.waitfor() if self.debug: print "Bob says something to Cindy" d3 = self.cindy.waitfor() # handle ConnectionDone as described before b_carol.callRemoteOnly("set", obj=4) b_cindy.callRemoteOnly("set", obj=5) return defer.DeferredList([d2,d3]) d.addCallback(_bobGotCarol) def _carolAndCindyCalled(res): if self.debug: print "Carol heard from Bob" ((carol_s, carol_result), (cindy_s, cindy_result)) = res self.failUnless(carol_s) self.failUnless(cindy_s) self.failUnlessEqual(carol_result, 4) self.failUnlessEqual(cindy_result, 5) d.addCallback(_carolAndCindyCalled) return d # test gifts in return values too def testReturn(self): self.createCharacters() d = self.createInitialReferences() def _introduce(res): self.bob.obj = self.bdave return self.abob.callRemote("get") d.addCallback(_introduce) def _check(adave): # this ought to be a RemoteReference to dave, usable by alice self.failUnless(isinstance(adave, RemoteReference)) return adave.callRemote("set", 12) d.addCallback(_check) def _check2(res): self.failUnlessEqual(self.dave.obj, 12) d.addCallback(_check2) return d def testReturnInContainer(self): self.createCharacters() d = self.createInitialReferences() def _introduce(res): self.bob.obj = {"foo": [(set([self.bdave]),)]} return self.abob.callRemote("get") d.addCallback(_introduce) def _check(obj): adave = list(obj["foo"][0][0])[0] # this ought to be a RemoteReference to dave, usable by alice self.failUnless(isinstance(adave, RemoteReference)) return adave.callRemote("set", 12) d.addCallback(_check) def _check2(res): self.failUnlessEqual(self.dave.obj, 12) d.addCallback(_check2) return d def testOrdering(self): self.createCharacters() self.bob.calls = [] d = self.createInitialReferences() def _introduce(res): # we send three messages to Bob. The second one contains the # reference to Carol. dl = [] dl.append(self.abob.callRemote("append", obj=1)) dl.append(self.abob.callRemote("append", obj=self.acarol)) dl.append(self.abob.callRemote("append", obj=3)) return defer.DeferredList(dl) d.addCallback(_introduce) def _checkBob(res): # this runs after all three messages have been acked by Bob self.failUnlessEqual(len(self.bob.calls), 3) self.failUnlessEqual(self.bob.calls[0], 1) self.failUnless(isinstance(self.bob.calls[1], RemoteReference)) self.failUnlessEqual(self.bob.calls[2], 3) d.addCallback(_checkBob) return d def testContainers(self): self.createCharacters() self.bob.calls = [] d = self.createInitialReferences() d.addCallback(lambda res: self.createMoreReferences()) def _introduce(res): # we send several messages to Bob, each of which has a container # with a gift inside it. This exercises the ready_deferred # handling inside containers. dl = [] cr = self.abob.callRemote dl.append(cr("append", set([self.acharlene]))) dl.append(cr("append", frozenset([self.achristine]))) dl.append(cr("append", [self.aclarisse])) dl.append(cr("append", obj=(self.acolette,))) dl.append(cr("append", {'a': self.acourtney})) # TODO: pass a gift as an attribute of a Copyable return defer.DeferredList(dl) d.addCallback(_introduce) def _checkBob(res): # this runs after all three messages have been acked by Bob self.failUnlessEqual(len(self.bob.calls), 5) bcharlene = self.bob.calls.pop(0) self.failUnless(isinstance(bcharlene, set)) self.failUnlessEqual(len(bcharlene), 1) self.failUnless(isinstance(list(bcharlene)[0], RemoteReference)) bchristine = self.bob.calls.pop(0) self.failUnless(isinstance(bchristine, frozenset)) self.failUnlessEqual(len(bchristine), 1) self.failUnless(isinstance(list(bchristine)[0], RemoteReference)) bclarisse = self.bob.calls.pop(0) self.failUnless(isinstance(bclarisse, list)) self.failUnlessEqual(len(bclarisse), 1) self.failUnless(isinstance(bclarisse[0], RemoteReference)) bcolette = self.bob.calls.pop(0) self.failUnless(isinstance(bcolette, tuple)) self.failUnlessEqual(len(bcolette), 1) self.failUnless(isinstance(bcolette[0], RemoteReference)) bcourtney = self.bob.calls.pop(0) self.failUnless(isinstance(bcourtney, dict)) self.failUnlessEqual(len(bcourtney), 1) self.failUnless(isinstance(bcourtney['a'], RemoteReference)) d.addCallback(_checkBob) return d def create_constrained_characters(self): self.alice = HelperTarget("alice") self.bob = ConstrainedHelper("bob") self.bob_url = self.tubB.registerReference(self.bob, "bob") self.carol = HelperTarget("carol") self.carol_url = self.tubC.registerReference(self.carol, "carol") self.dave = HelperTarget("dave") self.dave_url = self.tubD.registerReference(self.dave, "dave") def test_constraint(self): self.create_constrained_characters() self.bob.calls = [] d = self.createInitialReferences() def _introduce(res): return self.abob.callRemote("set", self.acarol) d.addCallback(_introduce) def _checkBob(res): self.failUnless(isinstance(self.bob.obj, RemoteReference)) d.addCallback(_checkBob) return d # this was used to alice's reference to carol (self.acarol) appeared in # alice's gift table at the right time, to make sure that the # RemoteReference is kept alive while the gift is in transit. The whole # introduction pattern is going to change soon, so it has been disabled # until I figure out what the new scheme ought to be asserting. def OFF_bobGotCarol(self, (balice,bcarol)): if self.debug: print "Bob got Carol" # Bob has received the gift self.bcarol = bcarol # wait for alice to receive bob's 'decgift' sequence, which was sent # by now (it is sent after bob receives the gift but before the # gift-bearing message is delivered). To make sure alice has received # it, send a message back along the same path. def _check_alice(res): if self.debug: print "Alice should have the decgift" # alice's gift table should be empty brokerAB = self.abob.tracker.broker self.failUnlessEqual(brokerAB.myGifts, {}) self.failUnlessEqual(brokerAB.myGiftsByGiftID, {}) d1 = self.alice.waitfor() d1.addCallback(_check_alice) # the ack from this message doesn't always make it back by the time # we end the test and hang up the connection. That connectionLost # causes the deferred that this returns to errback, triggering an # error, so we must be sure to discard any error from it. TODO: turn # this into balice.callRemoteOnly("set", 39), which will have the # same semantics from our point of view (but in addition it will tell # the recipient to not bother sending a response). balice.callRemote("set", 39).addErrback(lambda ignored: None) if self.debug: print "Bob says something to Carol" d2 = self.carol.waitfor() d = self.bcarol.callRemote("set", obj=12) d.addCallback(lambda res: d2) d.addCallback(self._carolCalled) d.addCallback(lambda res: d1) return d class Bad(Base, unittest.TestCase): # if the recipient cannot claim their gift, the caller should see an # errback. def setUp(self): Base.setUp(self) def test_swissnum(self): self.createCharacters() d = self.createInitialReferences() d.addCallback(lambda res: self.tubA.getReference(self.dave_url)) def _introduce(adave): # now break the gift to insure that Bob is unable to claim it. # The first way to do this is to simple mangle the swissnum, # which will result in a failure in remote_getReferenceByName. # NOTE: this will have to change when we modify the way gifts are # referenced, since tracker.url is scheduled to go away. adave.tracker.url = adave.tracker.url + ".MANGLED" return self.shouldFail(KeyError, "Bad.test_swissnum", "unable to find reference for name starting with 'da'", self.acarol.callRemote, "set", adave) d.addCallback(_introduce) # make sure we can still talk to Carol, though d.addCallback(lambda res: self.acarol.callRemote("set", 14)) d.addCallback(lambda res: self.failUnlessEqual(self.carol.obj, 14)) return d def test_tubid(self): self.createCharacters() d = self.createInitialReferences() d.addCallback(lambda res: self.tubA.getReference(self.dave_url)) def _introduce(adave): # The second way is to mangle the tubid, which will result in a # failure during negotiation. We mangle it by reversing the # characters: this makes it syntactically valid but highly # unlikely to remain the same. NOTE: this will have to change # when we modify the way gifts are referenced, since tracker.url # is scheduled to go away. (tubid, location_hints, name) = decode_furl(adave.tracker.url) tubid = "".join(reversed(tubid)) adave.tracker.url = encode_furl(tubid, location_hints, name) return self.shouldFail(BananaError, "Bad.test_tubid", "unknown TubID", self.acarol.callRemote, "set", adave) d.addCallback(_introduce) return d def test_location(self): self.createCharacters() d = self.createInitialReferences() d.addCallback(lambda res: self.tubA.getReference(self.dave_url)) def _introduce(adave): # The third way is to mangle the location hints, which will # result in a failure during negotiation as it attempts to # establish a TCP connection. (tubid, location_hints, name) = decode_furl(adave.tracker.url) # highly unlikely that there's anything listening on this port location_hints = ["tcp:127.0.0.1:2"] adave.tracker.url = encode_furl(tubid, location_hints, name) return self.shouldFail(ConnectionRefusedError, "Bad.test_location", "Connection was refused by other side", self.acarol.callRemote, "set", adave) d.addCallback(_introduce) return d def test_hang(self): f = protocol.Factory() f.protocol = protocol.Protocol # ignores all input p = reactor.listenTCP(0, f, interface="127.0.0.1") self.createCharacters() d = self.createInitialReferences() d.addCallback(lambda res: self.tubA.getReference(self.dave_url)) def _introduce(adave): # The next form of mangling is to connect to a port which never # responds, which could happen if a firewall were silently # dropping the TCP packets. We can't accurately simulate this # case, but we can connect to a port which accepts the connection # and then stays silent. This should trigger the overall # connection timeout. (tubid, location_hints, name) = decode_furl(adave.tracker.url) location_hints = ["tcp:127.0.0.1:%d" % p.getHost().port] adave.tracker.url = encode_furl(tubid, location_hints, name) self.tubD._test_options['connect_timeout'] = 2 return self.shouldFail(NegotiationError, "Bad.test_hang", "no connection established within client timeout", self.acarol.callRemote, "set", adave) d.addCallback(_introduce) def _stop_listening(res): d1 = p.stopListening() def _done_listening(x): return res d1.addCallback(_done_listening) return d1 d.addBoth(_stop_listening) return d def testReturn_swissnum(self): self.createCharacters() d = self.createInitialReferences() def _introduce(res): # now break the gift to insure that Alice is unable to claim it. # The first way to do this is to simple mangle the swissnum, # which will result in a failure in remote_getReferenceByName. # NOTE: this will have to change when we modify the way gifts are # referenced, since tracker.url is scheduled to go away. self.bdave.tracker.url = self.bdave.tracker.url + ".MANGLED" self.bob.obj = self.bdave return self.shouldFail(KeyError, "Bad.testReturn_swissnum", "unable to find reference for name starting with 'da'", self.abob.callRemote, "get") d.addCallback(_introduce) # make sure we can still talk to Bob, though d.addCallback(lambda res: self.abob.callRemote("set", 14)) d.addCallback(lambda res: self.failUnlessEqual(self.bob.obj, 14)) return d class LongFURL(Base, unittest.TestCase): # make sure the old 200-byte limit on gift FURLs is gone def setUp(self): def mangleLocation(portnum): loc = "127.0.0.1:%d" % portnum loc = ",".join([loc]*15) # 239 bytes of location, 281 of FURL return loc (self.tubA, self.tubB, self.tubC, self.tubD) = self.makeTubs(4, mangleLocation) def testGift(self): self.createCharacters() d = self.createInitialReferences() def _introduce(res): d2 = self.bob.waitfor() if self.debug: print "Alice introduces Carol to Bob" # send the gift. This might not get acked by the time the test is # done and everything is torn down, so we use callRemoteOnly self.abob.callRemoteOnly("set", obj=(self.alice, self.acarol)) return d2 # this fires with the gift that bob got d.addCallback(_introduce) def _bobGotCarol((balice,bcarol)): if self.debug: print "Bob got Carol" self.bcarol = bcarol if self.debug: print "Bob says something to Carol" d2 = self.carol.waitfor() # handle ConnectionDone as described before self.bcarol.callRemoteOnly("set", obj=12) return d2 d.addCallback(_bobGotCarol) def _carolCalled(res): if self.debug: print "Carol heard from Bob" self.failUnlessEqual(res, 12) d.addCallback(_carolCalled) return d class Enabled(Base, unittest.TestCase): def setUp(self): self.services = [Tub() for i in range(4)] self.tubA, self.tubB, self.tubC, self.tubD = self.services for s in self.services: s.startService() p = allocate_tcp_port() s.listenOn("tcp:%d:interface=127.0.0.1" % p) s.setLocation("127.0.0.1:%d" % p) self.tubIDs = [self.tubA.getShortTubID(), self.tubB.getShortTubID(), self.tubC.getShortTubID(), self.tubD.getShortTubID()] def get_connections(self, tub): self.failIf(tub.waitingForBrokers) return set([tr.getShortTubID() for tr in tub.brokers.keys()]) def testGiftsEnabled(self): # enabled is the default, so this shouldn't change anything self.tubB.setOption("accept-gifts", True) self.createCharacters() d = self.createInitialReferences() def _introduce(res): d2 = self.bob.waitfor() d3 = self.abob.callRemote("set", obj=(self.alice, self.acarol)) d3.addCallback(lambda _: d2) return d3 # this fires with the gift that bob got d.addCallback(_introduce) def _bobGotCarol((balice,bcarol)): A,B,C,D = self.tubIDs b_connections = self.get_connections(self.tubB) self.assertIn(C, b_connections) self.failUnlessEqual(b_connections, set([A, C, D])) d.addCallback(_bobGotCarol) return d def testGiftsDisabled(self): self.tubB.setOption("accept-gifts", False) self.createCharacters() self.bob.obj = None d = self.createInitialReferences() d.addCallback(lambda _: self.shouldFail(Violation, "testGiftsDisabled", "gifts are prohibited in this Tub", self.abob.callRemote, "set", obj=(self.alice, self.acarol))) d.addCallback(lambda _: self.failIf(self.bob.obj)) def _check_tub(_): A,B,C,D = self.tubIDs b_connections = self.get_connections(self.tubB) self.failIfIn(C, b_connections) self.failUnlessEqual(b_connections, set([A, D])) d.addCallback(_check_tub) return d def testGiftsDisabledReturn(self): self.tubA.setOption("accept-gifts", False) self.createCharacters() d = self.createInitialReferences() def _created(_): self.bob.obj = self.bdave return self.shouldFail(Violation, "testGiftsDisabledReturn", "gifts are prohibited in this Tub", self.abob.callRemote, "get") d.addCallback(_created) def _check_tub(_): A,B,C,D = self.tubIDs a_connections = self.get_connections(self.tubA) self.failIfIn(D, a_connections) self.failUnlessEqual(a_connections, set([B,C])) return d foolscap-0.13.1/src/foolscap/test/test_info.py0000644000076500000240000001570013204160675021752 0ustar warnerstaff00000000000000from zope.interface import implementer from twisted.trial import unittest from twisted.internet import defer, reactor from twisted.application import service from foolscap import info, reconnector, ipb, util from foolscap.api import Tub from foolscap.connections import tcp from foolscap.test.common import (certData_low, certData_high, Target) class Info(unittest.TestCase): def test_stages(self): ci = info.ConnectionInfo() self.assertEqual(ci.connected, False) self.assertEqual(ci.connectorStatuses, {}) self.assertEqual(ci.connectionHandlers, {}) self.assertEqual(ci.establishedAt, None) self.assertEqual(ci.winningHint, None) self.assertEqual(ci.listenerStatus, (None, None)) self.assertEqual(ci.lostAt, None) ci._describe_connection_handler("hint1", "tcp") ci._set_connection_status("hint1", "working") self.assertEqual(ci.connectorStatuses, {"hint1": "working"}) self.assertEqual(ci.connectionHandlers, {"hint1": "tcp"}) ci._set_connection_status("hint1", "successful") ci._set_winning_hint("hint1") ci._set_established_at(10.0) ci._set_connected(True) self.assertEqual(ci.connected, True) self.assertEqual(ci.connectorStatuses, {"hint1": "successful"}) self.assertEqual(ci.connectionHandlers, {"hint1": "tcp"}) self.assertEqual(ci.establishedAt, 10.0) self.assertEqual(ci.winningHint, "hint1") self.assertEqual(ci.listenerStatus, (None, None)) self.assertEqual(ci.lostAt, None) ci._set_connected(False) ci._set_lost_at(15.0) self.assertEqual(ci.connected, False) self.assertEqual(ci.lostAt, 15.0) @implementer(ipb.IConnectionHintHandler) class Handler: def __init__(self): self.asked = 0 self.accepted = 0 self._d = defer.Deferred() self._err = None def hint_to_endpoint(self, hint, reactor, update_status): self.asked += 1 self._update_status = update_status self._d = defer.Deferred() if self._err: raise self._err self.accepted += 1 update_status("resolving hint") return self._d def discard_status(status): pass class Connect(unittest.TestCase): def setUp(self): self.s = service.MultiService() self.s.startService() def tearDown(self): return self.s.stopService() def makeTub(self, hint_type, listener_test_options={}, extra_hint=None): tubA = Tub(certData=certData_low) tubA.setServiceParent(self.s) tubB = Tub(certData=certData_high) tubB.setServiceParent(self.s) self._tubA, self._tubB = tubA, tubB portnum = util.allocate_tcp_port() self._portnum = portnum port = "tcp:%d:interface=127.0.0.1" % portnum hint = "%s:127.0.0.1:%d" % (hint_type, portnum) if extra_hint: hint = hint + "," + extra_hint tubA.listenOn(port, _test_options=listener_test_options) tubA.setLocation(hint) self._target = Target() furl = tubA.registerReference(self._target) return furl, tubB, hint @defer.inlineCallbacks def testInfo(self): def tubA_sendHello_pause(d2): ci = tubB.getConnectionInfoForFURL(furl) self.assertEqual(ci.connectorStatuses, {hint: "negotiating"}) d2.callback(None) test_options = { "debug_pause_sendHello": tubA_sendHello_pause, } furl, tubB, hint = self.makeTub("tcp", test_options) h = Handler() tubB.removeAllConnectionHintHandlers() tubB.addConnectionHintHandler("tcp", h) d = tubB.getReference(furl) ci = tubB.getConnectionInfoForFURL(furl) self.assertEqual(ci.connectorStatuses, {hint: "resolving hint"}) h._d.callback(tcp.DefaultTCP().hint_to_endpoint(hint, reactor, discard_status)) ci = tubB.getConnectionInfoForFURL(furl) self.assertEqual(ci.connectorStatuses, {hint: "connecting"}) # we use debug_pause_sendHello to catch "negotiating" here, then wait rref = yield d self.failUnlessEqual(h.asked, 1) self.failUnlessEqual(h.accepted, 1) ci = tubB.getConnectionInfoForFURL(furl) self.assertEqual(ci.connectorStatuses, {hint: "successful"}) del rref def testNoHandler(self): furl, tubB, hint = self.makeTub("missing", extra_hint="slow:foo") missing_hint, extra = hint.split(",") tubB.removeAllConnectionHintHandlers() h = Handler() tubB.addConnectionHintHandler("slow", h) d = tubB.getReference(furl) del d # XXX ci = tubB.getConnectionInfoForFURL(furl) cs = ci.connectorStatuses self.assertEqual(cs["slow:foo"], "resolving hint") self.assertEqual(cs[missing_hint], "bad hint: no handler registered") h._update_status("phase2") ci = tubB.getConnectionInfoForFURL(furl) cs = ci.connectorStatuses self.assertEqual(cs["slow:foo"], "phase2") @defer.inlineCallbacks def testListener(self): furl, tubB, hint = self.makeTub("tcp") rref1 = yield tubB.getReference(furl) yield rref1.callRemote("free", Target()) rref2 = self._target.calls[0][0][0] ci = rref2.getConnectionInfo() self.assertEqual(ci.connectorStatuses, {}) (listener, status) = ci.listenerStatus self.assertEqual(status, "successful") self.assertEqual(listener, "Listener on IPv4Address(TCP, '127.0.0.1', %d)" % self._portnum) @defer.inlineCallbacks def testLoopback(self): furl, tubB, hint = self.makeTub("tcp") rref1 = yield self._tubA.getReference(furl) ci = rref1.getConnectionInfo() self.assertEqual(ci.connectorStatuses, {"loopback": "connected"}) self.assertEqual(ci.listenerStatus, (None, None)) class Reconnection(unittest.TestCase): def test_stages(self): ri = reconnector.ReconnectionInfo() self.assertEqual(ri.state, "unstarted") self.assertEqual(ri.connectionInfo, None) self.assertEqual(ri.lastAttempt, None) self.assertEqual(ri.nextAttempt, None) ci = object() ri._set_state("connecting") ri._set_connection_info(ci) ri._set_last_attempt(10.0) self.assertEqual(ri.state, "connecting") self.assertEqual(ri.connectionInfo, ci) self.assertEqual(ri.lastAttempt, 10.0) self.assertEqual(ri.nextAttempt, None) ri._set_state("connected") self.assertEqual(ri.state, "connected") ri._set_state("waiting") ri._set_connection_info(None) ri._set_next_attempt(20.0) self.assertEqual(ri.state, "waiting") self.assertEqual(ri.connectionInfo, None) self.assertEqual(ri.lastAttempt, 10.0) self.assertEqual(ri.nextAttempt, 20.0) foolscap-0.13.1/src/foolscap/test/test_interfaces.py0000644000076500000240000003107012766553111023143 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test.test_interfaces -*- from zope.interface import implementsOnly from twisted.trial import unittest from foolscap import schema, remoteinterface from foolscap.api import RemoteInterface from foolscap.remoteinterface import getRemoteInterface, RemoteMethodSchema from foolscap.remoteinterface import RemoteInterfaceRegistry from foolscap.tokens import Violation from foolscap.referenceable import RemoteReference from foolscap.test.common import TargetMixin from foolscap.test.common import getRemoteInterfaceName, Target, RIMyTarget, \ RIMyTarget2, TargetWithoutInterfaces, IFoo, Foo, TypesTarget, RIDummy, \ DummyTarget class Target2(Target): implementsOnly(IFoo, RIMyTarget2) class TestInterface(TargetMixin, unittest.TestCase): def testTypes(self): self.failUnless(isinstance(RIMyTarget, remoteinterface.RemoteInterfaceClass)) self.failUnless(isinstance(RIMyTarget2, remoteinterface.RemoteInterfaceClass)) def testRegister(self): reg = RemoteInterfaceRegistry self.failUnlessEqual(reg["RIMyTarget"], RIMyTarget) self.failUnlessEqual(reg["RIMyTargetInterface2"], RIMyTarget2) def testDuplicateRegistry(self): try: class RIMyTarget(RemoteInterface): def foo(bar=int): return int except remoteinterface.DuplicateRemoteInterfaceError: pass else: self.fail("duplicate registration not caught") def testInterface1(self): # verify that we extract the right interfaces from a local object. # also check that the registry stuff works. self.setupBrokers() rr, target = self.setupTarget(Target()) iface = getRemoteInterface(target) self.failUnlessEqual(iface, RIMyTarget) iname = getRemoteInterfaceName(target) self.failUnlessEqual(iname, "RIMyTarget") self.failUnlessIdentical(RemoteInterfaceRegistry["RIMyTarget"], RIMyTarget) rr, target = self.setupTarget(Target2()) iname = getRemoteInterfaceName(target) self.failUnlessEqual(iname, "RIMyTargetInterface2") self.failUnlessIdentical(\ RemoteInterfaceRegistry["RIMyTargetInterface2"], RIMyTarget2) def testInterface2(self): # verify that RemoteInterfaces have the right attributes t = Target() iface = getRemoteInterface(t) self.failUnlessEqual(iface, RIMyTarget) # 'add' is defined with 'def' s1 = RIMyTarget['add'] self.failUnless(isinstance(s1, RemoteMethodSchema)) ok, s2 = s1.getKeywordArgConstraint("a") self.failUnless(ok) self.failUnless(isinstance(s2, schema.IntegerConstraint)) self.failUnless(s2.checkObject(12, False) == None) self.failUnlessRaises(schema.Violation, s2.checkObject, "string", False) s3 = s1.getResponseConstraint() self.failUnless(isinstance(s3, schema.IntegerConstraint)) # 'add1' is defined as a class attribute s1 = RIMyTarget['add1'] self.failUnless(isinstance(s1, RemoteMethodSchema)) ok, s2 = s1.getKeywordArgConstraint("a") self.failUnless(ok) self.failUnless(isinstance(s2, schema.IntegerConstraint)) self.failUnless(s2.checkObject(12, False) == None) self.failUnlessRaises(schema.Violation, s2.checkObject, "string", False) s3 = s1.getResponseConstraint() self.failUnless(isinstance(s3, schema.IntegerConstraint)) s1 = RIMyTarget['join'] self.failUnless(isinstance(s1.getKeywordArgConstraint("a")[1], schema.StringConstraint)) self.failUnless(isinstance(s1.getKeywordArgConstraint("c")[1], schema.IntegerConstraint)) s3 = RIMyTarget['join'].getResponseConstraint() self.failUnless(isinstance(s3, schema.StringConstraint)) s1 = RIMyTarget['disputed'] self.failUnless(isinstance(s1.getKeywordArgConstraint("a")[1], schema.IntegerConstraint)) s3 = s1.getResponseConstraint() self.failUnless(isinstance(s3, schema.IntegerConstraint)) def testInterface3(self): t = TargetWithoutInterfaces() iface = getRemoteInterface(t) self.failIf(iface) def testStack(self): # when you violate your outbound schema, the Failure you get should # have a stack trace that includes the actual callRemote invocation. # Sometimes the stack trace doesn't include source code (either we # have .pyc files but not .py files, or because the code is coming # from an .egg). So this test merely asserts that test_interfaces.py # is present in the trace, followed by either a source code line that # mentions callRemote, or the filename/linenumber/functionname line # that mentions callRemote. self.setupBrokers() rr, target = self.setupTarget(Target(), True) d = rr.callRemote('add', "not a number", "oops") def _check_failure(f): s = f.getTraceback().split("\n") for i in range(len(s)): line = s[i] if ("test_interfaces.py" in line and i+2 < len(s) and ("rr.callRemote" in s[i+1] or "in callRemote" in s[i+2])): return # all good print "failure looked like this:" print f self.fail("didn't see invocation of callRemote in stacktrace") d.addCallbacks(lambda res: self.fail("hey, this was supposed to fail"), _check_failure) return d class Types(TargetMixin, unittest.TestCase): def setUp(self): TargetMixin.setUp(self) self.setupBrokers() def deferredShouldFail(self, d, ftype=None, checker=None): if not ftype and not checker: d.addCallbacks(lambda res: self.fail("hey, this was supposed to fail"), lambda f: None) elif ftype and not checker: d.addCallbacks(lambda res: self.fail("hey, this was supposed to fail"), lambda f: f.trap(ftype) or None) else: d.addCallbacks(lambda res: self.fail("hey, this was supposed to fail"), checker) def testCall(self): rr, target = self.setupTarget(Target(), True) d = rr.callRemote('add', 3, 4) # enforces schemas d.addCallback(lambda res: self.failUnlessEqual(res, 7)) return d def testFail(self): # make sure exceptions (and thus CopiedFailures) pass a schema check rr, target = self.setupTarget(Target(), True) d = rr.callRemote('fail') self.deferredShouldFail(d, ftype=ValueError) return d def testNoneGood(self): rr, target = self.setupTarget(TypesTarget(), True) d = rr.callRemote('returns_none', True) d.addCallback(lambda res: self.failUnlessEqual(res, None)) return d def testNoneBad(self): rr, target = self.setupTarget(TypesTarget(), True) d = rr.callRemote('returns_none', False) def _check_failure(f): f.trap(Violation) self.failUnlessIn("(in return value of .returns_none", str(f)) self.failUnlessIn("'not None' is not None", str(f)) self.deferredShouldFail(d, checker=_check_failure) return d def testTakesRemoteInterfaceGood(self): rr, target = self.setupTarget(TypesTarget(), True) d = rr.callRemote('takes_remoteinterface', DummyTarget()) d.addCallback(lambda res: self.failUnlessEqual(res, "good")) return d def testTakesRemoteInterfaceBad(self): rr, target = self.setupTarget(TypesTarget(), True) # takes_remoteinterface is specified to accept an RIDummy d = rr.callRemote('takes_remoteinterface', 12) def _check_failure(f): f.trap(Violation) self.failUnlessIn("RITypes.takes_remoteinterface(a=))", str(f)) self.failUnlessIn("'12' is not a Referenceable", str(f)) self.deferredShouldFail(d, checker=_check_failure) return d def testTakesRemoteInterfaceBad2(self): rr, target = self.setupTarget(TypesTarget(), True) # takes_remoteinterface is specified to accept an RIDummy d = rr.callRemote('takes_remoteinterface', TypesTarget()) def _check_failure(f): f.trap(Violation) self.failUnlessIn("RITypes.takes_remoteinterface(a=))", str(f)) self.failUnlessIn(" does not provide RemoteInterface ", str(f)) self.failUnlessIn("foolscap.test.common.RIDummy", str(f)) self.deferredShouldFail(d, checker=_check_failure) return d def failUnlessRemoteProvides(self, obj, riface): # TODO: really, I want to just be able to say: # self.failUnless(RIDummy.providedBy(res)) iface = obj.tracker.interface # TODO: this test probably doesn't handle subclasses of # RemoteInterface, which might be useful (if it even works) if not iface or iface != riface: self.fail("%s does not provide RemoteInterface %s" % (obj, riface)) def testReturnsRemoteInterfaceGood(self): rr, target = self.setupTarget(TypesTarget(), True) d = rr.callRemote('returns_remoteinterface', 1) def _check(res): self.failUnless(isinstance(res, RemoteReference)) #self.failUnless(RIDummy.providedBy(res)) self.failUnlessRemoteProvides(res, RIDummy) d.addCallback(_check) return d def testReturnsRemoteInterfaceBad(self): rr, target = self.setupTarget(TypesTarget(), True) # returns_remoteinterface is specified to return an RIDummy d = rr.callRemote('returns_remoteinterface', 0) def _check_failure(f): f.trap(Violation) self.failUnlessIn("(in return value of .returns_remoteinterface)", str(f)) self.failUnlessIn("'15' is not a Referenceable", str(f)) self.deferredShouldFail(d, checker=_check_failure) return d def testReturnsRemoteInterfaceBad2(self): rr, target = self.setupTarget(TypesTarget(), True) # returns_remoteinterface is specified to return an RIDummy d = rr.callRemote('returns_remoteinterface', -1) def _check_failure(f): f.trap(Violation) self.failUnlessIn("(in return value of .returns_remoteinterface)", str(f)) self.failUnlessIn(" 4, "b.pings=%d, b.pongs=%d" % (b.pings, b.pongs)) # getDataLastReceivedAt() should be active last = rref.getDataLastReceivedAt() now = time.time() self.failUnless(-10 < now-last < 10, now-last) # and the connection should still be alive and usable return rref.callRemote("add", 1, 2) d.addCallback(_count_pings) def _check_add(res): self.failUnlessEqual(res, 3) d.addCallback(_check_add) return d def do_testDisconnect(self, which): # establish a connection with a very short disconnect timeout, so it # will be abandoned. We only set this on one side, since either the # initiating side or the receiving side should be able to timeout the # connection. Because we don't set keepaliveTimeout, there will be no # keepalives, so if we don't use the connection for 0.5 seconds, it # will be dropped. self.services[which].setOption("disconnectTimeout", 0.5) d = self.getRef() d.addCallback(self.stall, 2) def _check_ref(rref): d2 = rref.callRemote("add", 1, 2) def _check(res): self.failUnless(isinstance(res, Failure)) self.failUnless(res.check(DeadReferenceError), res.type) d2.addBoth(_check) return d2 d.addCallback(_check_ref) return d def testDisconnect0(self): return self.do_testDisconnect(0) def testDisconnect1(self): return self.do_testDisconnect(1) def do_testNoDisconnect(self, which): # establish a connection with a short disconnect timeout, but an even # shorter keepalive timeout, so the connection should stay alive. We # only provide the keepalives on one side, but enforce the disconnect # timeout on both: just one side doing keepalives should keep the # whole connection alive. self.services[which].setOption("keepaliveTimeout", 0.1) self.services[0].setOption("disconnectTimeout", 1.0) self.services[1].setOption("disconnectTimeout", 1.0) d = self.getRef() d.addCallback(self.stall, 2) def _check(rref): # the connection should still be alive return rref.callRemote("add", 1, 2) d.addCallback(_check) def _check_add(res): self.failUnlessEqual(res, 3) d.addCallback(_check_add) return d def testNoDisconnect0(self): return self.do_testNoDisconnect(0) def testNoDisconnect1(self): return self.do_testNoDisconnect(1) foolscap-0.13.1/src/foolscap/test/test_listener.py0000644000076500000240000000474413204160675022652 0ustar warnerstaff00000000000000from twisted.trial import unittest from twisted.internet import reactor, endpoints from twisted.internet.defer import inlineCallbacks from twisted.application import service from foolscap.api import Tub from foolscap.test.common import (certData_low, certData_high, Target, ShouldFailMixin) from foolscap import util class Listeners(ShouldFailMixin, unittest.TestCase): def setUp(self): self.s = service.MultiService() self.s.startService() def tearDown(self): return self.s.stopService() def makeTubs(self): tubA = Tub(certData=certData_low) tubA.setServiceParent(self.s) tubB = Tub(certData=certData_high) tubB.setServiceParent(self.s) return tubA, tubB @inlineCallbacks def test_string(self): tubA, tubB = self.makeTubs() portnum = util.allocate_tcp_port() tubA.listenOn("tcp:%d:interface=127.0.0.1" % portnum) tubA.setLocation("tcp:127.0.0.1:%d" % portnum) furl = tubA.registerReference(Target()) yield tubB.getReference(furl) @inlineCallbacks def test_endpoint(self): tubA, tubB = self.makeTubs() portnum = util.allocate_tcp_port() ep = endpoints.TCP4ServerEndpoint(reactor, portnum, interface="127.0.0.1") tubA.listenOn(ep) tubA.setLocation("tcp:127.0.0.1:%d" % portnum) furl = tubA.registerReference(Target()) yield tubB.getReference(furl) @inlineCallbacks def test_parsed_endpoint(self): tubA, tubB = self.makeTubs() portnum = util.allocate_tcp_port() ep = endpoints.serverFromString(reactor, "tcp:%d:interface=127.0.0.1" % portnum) tubA.listenOn(ep) tubA.setLocation("tcp:127.0.0.1:%d" % portnum) furl = tubA.registerReference(Target()) yield tubB.getReference(furl) @inlineCallbacks def test_nonqualified_port(self): tubA, tubB = self.makeTubs() portnum = util.allocate_tcp_port() import warnings with warnings.catch_warnings(): warnings.simplefilter("ignore") tubA.listenOn("%d" % portnum) # this is deprecated tubA.setLocation("tcp:127.0.0.1:%d" % portnum) furl = tubA.registerReference(Target()) yield tubB.getReference(furl) def test_invalid(self): tubA, tubB = self.makeTubs() self.assertRaises(TypeError, tubA.listenOn, 42) foolscap-0.13.1/src/foolscap/test/test_logging.py0000644000076500000240000032437613204511065022452 0ustar warnerstaff00000000000000 import os, sys, json, time, bz2, base64, re import mock from cStringIO import StringIO from zope.interface import implements from twisted.trial import unittest from twisted.application import service from twisted.internet import defer, reactor from twisted.internet.defer import inlineCallbacks, returnValue try: from twisted import logger as twisted_logger except ImportError: twisted_logger = None from twisted.web import client from twisted.python import log as twisted_log from twisted.python import failure, runtime, usage import foolscap from foolscap.logging import gatherer, log, tail, incident, cli, web, \ publish, dumper, flogfile from foolscap.logging.interfaces import RILogObserver from foolscap.util import format_time, allocate_tcp_port from foolscap.eventual import fireEventually, flushEventualQueue from foolscap.tokens import NoLocationError from foolscap.test.common import PollMixin, StallMixin from foolscap.api import RemoteException, Referenceable, Tub class Basic(unittest.TestCase): def testLog(self): l = log.FoolscapLogger() l.explain_facility("ui", "this terse string fully describes the gui") l.msg("one") l.msg("two") l.msg(message="three") l.msg("one=%d, two=%d", 1, 2) l.msg("survive 100% of weird inputs") l.msg(format="foo=%(foo)s, bar=%(bar)s", foo="foo", bar="bar") l.msg() # useless, but make sure it doesn't crash l.msg("ui message", facility="ui") l.msg("so boring it won't even be generated", level=log.NOISY-1) l.msg("blah blah", level=log.NOISY) l.msg("opening file", level=log.OPERATIONAL) l.msg("funny, that doesn't usually happen", level=log.UNUSUAL) l.msg("configuration change noticed", level=log.INFREQUENT) l.msg("error, but recoverable", level=log.CURIOUS) l.msg("ok, that shouldn't have happened", level=log.WEIRD) l.msg("hash doesn't match.. what the hell?", level=log.SCARY) l.msg("I looked into the trap, ray", level=log.BAD) def testStacktrace(self): l = log.FoolscapLogger() l.msg("how did we get here?", stacktrace=True) def testFailure(self): l = log.FoolscapLogger() f1 = failure.Failure(ValueError("bad value")) l.msg("failure1", failure=f1) # real RemoteExceptions always wrap CopiedFailure, so this is not # really accurate. However, it's a nuisance to create a real # CopiedFailure: look in # test_call.ExamineFailuresMixin._examine_raise for test code that # exercises this properly. f2 = failure.Failure(RemoteException(f1)) l.msg("failure2", failure=f2) def testParent(self): l = log.FoolscapLogger() p1 = l.msg("operation requested", level=log.OPERATIONAL) l.msg("first step", level=log.NOISY, parent=p1) l.msg("second step", level=log.NOISY, parent=p1) l.msg("second step EXPLODED", level=log.WEIRD, parent=p1) p2 = l.msg("third step", parent=p1) l.msg("fourth step", parent=p1) l.msg("third step deferred activity finally completed", parent=p2) l.msg("operation complete", level=log.OPERATIONAL, parent=p1) l.msg("override number, for some unknown reason", num=45) def testTheLogger(self): log.msg("This goes to the One True Logger") def testTubLogger(self): t = Tub() t.log("this goes into the tub") class Advanced(unittest.TestCase): def testObserver(self): l = log.FoolscapLogger() out = [] l.addObserver(out.append) l.set_generation_threshold(log.OPERATIONAL) l.msg("one") l.msg("two") l.msg("ignored", level=log.NOISY) d = fireEventually() def _check(res): self.failUnlessEqual(len(out), 2) self.failUnlessEqual(out[0]["message"], "one") self.failUnlessEqual(out[1]["message"], "two") d.addCallback(_check) return d def testFileObserver(self): basedir = "logging/Advanced/FileObserver" os.makedirs(basedir) l = log.FoolscapLogger() fn = os.path.join(basedir, "observer-log.out") ob = log.LogFileObserver(fn) l.addObserver(ob.msg) l.msg("one") l.msg("two") d = fireEventually() def _check(res): l.removeObserver(ob.msg) ob._logFile.close() f = open(fn, "rb") expected_magic = f.read(len(flogfile.MAGIC)) self.failUnlessEqual(expected_magic, flogfile.MAGIC) events = [] for line in f: events.append(json.loads(line)) self.failUnlessEqual(len(events), 3) self.failUnlessEqual(events[0]["header"]["type"], "log-file-observer") self.failUnlessEqual(events[0]["header"]["threshold"], log.OPERATIONAL) self.failUnlessEqual(events[1]["from"], "local") self.failUnlessEqual(events[2]["d"]["message"], "two") d.addCallback(_check) return d def testDisplace(self): l = log.FoolscapLogger() l.set_buffer_size(log.OPERATIONAL, 3) l.msg("one") l.msg("two") l.msg("three") items = l.buffers[None][log.OPERATIONAL] self.failUnlessEqual(len(items), 3) l.msg("four") # should displace "one" self.failUnlessEqual(len(items), 3) m0 = items[0] self.failUnlessEqual(type(m0), dict) self.failUnlessEqual(m0['message'], "two") self.failUnlessEqual(items[-1]['message'], "four") def testFacilities(self): l = log.FoolscapLogger() l.explain_facility("ui", "This is the UI.") l.msg("one", facility="ui") l.msg("two") items = l.buffers["ui"][log.OPERATIONAL] self.failUnlessEqual(len(items), 1) self.failUnlessEqual(items[0]["message"], "one") def testOnePriority(self): l = log.FoolscapLogger() l.msg("one", level=log.NOISY) l.msg("two", level=log.WEIRD) l.msg("three", level=log.NOISY) items = l.buffers[None][log.NOISY] self.failUnlessEqual(len(items), 2) self.failUnlessEqual(items[0]['message'], "one") self.failUnlessEqual(items[1]['message'], "three") items = l.buffers[None][log.WEIRD] self.failUnlessEqual(len(items), 1) self.failUnlessEqual(items[0]['message'], "two") def testPriorities(self): l = log.FoolscapLogger() l.set_buffer_size(log.NOISY, 3) l.set_buffer_size(log.WEIRD, 3) l.set_buffer_size(log.WEIRD, 4, "new.facility") l.msg("one", level=log.WEIRD) l.msg("two", level=log.NOISY) l.msg("three", level=log.NOISY) l.msg("four", level=log.WEIRD) l.msg("five", level=log.NOISY) l.msg("six", level=log.NOISY) l.msg("seven", level=log.NOISY) items = l.buffers[None][log.NOISY] self.failUnlessEqual(len(items), 3) self.failUnlessEqual(items[0]['message'], "five") self.failUnlessEqual(items[-1]['message'], "seven") items = l.buffers[None][log.WEIRD] self.failUnlessEqual(len(items), 2) self.failUnlessEqual(items[0]['message'], "one") self.failUnlessEqual(items[-1]['message'], "four") def testHierarchy(self): l = log.FoolscapLogger() n = l.msg("one") n2 = l.msg("two", parent=n) l.msg("three", parent=n2) class ErrorfulQualifier(incident.IncidentQualifier): def __init__(self): self._first = True def check_event(self, ev): if self._first: self._first = False raise ValueError("oops") return False class NoStdio(unittest.TestCase): # bug #244 is caused, in part, by Foolscap-side logging failures which # write an error message ("unable to serialize X") to stderr, which then # gets captured by twisted's logging (when run in a program under # twistd), then fed back into foolscap logging. Check that unserializable # objects don't cause anything to be written to a mock stdout/stderr # object. # # FoolscapLogger used stdio in two places: # * msg() when format_message() throws # * add_event() when IncidentQualifier.event() throws def setUp(self): self.fl = log.FoolscapLogger() self.mock_stdout = StringIO() self.mock_stderr = StringIO() self.orig_stdout = sys.stdout self.orig_stderr = sys.stderr sys.stdout = self.mock_stdout sys.stderr = self.mock_stderr def tearDown(self): sys.stdout = self.orig_stdout sys.stderr = self.orig_stderr def check_stdio(self): self.failUnlessEqual(self.mock_stdout.getvalue(), "") self.failUnlessEqual(self.mock_stderr.getvalue(), "") def test_unformattable(self): self.fl.msg(format="one=%(unformattable)s") # missing format key self.check_stdio() def test_unserializable_incident(self): # one #244 pathway involved an unserializable event that caused an # exception during IncidentReporter.incident_declared(), as it tried # to record all recent events. We can test the lack of stdio by using # a qualifier that throws an error directly. self.fl.setIncidentQualifier(ErrorfulQualifier()) self.fl.activate_incident_qualifier() # make sure we set it up correctly self.failUnless(self.fl.active_incident_qualifier) self.fl.msg("oops", arg=lambda : "lambdas are unserializable", level=log.BAD) self.check_stdio() # The internal error will cause a new "metaevent" to be recorded. The # original event may or may not get recorded first, depending upon # the error (i.e. does it happen before or after buffer.append is # called). Also, get_buffered_events() is unordered. So search for # the right one. events = [e for e in self.fl.get_buffered_events() if e.get("facility") == "foolscap/internal-error"] self.assertEqual(len(events), 1) m = events[0]["message"] expected = "internal error in log._msg, args=('oops',)" self.assert_(m.startswith(expected), m) self.assertIn("ValueError('oops'", m) def ser(what): return json.dumps(what, cls=flogfile.ExtendedEncoder) class Serialization(unittest.TestCase): def test_lazy_serialization(self): # Both foolscap and twisted allow (somewhat) arbitrary kwargs in the # log.msg() call. Twisted will either discard the event (if nobody is # listening), or stringify it right away. # # Foolscap does neither. It records the event (kwargs and all) in a # circular buffer, so a later observer can learn about them (either # 'flogtool tail' or a stored Incident file). And it stores the # arguments verbatim, leaving stringification to the future observer # (if they want it), so tools can filter events without using regexps # or parsing prematurely-flattened strings. # # Test this by logging a mutable object, modifying it, then checking # the buffer. We expect to see the modification. fl = log.FoolscapLogger() mutable = {"key": "old"} fl.msg("one", arg=mutable) mutable["key"] = "new" events = list(fl.get_buffered_events()) self.failUnless(events[0]["arg"]["key"], "new") def test_failure(self): try: raise ValueError("oops5") except ValueError: f = failure.Failure() out = json.loads(ser({"f": f}))["f"] self.assertEqual(out["@"], "Failure") self.assertIn("Failure exceptions.ValueError: oops5", out["repr"]) self.assertIn("traceback", out) def test_unserializable(self): # The code that serializes log events to disk (with JSON) tries very # hard to get *something* recorded, even when you give log.msg() # something strange. self.assertEqual(json.loads(ser({"a": 1})), {"a": 1}) unjsonable = [set([1,2])] self.assertEqual(json.loads(ser(unjsonable)), [{'@': 'UnJSONable', 'repr': 'set([1, 2])', 'message': "log.msg() was given an object that could not be encoded into JSON. I've replaced it with this UnJSONable object. The object's repr is in .repr"}]) # if the repr() fails, we get a different message class Unreprable: def __repr__(self): raise ValueError("oops7") unrep = [Unreprable()] self.assertEqual(json.loads(ser(unrep)), [{"@": "Unreprable", "exception_repr": "ValueError('oops7',)", "message": "log.msg() was given an object that could not be encoded into JSON, and when I tried to repr() it I got an error too. I've put the repr of the exception in .exception_repr", }]) # and if repr()ing the failed repr() exception fails, we give up real_repr = repr def really_bad_repr(o): if isinstance(o, ValueError): raise TypeError("oops9") return real_repr(o) import __builtin__ assert __builtin__.repr is repr with mock.patch("__builtin__.repr", really_bad_repr): s = ser(unrep) self.assertEqual(json.loads(s), [{"@": "ReallyUnreprable", "message": "log.msg() was given an object that could not be encoded into JSON, and when I tried to repr() it I got an error too. That exception wasn't repr()able either. I give up. Good luck.", }]) def test_not_pickle(self): # Older versions of Foolscap used pickle to store events into the # Incident log, and dealt with errors by dropping the event. Newer ones # use JSON, and use a placeholder when errors occur. Test that # pickleable (but not JSON-able) objects are *not* written to the file # directly, but are replaced by an "unjsonable" placeholder. basedir = "logging/Serialization/not_pickle" os.makedirs(basedir) fl = log.FoolscapLogger() ir = incident.IncidentReporter(basedir, fl, "tubid") ir.TRAILING_DELAY = None fl.msg("first") unjsonable = [object()] # still picklable unserializable = [lambda: "neither pickle nor JSON can capture me"] # having unserializble data in the logfile should not break the rest fl.msg("unjsonable", arg=unjsonable) fl.msg("unserializable", arg=unserializable) fl.msg("last") events = list(fl.get_buffered_events()) # if unserializable data breaks incident reporting, this # incident_declared() call will cause an exception ir.incident_declared(events[0]) # that won't record any trailing events, but does # eventually(finished_Recording), so wait for that to conclude d = flushEventualQueue() def _check(_): files = os.listdir(basedir) self.failUnlessEqual(len(files), 1) fn = os.path.join(basedir, files[0]) events = list(flogfile.get_events(fn)) self.failUnlessEqual(events[0]["header"]["type"], "incident") self.failUnlessEqual(events[1]["d"]["message"], "first") self.failUnlessEqual(len(events), 5) # actually this should record 5 events: both unrecordable events # should be replaced with error messages that *are* recordable self.failUnlessEqual(events[2]["d"]["message"], "unjsonable") self.failUnlessEqual(events[2]["d"]["arg"][0]["@"], "UnJSONable") self.failUnlessEqual(events[3]["d"]["message"], "unserializable") self.failUnlessEqual(events[3]["d"]["arg"][0]["@"], "UnJSONable") self.failUnlessEqual(events[4]["d"]["message"], "last") d.addCallback(_check) return d class SuperstitiousQualifier(incident.IncidentQualifier): def check_event(self, ev): if "thirteen" in ev.get("message", ""): return True return False class ImpatientReporter(incident.IncidentReporter): TRAILING_DELAY = 1.0 TRAILING_EVENT_LIMIT = 3 class NoFollowUpReporter(incident.IncidentReporter): TRAILING_DELAY = None class LogfileReaderMixin: def _read_logfile(self, fn): return list(flogfile.get_events(fn)) class Incidents(unittest.TestCase, PollMixin, LogfileReaderMixin): def test_basic(self): l = log.FoolscapLogger() self.failUnlessEqual(l.incidents_declared, 0) # no qualifiers are run until a logdir is provided l.msg("one", level=log.BAD) self.failUnlessEqual(l.incidents_declared, 0) l.setLogDir("logging/Incidents/basic") l.setLogDir("logging/Incidents/basic") # this should be idempotent got_logdir = l.logdir self.failUnlessEqual(got_logdir, os.path.abspath("logging/Incidents/basic")) # qualifiers should be run now l.msg("two") l.msg("3-trigger", level=log.BAD) self.failUnlessEqual(l.incidents_declared, 1) self.failUnless(l.get_active_incident_reporter()) # at this point, the uncompressed logfile should be present, and it # should contain all the events up to and including the trigger files = os.listdir(got_logdir) self.failUnlessEqual(len(files), 2) # the uncompressed one will sort earlier, since it lacks the .bz2 # extension files.sort() self.failUnlessEqual(files[0] + ".bz2.tmp", files[1]) # unix systems let us look inside the uncompressed file while it's # still being written to by the recorder if runtime.platformType == "posix": events = self._read_logfile(os.path.join(got_logdir, files[0])) self.failUnlessEqual(len(events), 1+3) #header = events[0] self.failUnless("header" in events[0]) self.failUnlessEqual(events[0]["header"]["trigger"]["message"], "3-trigger") self.failUnlessEqual(events[0]["header"]["versions"]["foolscap"], foolscap.__version__) self.failUnlessEqual(events[3]["d"]["message"], "3-trigger") l.msg("4-trailing") # this will take 5 seconds to finish trailing events d = self.poll(lambda: bool(l.incidents_recorded), 1.0) def _check(res): self.failUnlessEqual(len(l.recent_recorded_incidents), 1) fn = l.recent_recorded_incidents[0] events = self._read_logfile(fn) self.failUnlessEqual(len(events), 1+4) self.failUnless("header" in events[0]) self.failUnlessEqual(events[0]["header"]["trigger"]["message"], "3-trigger") self.failUnlessEqual(events[0]["header"]["versions"]["foolscap"], foolscap.__version__) self.failUnlessEqual(events[3]["d"]["message"], "3-trigger") self.failUnlessEqual(events[4]["d"]["message"], "4-trailing") d.addCallback(_check) return d def test_qualifier1(self): l = log.FoolscapLogger() l.setIncidentQualifier(SuperstitiousQualifier()) l.setLogDir("logging/Incidents/qualifier1") l.msg("1", level=log.BAD) self.failUnlessEqual(l.incidents_declared, 0) def test_qualifier2(self): l = log.FoolscapLogger() # call them in the other order l.setLogDir("logging/Incidents/qualifier2") l.setIncidentQualifier(SuperstitiousQualifier()) l.msg("1", level=log.BAD) self.failUnlessEqual(l.incidents_declared, 0) def test_customize(self): l = log.FoolscapLogger() l.setIncidentQualifier(SuperstitiousQualifier()) l.setLogDir("logging/Incidents/customize") # you set the reporter *class*, not an instance bad_ir = ImpatientReporter("basedir", "logger", "tubid") self.failUnlessRaises((AssertionError, TypeError), l.setIncidentReporterFactory, bad_ir) l.setIncidentReporterFactory(ImpatientReporter) l.msg("1", level=log.BAD) self.failUnlessEqual(l.incidents_declared, 0) l.msg("2") l.msg("thirteen is scary") self.failUnlessEqual(l.incidents_declared, 1) l.msg("4") l.msg("5") l.msg("6") # this should hit the trailing event limit l.msg("7") # this should not be recorded d = self.poll(lambda: bool(l.incidents_recorded), 1.0) def _check(res): self.failUnlessEqual(len(l.recent_recorded_incidents), 1) fn = l.recent_recorded_incidents[0] events = self._read_logfile(fn) self.failUnlessEqual(len(events), 1+6) self.failUnlessEqual(events[-1]["d"]["message"], "6") d.addCallback(_check) return d def test_overlapping(self): l = log.FoolscapLogger() l.setLogDir("logging/Incidents/overlapping") got_logdir = l.logdir self.failUnlessEqual(got_logdir, os.path.abspath("logging/Incidents/overlapping")) d = defer.Deferred() def _go(name, trigger): d.callback( (name, trigger) ) l.addImmediateIncidentObserver(_go) l.setIncidentReporterFactory(ImpatientReporter) l.msg("1") l.msg("2-trigger", level=log.BAD) self.failUnlessEqual(l.incidents_declared, 1) self.failUnless(l.get_active_incident_reporter()) l.msg("3-trigger", level=log.BAD) self.failUnlessEqual(l.incidents_declared, 2) self.failUnless(l.get_active_incident_reporter()) def _check(res): self.failUnlessEqual(l.incidents_recorded, 1) self.failUnlessEqual(len(l.recent_recorded_incidents), 1) # at this point, the logfile should be present, and it should # contain all the events up to and including both triggers files = os.listdir(got_logdir) self.failUnlessEqual(len(files), 1) events = self._read_logfile(os.path.join(got_logdir, files[0])) self.failUnlessEqual(len(events), 1+3) self.failUnlessEqual(events[0]["header"]["trigger"]["message"], "2-trigger") self.failUnlessEqual(events[1]["d"]["message"], "1") self.failUnlessEqual(events[2]["d"]["message"], "2-trigger") self.failUnlessEqual(events[3]["d"]["message"], "3-trigger") d.addCallback(_check) return d def test_classify(self): l = log.FoolscapLogger() l.setIncidentReporterFactory(incident.NonTrailingIncidentReporter) l.setLogDir("logging/Incidents/classify") got_logdir = l.logdir l.msg("foom", level=log.BAD, failure=failure.Failure(RuntimeError())) d = fireEventually() def _check(res): files = [fn for fn in os.listdir(got_logdir) if fn.endswith(".bz2")] self.failUnlessEqual(len(files), 1) ic = incident.IncidentClassifier() def classify_foom(trigger): if "foom" in trigger.get("message",""): return "foom" ic.add_classifier(classify_foom) options = incident.ClassifyOptions() options.parseOptions([os.path.join(got_logdir, fn) for fn in files]) options.stdout = StringIO() ic.run(options) out = options.stdout.getvalue() self.failUnless(out.strip().endswith(": foom"), out) ic2 = incident.IncidentClassifier() options = incident.ClassifyOptions() options.parseOptions(["--verbose"] + [os.path.join(got_logdir, fn) for fn in files]) options.stdout = StringIO() ic2.run(options) out = options.stdout.getvalue() self.failUnlessIn(".flog.bz2: unknown\n", out) # this should have a pprinted trigger dictionary self.failUnless(re.search(r"u?'message': u?'foom',", out), out) self.failUnlessIn("'num': 0,", out) self.failUnlessIn("RuntimeError", out) d.addCallback(_check) return d class Observer(Referenceable): implements(RILogObserver) def __init__(self): self.messages = [] self.incidents = [] self.done_with_incidents = False def remote_msg(self, d): self.messages.append(d) def remote_new_incident(self, name, trigger): self.incidents.append( (name, trigger) ) def remote_done_with_incident_catchup(self): self.done_with_incidents = True class MyGatherer(gatherer.GathererService): verbose = False def __init__(self, rotate, use_bzip, basedir): portnum = allocate_tcp_port() with open(os.path.join(basedir, "port"), "w") as f: f.write("tcp:%d\n" % portnum) with open(os.path.join(basedir, "location"), "w") as f: f.write("tcp:127.0.0.1:%d\n" % portnum) gatherer.GathererService.__init__(self, rotate, use_bzip, basedir) def remote_logport(self, nodeid, publisher): d = gatherer.GathererService.remote_logport(self, nodeid, publisher) d.addBoth(lambda res: self.d.callback(publisher)) class SampleError(Exception): """a sample error""" class Publish(PollMixin, unittest.TestCase): def setUp(self): self.parent = service.MultiService() self.parent.startService() # make the MAX_QUEUE_SIZE smaller to speed up the test, and restore # it when we're done. The normal value is 2000, chosen to bound the # queue to perhaps 1MB. Lowering the size from 2000 to 500 speeds up # the test from about 10s to 5s. self.saved_queue_size = publish.Subscription.MAX_QUEUE_SIZE publish.Subscription.MAX_QUEUE_SIZE = 500 def tearDown(self): publish.Subscription.MAX_QUEUE_SIZE = self.saved_queue_size d = defer.succeed(None) d.addCallback(lambda res: self.parent.stopService()) d.addCallback(flushEventualQueue) return d def test_logport_furlfile1(self): basedir = "logging/Publish/logport_furlfile1" os.makedirs(basedir) furlfile = os.path.join(basedir, "logport.furl") t = Tub() # setOption before setServiceParent t.setOption("logport-furlfile", furlfile) t.setServiceParent(self.parent) self.failUnlessRaises(NoLocationError, t.getLogPort) self.failUnlessRaises(NoLocationError, t.getLogPortFURL) portnum = allocate_tcp_port() t.listenOn("tcp:%d:interface=127.0.0.1" % portnum) self.failIf(os.path.exists(furlfile)) t.setLocation("127.0.0.1:%d" % portnum) logport_furl = open(furlfile, "r").read().strip() self.failUnlessEqual(logport_furl, t.getLogPortFURL()) def test_logport_furlfile2(self): basedir = "logging/Publish/logport_furlfile2" os.makedirs(basedir) furlfile = os.path.join(basedir, "logport.furl") t = Tub() # setServiceParent before setOption t.setServiceParent(self.parent) self.failUnlessRaises(NoLocationError, t.getLogPort) self.failUnlessRaises(NoLocationError, t.getLogPortFURL) portnum = allocate_tcp_port() t.listenOn("tcp:%d:interface=127.0.0.1" % portnum) t.setOption("logport-furlfile", furlfile) self.failIf(os.path.exists(furlfile)) t.setLocation("127.0.0.1:%d" % portnum) logport_furl = open(furlfile, "r").read().strip() self.failUnlessEqual(logport_furl, t.getLogPortFURL()) def test_logpublisher(self): basedir = "logging/Publish/logpublisher" os.makedirs(basedir) furlfile = os.path.join(basedir, "logport.furl") t = Tub() t.setServiceParent(self.parent) portnum = allocate_tcp_port() t.listenOn("tcp:%d:interface=127.0.0.1" % portnum) self.failUnlessRaises(NoLocationError, t.getLogPort) self.failUnlessRaises(NoLocationError, t.getLogPortFURL) t.setLocation("127.0.0.1:%d" % portnum) t.setOption("logport-furlfile", furlfile) logport_furl = t.getLogPortFURL() logport_furl2 = open(furlfile, "r").read().strip() self.failUnlessEqual(logport_furl, logport_furl2) tw_log = twisted_log.LogPublisher() tlb = t.setOption("bridge-twisted-logs", tw_log) t2 = Tub() t2.setServiceParent(self.parent) ob = Observer() d = t2.getReference(logport_furl) def _got_logport(logport): d = logport.callRemote("get_versions") def _check(versions): self.failUnlessEqual(versions["foolscap"], foolscap.__version__) d.addCallback(_check) # note: catch_up=False, so this message won't be sent log.msg("message 0 here, before your time") d.addCallback(lambda res: logport.callRemote("subscribe_to_all", ob)) def _emit(subscription): self._subscription = subscription log.msg("message 1 here") tw_log.msg("message 2 here") # switch to generic (no tubid) bridge log.unbridgeLogsFromTwisted(tw_log, tlb) log.bridgeLogsFromTwisted(None, tw_log) tw_log.msg("message 3 here") tw_log.msg(format="%(foo)s is foo", foo="foo") log.err(failure.Failure(SampleError("err1"))) log.err(SampleError("err2")) # simulate twisted.python.log.err, which is unfortunately # not a method of LogPublisher def err(_stuff=None, _why=None): if isinstance(_stuff, Exception): tw_log.msg(failure=failure.Failure(_stuff), isError=1, why=_why) else: tw_log.msg(failure=_stuff, isError=1, why=_why) err(failure.Failure(SampleError("err3"))) err(SampleError("err4")) d.addCallback(_emit) # wait until we've seen all the messages, or the test times out d.addCallback(lambda res: self.poll(lambda: len(ob.messages) >= 8)) def _check_observer(res): msgs = ob.messages self.failUnlessEqual(len(msgs), 8) self.failUnlessEqual(msgs[0]["message"], "message 1 here") self.failUnlessEqual(msgs[1]["from-twisted"], True) self.failUnlessEqual(msgs[1]["message"], "message 2 here") self.failUnlessEqual(msgs[1]["tubID"], t.tubID) self.failUnlessEqual(msgs[2]["from-twisted"], True) self.failUnlessEqual(msgs[2]["message"], "message 3 here") self.failUnlessEqual(msgs[2]["tubID"], None) self.failUnlessEqual(msgs[3]["from-twisted"], True) self.failUnlessEqual(msgs[3]["message"], "foo is foo") # check the errors self.failUnlessEqual(msgs[4]["message"], "") self.failUnless(msgs[4]["isError"]) self.failUnless("failure" in msgs[4]) self.failUnless(msgs[4]["failure"].check(SampleError)) self.failUnless("err1" in str(msgs[4]["failure"])) self.failUnlessEqual(msgs[5]["message"], "") self.failUnless(msgs[5]["isError"]) self.failUnless("failure" in msgs[5]) self.failUnless(msgs[5]["failure"].check(SampleError)) self.failUnless("err2" in str(msgs[5]["failure"])) # errors coming from twisted are stringified self.failUnlessEqual(msgs[6]["from-twisted"], True) self.failUnless("Unhandled Error" in msgs[6]["message"]) self.failUnless("SampleError: err3" in msgs[6]["message"]) self.failUnless(msgs[6]["isError"]) self.failUnlessEqual(msgs[7]["from-twisted"], True) self.failUnless("Unhandled Error" in msgs[7]["message"]) self.failUnless("SampleError: err4" in msgs[7]["message"]) self.failUnless(msgs[7]["isError"]) d.addCallback(_check_observer) def _done(res): return logport.callRemote("unsubscribe", self._subscription) d.addCallback(_done) return d d.addCallback(_got_logport) return d def test_logpublisher_overload(self): basedir = "logging/Publish/logpublisher_overload" os.makedirs(basedir) furlfile = os.path.join(basedir, "logport.furl") t = Tub() t.setServiceParent(self.parent) portnum = allocate_tcp_port() t.listenOn("tcp:%d:interface=127.0.0.1" % portnum) t.setLocation("127.0.0.1:%d" % portnum) t.setOption("logport-furlfile", furlfile) logport_furl = t.getLogPortFURL() logport_furl2 = open(furlfile, "r").read().strip() self.failUnlessEqual(logport_furl, logport_furl2) t2 = Tub() t2.setServiceParent(self.parent) ob = Observer() d = t2.getReference(logport_furl) def _got_logport(logport): d = logport.callRemote("subscribe_to_all", ob) def _emit(subscription): self._subscription = subscription for i in range(10000): log.msg("message %d here" % i) d.addCallback(_emit) # now we wait until the observer has seen nothing for a full # second. I'd prefer something faster and more deterministic, but # this ought to handle the normal slow-host cases. expected = publish.Subscription.MAX_QUEUE_SIZE def _check_f(): return bool(len(ob.messages) >= expected) d.addCallback(lambda res: self.poll(_check_f, 0.2)) # TODO: I'm not content with that polling, and would prefer to do # something faster and more deterministic #d.addCallback(fireEventually) #d.addCallback(fireEventually) def _check_observer(res): msgs = ob.messages self.failUnlessEqual(len(msgs), expected) # since we discard new messages during overload (and preserve # old ones), we should see 0..MAX_QUEUE_SIZE-1. got = [] for m in msgs: ignored1, number_s, ignored2 = m["message"].split() number = int(number_s) got.append(number) self.failUnlessEqual(got, sorted(got)) self.failUnlessEqual(got, range(expected)) d.addCallback(_check_observer) def _done(res): return logport.callRemote("unsubscribe", self._subscription) d.addCallback(_done) return d d.addCallback(_got_logport) return d def test_logpublisher_catchup(self): basedir = "logging/Publish/logpublisher_catchup" os.makedirs(basedir) furlfile = os.path.join(basedir, "logport.furl") t = Tub() t.setServiceParent(self.parent) portnum = allocate_tcp_port() t.listenOn("tcp:%d:interface=127.0.0.1" % portnum) t.setLocation("127.0.0.1:%d" % portnum) t.setOption("logport-furlfile", furlfile) logport_furl = t.getLogPortFURL() t2 = Tub() t2.setServiceParent(self.parent) ob = Observer() d = t2.getReference(logport_furl) def _got_logport(logport): d = logport.callRemote("get_versions") def _check_versions(versions): self.failUnlessEqual(versions["foolscap"], foolscap.__version__) d.addCallback(_check_versions) d.addCallback(lambda res: logport.callRemote("get_pid")) def _check_pid(pid): self.failUnlessEqual(pid, os.getpid()) d.addCallback(_check_pid) # note: catch_up=True, so this message *will* be sent. Also note # that we need this message to be unique, since our logger will # stash messages recorded by other test cases, and we don't want # to confuse the two. log.msg("this is an early message") d.addCallback(lambda res: logport.callRemote("subscribe_to_all", ob, True)) def _emit(subscription): self._subscription = subscription log.msg("this is a later message") d.addCallback(_emit) # wait until we've received the later message def _check_f(): for m in ob.messages: if m.get("message") == "this is a later message": return True return False d.addCallback(lambda res: self.poll(_check_f)) def _check_observer(res): msgs = ob.messages # this gets everything that's been logged since the unit # tests began. The Reconnector that's used by # logport-furlfile will cause some uncertainty.. negotiation # messages might be interleaved with the ones that we # actually care about. So what we verify is that both of our # messages appear *somewhere*, and that they show up in the # correct order. self.failUnless(len(msgs) >= 2, len(msgs)) first = None second = None for i,m in enumerate(msgs): if m.get("message") == "this is an early message": first = i if m.get("message") == "this is a later message": second = i self.failUnless(first is not None) self.failUnless(second is not None) self.failUnless(first < second, "%d is not before %d" % (first, second)) d.addCallback(_check_observer) def _done(res): return logport.callRemote("unsubscribe", self._subscription) d.addCallback(_done) return d d.addCallback(_got_logport) return d class IncidentPublisher(PollMixin, unittest.TestCase): def setUp(self): self.parent = service.MultiService() self.parent.startService() def tearDown(self): d = defer.succeed(None) d.addCallback(lambda res: self.parent.stopService()) d.addCallback(flushEventualQueue) return d def _write_to(self, logdir, fn, data="stuff"): f = open(os.path.join(logdir, fn), "w") f.write(data) f.close() def test_list_incident_names(self): basedir = "logging/IncidentPublisher/list_incident_names" os.makedirs(basedir) t = Tub() t.setLocation("127.0.0.1:1234") t.logger = self.logger = log.FoolscapLogger() logdir = os.path.join(basedir, "logdir") t.logger.setLogDir(logdir) p = t.getLogPort() # dump some other files in the incident directory self._write_to(logdir, "distraction.bz2") self._write_to(logdir, "noise") # and a few real-looking incidents I1 = "incident-2008-07-29-204211-aspkxoi" I2 = "incident-2008-07-30-112233-wodaei" I1_abs = os.path.abspath(os.path.join(logdir, I1 + ".flog")) I2_abs = os.path.abspath(os.path.join(logdir, I2 + ".flog.bz2")) self._write_to(logdir, I1 + ".flog") self._write_to(logdir, I2 + ".flog.bz2") all = list(p.list_incident_names()) self.failUnlessEqual(set([name for (name,fn) in all]), set([I1, I2])) imap = dict(all) self.failUnlessEqual(imap[I1], I1_abs) self.failUnlessEqual(imap[I2], I2_abs) new = list(p.list_incident_names(since=I1)) self.failUnlessEqual(set([name for (name,fn) in new]), set([I2])) def test_get_incidents(self): basedir = "logging/IncidentPublisher/get_incidents" os.makedirs(basedir) furlfile = os.path.join(basedir, "logport.furl") t = Tub() t.logger = self.logger = log.FoolscapLogger() logdir = os.path.join(basedir, "logdir") t.logger.setLogDir(logdir) t.logger.setIncidentReporterFactory(incident.NonTrailingIncidentReporter) # dump some other files in the incident directory f = open(os.path.join(logdir, "distraction.bz2"), "w") f.write("stuff") f.close() f = open(os.path.join(logdir, "noise"), "w") f.write("stuff") f.close() # fill the buffers with some messages t.logger.msg("one") t.logger.msg("two") # and trigger an incident t.logger.msg("three", level=log.WEIRD) # the NonTrailingIncidentReporter needs a turn before it will have # finished recording the event: the getReference() call will suffice. # now set up a Tub to connect to the logport t.setServiceParent(self.parent) portnum = allocate_tcp_port() t.listenOn("tcp:%d:interface=127.0.0.1" % portnum) t.setLocation("127.0.0.1:%d" % portnum) t.setOption("logport-furlfile", furlfile) logport_furl = t.getLogPortFURL() logport_furl2 = open(furlfile, "r").read().strip() self.failUnlessEqual(logport_furl, logport_furl2) t2 = Tub() t2.setServiceParent(self.parent) d = t2.getReference(logport_furl) def _got_logport(logport): d = logport.callRemote("list_incidents") d.addCallback(self._check_listed) d.addCallback(lambda res: logport.callRemote("get_incident", self.i_name)) d.addCallback(self._check_incident) def _decompress(res): # now we manually decompress the logfile for that incident, # to exercise the code that provides access to incidents that # did not finish their trailing-gather by the time the # application was shut down assert not self.i_name.endswith(".bz2") fn1 = os.path.join(logdir, self.i_name) + ".flog.bz2" fn2 = fn1[:-len(".bz2")] f1 = bz2.BZ2File(fn1, "r") f2 = open(fn2, "wb") f2.write(f1.read()) f2.close() f1.close() os.unlink(fn1) d.addCallback(_decompress) # and do it again d.addCallback(lambda res: logport.callRemote("list_incidents")) d.addCallback(self._check_listed) d.addCallback(lambda res: logport.callRemote("get_incident", self.i_name)) d.addCallback(self._check_incident) return d d.addCallback(_got_logport) return d def _check_listed(self, incidents): self.failUnless(isinstance(incidents, dict)) self.failUnlessEqual(len(incidents), 1) self.i_name = i_name = incidents.keys()[0] self.failUnless(i_name.startswith("incident")) self.failIf(i_name.endswith(".flog") or i_name.endswith(".bz2")) trigger = incidents[i_name] self.failUnlessEqual(trigger["message"], "three") def _check_incident(self, (header, events) ): self.failUnlessEqual(header["type"], "incident") self.failUnlessEqual(header["trigger"]["message"], "three") self.failUnlessEqual(len(events), 3) self.failUnlessEqual(events[0]["message"], "one") def test_subscribe(self): basedir = "logging/IncidentPublisher/subscribe" os.makedirs(basedir) t = Tub() t.logger = self.logger = log.FoolscapLogger() logdir = os.path.join(basedir, "logdir") t.logger.setLogDir(logdir) t.logger.setIncidentReporterFactory(incident.NonTrailingIncidentReporter) # fill the buffers with some messages t.logger.msg("boring") t.logger.msg("blah") # and trigger the first incident t.logger.msg("one", level=log.WEIRD) # the NonTrailingIncidentReporter needs a turn before it will have # finished recording the event: the getReference() call will suffice. # now set up a Tub to connect to the logport t.setServiceParent(self.parent) portnum = allocate_tcp_port() t.listenOn("tcp:%d:interface=127.0.0.1" % portnum) t.setLocation("127.0.0.1:%d" % portnum) logport_furl = t.getLogPortFURL() ob = Observer() t2 = Tub() t2.setServiceParent(self.parent) d = t2.getReference(logport_furl) def _got_logport(logport): self._logport = logport d2 = logport.callRemote("subscribe_to_incidents", ob) # no catchup return d2 d.addCallback(_got_logport) def _subscribed(subscription): self._subscription = subscription d.addCallback(_subscribed) # pause long enough for the incident names to change d.addCallback(lambda res: time.sleep(2)) d.addCallback(lambda res: t.logger.msg("two", level=log.WEIRD)) d.addCallback(lambda res: self.poll(lambda: bool(ob.incidents), 0.1)) def _triggerof(incident): (name, trigger) = incident return trigger["message"] def _check_new(res): self.failUnlessEqual(len(ob.incidents), 1) self.failUnlessEqual(_triggerof(ob.incidents[0]), "two") d.addCallback(_check_new) d.addCallback(lambda res: self._subscription.callRemote("unsubscribe")) # now subscribe and catch up on all incidents ob2 = Observer() d.addCallback(lambda res: self._logport.callRemote("subscribe_to_incidents", ob2, True, "")) d.addCallback(_subscribed) d.addCallback(lambda res: self.poll(lambda: ob2.done_with_incidents, 0.1)) def _check_all(res): self.failUnlessEqual(len(ob2.incidents), 2) self.failUnlessEqual(_triggerof(ob2.incidents[0]), "one") self.failUnlessEqual(_triggerof(ob2.incidents[1]), "two") d.addCallback(_check_all) d.addCallback(lambda res: time.sleep(2)) d.addCallback(lambda res: t.logger.msg("three", level=log.WEIRD)) d.addCallback(lambda res: self.poll(lambda: len(ob2.incidents) >= 3, 0.1)) def _check_all2(res): self.failUnlessEqual(len(ob2.incidents), 3) self.failUnlessEqual(_triggerof(ob2.incidents[0]), "one") self.failUnlessEqual(_triggerof(ob2.incidents[1]), "two") self.failUnlessEqual(_triggerof(ob2.incidents[2]), "three") d.addCallback(_check_all2) d.addCallback(lambda res: self._subscription.callRemote("unsubscribe")) # test the since= argument, setting it equal to the name of the # second incident. This should give us the third incident. ob3 = Observer() d.addCallback(lambda res: self._logport.callRemote("subscribe_to_incidents", ob3, True, ob2.incidents[1][0])) d.addCallback(_subscribed) d.addCallback(lambda res: self.poll(lambda: ob3.done_with_incidents, 0.1)) def _check_since(res): self.failUnlessEqual(len(ob3.incidents), 1) self.failUnlessEqual(_triggerof(ob3.incidents[0]), "three") d.addCallback(_check_since) d.addCallback(lambda res: time.sleep(2)) d.addCallback(lambda res: t.logger.msg("four", level=log.WEIRD)) d.addCallback(lambda res: self.poll(lambda: len(ob3.incidents) >= 2, 0.1)) def _check_since2(res): self.failUnlessEqual(len(ob3.incidents), 2) self.failUnlessEqual(_triggerof(ob3.incidents[0]), "three") self.failUnlessEqual(_triggerof(ob3.incidents[1]), "four") d.addCallback(_check_since2) d.addCallback(lambda res: self._subscription.callRemote("unsubscribe")) return d test_subscribe.timeout = 20 class MyIncidentGathererService(gatherer.IncidentGathererService): verbose = False cb_new_incident = None def remote_logport(self, nodeid, publisher): d = gatherer.IncidentGathererService.remote_logport(self, nodeid, publisher) d.addCallback(lambda res: self.d.callback(publisher)) return d def new_incident(self, abs_fn, rel_fn, nodeid_s, incident): gatherer.IncidentGathererService.new_incident(self, abs_fn, rel_fn, nodeid_s, incident) if self.cb_new_incident: self.cb_new_incident((abs_fn, rel_fn)) class IncidentGatherer(unittest.TestCase, PollMixin, StallMixin, LogfileReaderMixin): def setUp(self): self.parent = service.MultiService() self.parent.startService() self.logger = log.FoolscapLogger() self.logger.setIncidentReporterFactory(NoFollowUpReporter) def tearDown(self): d = defer.succeed(None) d.addCallback(lambda res: self.parent.stopService()) d.addCallback(flushEventualQueue) return d def create_incident_gatherer(self, basedir, classifiers=[]): # create an incident gatherer, which will make its own Tub ig_basedir = os.path.join(basedir, "ig") if not os.path.isdir(ig_basedir): os.mkdir(ig_basedir) portnum = allocate_tcp_port() with open(os.path.join(ig_basedir, "port"), "w") as f: f.write("tcp:%d\n" % portnum) with open(os.path.join(ig_basedir, "location"), "w") as f: f.write("tcp:127.0.0.1:%d\n" % portnum) null = StringIO() ig = MyIncidentGathererService(classifiers=classifiers, basedir=ig_basedir, stdout=null) ig.d = defer.Deferred() return ig def create_connected_tub(self, ig): t = Tub() t.logger = self.logger t.setServiceParent(self.parent) portnum = allocate_tcp_port() t.listenOn("tcp:%d:interface=127.0.0.1" % portnum) t.setLocation("127.0.0.1:%d" % portnum) t.setOption("log-gatherer-furl", ig.my_furl) def test_connect(self): basedir = "logging/IncidentGatherer/connect" os.makedirs(basedir) self.logger.setLogDir(basedir) ig = self.create_incident_gatherer(basedir) ig.setServiceParent(self.parent) self.create_connected_tub(ig) d = ig.d # give the call to remote_logport a chance to retire d.addCallback(self.stall, 0.5) return d def test_emit(self): basedir = "logging/IncidentGatherer/emit" os.makedirs(basedir) self.logger.setLogDir(basedir) ig = self.create_incident_gatherer(basedir) ig.setServiceParent(self.parent) incident_d = defer.Deferred() ig.cb_new_incident = incident_d.callback self.create_connected_tub(ig) d = ig.d d.addCallback(lambda res: self.logger.msg("boom", level=log.WEIRD)) d.addCallback(lambda res: incident_d) def _new_incident((abs_fn, rel_fn)): events = self._read_logfile(abs_fn) header = events[0]["header"] self.failUnless("trigger" in header) self.failUnlessEqual(header["trigger"]["message"], "boom") e = events[1]["d"] self.failUnlessEqual(e["message"], "boom") # it should have been classified as "unknown" unknowns_fn = os.path.join(ig.basedir, "classified", "unknown") unknowns = [fn.strip() for fn in open(unknowns_fn,"r").readlines()] self.failUnlessEqual(len(unknowns), 1) self.failUnlessEqual(unknowns[0], rel_fn) d.addCallback(_new_incident) # now shut down the gatherer, create a new one with the same basedir # (with some classifier functions), remove the existing # classifications, and start it up. It should reclassify everything # at startup. d.addCallback(lambda res: ig.disownServiceParent()) def classify_boom(trigger): if "boom" in trigger.get("message",""): return "boom" def classify_foom(trigger): if "foom" in trigger.get("message",""): return "foom" incident_d2 = defer.Deferred() def _update_classifiers(res): self.remove_classified_incidents(ig) ig2 = self.create_incident_gatherer(basedir, [classify_boom]) ##ig2.add_classifier(classify_foom) # we add classify_foom by writing it into a file, to exercise the # look-for-classifier-files code foomfile = os.path.join(ig2.basedir, "classify_foom.py") f = open(foomfile, "w") f.write(''' def classify_incident(trigger): if "foom" in trigger.get("message",""): return "foom" ''') f.close() ig2.setServiceParent(self.parent) # now that it's been read, delete it to avoid affecting later # runs os.unlink(foomfile) self.ig2 = ig2 # incidents should be classified in startService unknowns_fn = os.path.join(ig.basedir, "classified", "unknown") self.failIf(os.path.exists(unknowns_fn)) booms_fn = os.path.join(ig.basedir, "classified", "boom") booms = [fn.strip() for fn in open(booms_fn,"r").readlines()] self.failUnlessEqual(len(booms), 1) fooms_fn = os.path.join(ig.basedir, "classified", "foom") self.failIf(os.path.exists(fooms_fn)) ig2.cb_new_incident = incident_d2.callback return ig2.d d.addCallback(_update_classifiers) d.addCallback(lambda res: self.logger.msg("foom", level=log.WEIRD)) d.addCallback(lambda res: incident_d2) def _new_incident2((abs_fn, rel_fn)): # this one should be classified as "foom" # it should have been classified as "unknown" fooms_fn = os.path.join(ig.basedir, "classified", "foom") fooms = [fn.strip() for fn in open(fooms_fn,"r").readlines()] self.failUnlessEqual(len(fooms), 1) self.failUnlessEqual(fooms[0], rel_fn) unknowns_fn = os.path.join(ig.basedir, "classified", "unknown") self.failIf(os.path.exists(unknowns_fn)) d.addCallback(_new_incident2) d.addCallback(lambda res: self.ig2.disownServiceParent()) # if we remove just classified/boom, then those incidents should be # reclassified def _remove_boom_incidents(res): booms_fn = os.path.join(ig.basedir, "classified", "boom") os.remove(booms_fn) ig2a = self.create_incident_gatherer(basedir, [classify_boom, classify_foom]) ig2a.setServiceParent(self.parent) self.ig2a = ig2a # now classified/boom should be back, and the other files should # have been left untouched booms = [fn.strip() for fn in open(booms_fn,"r").readlines()] self.failUnlessEqual(len(booms), 1) d.addCallback(_remove_boom_incidents) d.addCallback(lambda res: self.ig2a.disownServiceParent()) # and if we remove the classification functions (but do *not* remove # the classified incidents), the new gatherer should not reclassify # anything def _update_classifiers_again(res): ig3 = self.create_incident_gatherer(basedir) ig3.setServiceParent(self.parent) self.ig3 = ig3 unknowns_fn = os.path.join(ig.basedir, "classified", "unknown") self.failIf(os.path.exists(unknowns_fn)) booms_fn = os.path.join(ig.basedir, "classified", "boom") booms = [fn.strip() for fn in open(booms_fn,"r").readlines()] self.failUnlessEqual(len(booms), 1) fooms_fn = os.path.join(ig.basedir, "classified", "foom") fooms = [fn.strip() for fn in open(fooms_fn,"r").readlines()] self.failUnlessEqual(len(fooms), 1) return ig3.d d.addCallback(_update_classifiers_again) d.addCallback(lambda res: self.ig3.disownServiceParent()) # and if we remove all the stored incidents (and the 'latest' # record), the gatherer will grab everything. This exercises the # only-grab-one-at-a-time code. I verified this manually, by adding a # print to the avoid-duplicate clause of # IncidentObserver.maybe_fetch_incident . def _create_ig4(res): ig4 = self.create_incident_gatherer(basedir) for nodeid in os.listdir(os.path.join(ig4.basedir, "incidents")): nodedir = os.path.join(ig4.basedir, "incidents", nodeid) for fn in os.listdir(nodedir): os.unlink(os.path.join(nodedir, fn)) os.rmdir(nodedir) ig4.setServiceParent(self.parent) self.ig4 = ig4 d.addCallback(_create_ig4) d.addCallback(lambda res: self.poll(lambda : self.ig4.incidents_received == 2)) return d def remove_classified_incidents(self, ig): classified = os.path.join(ig.basedir, "classified") for category in os.listdir(classified): os.remove(os.path.join(classified, category)) os.rmdir(classified) class Gatherer(unittest.TestCase, LogfileReaderMixin, StallMixin, PollMixin): def setUp(self): self.parent = service.MultiService() self.parent.startService() def tearDown(self): d = defer.succeed(None) d.addCallback(lambda res: self.parent.stopService()) d.addCallback(flushEventualQueue) return d def _emit_messages_and_flush(self, res, t): log.msg("gathered message here") try: raise SampleError("whoops1") except: log.err() try: raise SampleError("whoops2") except SampleError: log.err(failure.Failure()) d = self.stall(None, 1.0) d.addCallback(lambda res: t.disownServiceParent()) # that will disconnect from the gatherer, which will flush the logfile d.addCallback(self.stall, 1.0) return d def _check_gatherer(self, fn, starting_timestamp, expected_tubid): events = [] for e in self._read_logfile(fn): # discard internal foolscap events, like connection # negotiation if "d" in e and "foolscap" in e["d"].get("facility", ""): pass else: events.append(e) if len(events) != 4: from pprint import pprint pprint(events) self.failUnlessEqual(len(events), 4) # header data = events.pop(0) self.failUnless(isinstance(data, dict)) self.failUnless("header" in data) self.failUnlessEqual(data["header"]["type"], "gatherer") self.failUnlessEqual(data["header"]["start"], starting_timestamp) # grab the first event from the log data = events.pop(0) self.failUnless(isinstance(data, dict)) self.failUnlessEqual(data['from'], expected_tubid) self.failUnlessEqual(data['d']['message'], "gathered message here") # grab the second event from the log data = events.pop(0) self.failUnless(isinstance(data, dict)) self.failUnlessEqual(data['from'], expected_tubid) self.failUnlessEqual(data['d']['message'], "") self.failUnless(data['d']["isError"]) self.failUnless("failure" in data['d']) self.failUnlessIn("SampleError", data['d']["failure"]["repr"]) self.failUnlessIn("whoops1", data['d']["failure"]["repr"]) # grab the third event from the log data = events.pop(0) self.failUnless(isinstance(data, dict)) self.failUnlessEqual(data['from'], expected_tubid) self.failUnlessEqual(data['d']['message'], "") self.failUnless(data['d']["isError"]) self.failUnless("failure" in data['d']) self.failUnlessIn("SampleError", data['d']["failure"]["repr"]) self.failUnlessIn("whoops2", data['d']["failure"]["repr"]) def test_wrongdir(self): basedir = "logging/Gatherer/wrongdir" os.makedirs(basedir) # create a LogGatherer with an unspecified basedir: it should look # for a .tac file in the current directory, not see it, and complain e = self.failUnlessRaises(RuntimeError, gatherer.GathererService, None, True, None) self.failUnless("running in the wrong directory" in str(e)) def test_log_gatherer(self): # setLocation, then set log-gatherer-furl. Also, use bzip=True for # this one test. basedir = "logging/Gatherer/log_gatherer" os.makedirs(basedir) # create a gatherer, which will create its own Tub gatherer = MyGatherer(None, True, basedir) gatherer.d = defer.Deferred() gatherer.setServiceParent(self.parent) # that will start the gatherer gatherer_furl = gatherer.my_furl starting_timestamp = gatherer._starting_timestamp t = Tub() expected_tubid = t.tubID assert t.tubID is not None t.setServiceParent(self.parent) portnum = allocate_tcp_port() t.listenOn("tcp:%d:interface=127.0.0.1" % portnum) t.setLocation("127.0.0.1:%d" % portnum) t.setOption("log-gatherer-furl", gatherer_furl) # about now, the node will be contacting the Gatherer and # offering its logport. # gatherer.d will be fired when subscribe_to_all() has finished d = gatherer.d d.addCallback(self._emit_messages_and_flush, t) # We use do_rotate() to force logfile rotation before checking # contents of the file, so we know it's been written out to disk d.addCallback(lambda res: gatherer.do_rotate()) d.addCallback(self._check_gatherer, starting_timestamp, expected_tubid) return d test_log_gatherer.timeout = 20 def test_log_gatherer_multiple(self): # setLocation, then set log-gatherer-furl. basedir = "logging/Gatherer/log_gatherer_multiple" os.makedirs(basedir) # create a gatherer, which will create its own Tub gatherer1_basedir = os.path.join(basedir, "gatherer1") os.makedirs(gatherer1_basedir) gatherer1 = MyGatherer(None, False, gatherer1_basedir) gatherer1.d = defer.Deferred() gatherer1.setServiceParent(self.parent) # that will start the gatherer gatherer1_furl = gatherer1.my_furl starting_timestamp1 = gatherer1._starting_timestamp # create a second one gatherer2_basedir = os.path.join(basedir, "gatherer2") os.makedirs(gatherer2_basedir) gatherer2 = MyGatherer(None, False, gatherer2_basedir) gatherer2.d = defer.Deferred() gatherer2.setServiceParent(self.parent) # that will start the gatherer gatherer2_furl = gatherer2.my_furl starting_timestamp2 = gatherer2._starting_timestamp t = Tub() expected_tubid = t.tubID assert t.tubID is not None t.setServiceParent(self.parent) portnum = allocate_tcp_port() t.listenOn("tcp:%d:interface=127.0.0.1" % portnum) t.setLocation("127.0.0.1:%d" % portnum) t.setOption("log-gatherer-furl", (gatherer1_furl, gatherer2_furl)) # about now, the node will be contacting the Gatherers and # offering its logport. # gatherer.d and gatherer2.d will be fired when subscribe_to_all() # has finished dl = defer.DeferredList([gatherer1.d, gatherer2.d]) dl.addCallback(self._emit_messages_and_flush, t) dl.addCallback(lambda res: gatherer1.do_rotate()) dl.addCallback(self._check_gatherer, starting_timestamp1, expected_tubid) dl.addCallback(lambda res: gatherer2.do_rotate()) dl.addCallback(self._check_gatherer, starting_timestamp2, expected_tubid) return dl test_log_gatherer_multiple.timeout = 40 def test_log_gatherer2(self): # set log-gatherer-furl, then setLocation. Also, use a timed rotator. basedir = "logging/Gatherer/log_gatherer2" os.makedirs(basedir) # create a gatherer, which will create its own Tub gatherer = MyGatherer(3600, False, basedir) gatherer.d = defer.Deferred() gatherer.setServiceParent(self.parent) # that will start the gatherer gatherer_furl = gatherer.my_furl starting_timestamp = gatherer._starting_timestamp t = Tub() expected_tubid = t.tubID assert t.tubID is not None t.setServiceParent(self.parent) portnum = allocate_tcp_port() t.listenOn("tcp:%d:interface=127.0.0.1" % portnum) t.setOption("log-gatherer-furl", gatherer_furl) t.setLocation("127.0.0.1:%d" % portnum) d = gatherer.d d.addCallback(self._emit_messages_and_flush, t) d.addCallback(lambda res: gatherer.do_rotate()) d.addCallback(self._check_gatherer, starting_timestamp, expected_tubid) return d test_log_gatherer2.timeout = 20 def test_log_gatherer_furlfile(self): # setLocation, then set log-gatherer-furlfile basedir = "logging/Gatherer/log_gatherer_furlfile" os.makedirs(basedir) # create a gatherer, which will create its own Tub gatherer = MyGatherer(None, False, basedir) gatherer.d = defer.Deferred() gatherer.setServiceParent(self.parent) # that will start the gatherer gatherer_furlfile = os.path.join(basedir, gatherer.furlFile) starting_timestamp = gatherer._starting_timestamp t = Tub() expected_tubid = t.tubID assert t.tubID is not None t.setServiceParent(self.parent) portnum = allocate_tcp_port() t.listenOn("tcp:%d:interface=127.0.0.1" % portnum) t.setLocation("127.0.0.1:%d" % portnum) t.setOption("log-gatherer-furlfile", gatherer_furlfile) d = gatherer.d d.addCallback(self._emit_messages_and_flush, t) d.addCallback(lambda res: gatherer.do_rotate()) d.addCallback(self._check_gatherer, starting_timestamp, expected_tubid) return d test_log_gatherer_furlfile.timeout = 20 def test_log_gatherer_furlfile2(self): # set log-gatherer-furlfile, then setLocation basedir = "logging/Gatherer/log_gatherer_furlfile2" os.makedirs(basedir) # create a gatherer, which will create its own Tub gatherer = MyGatherer(None, False, basedir) gatherer.d = defer.Deferred() gatherer.setServiceParent(self.parent) # that will start the gatherer gatherer_furlfile = os.path.join(basedir, gatherer.furlFile) starting_timestamp = gatherer._starting_timestamp t = Tub() expected_tubid = t.tubID assert t.tubID is not None t.setServiceParent(self.parent) portnum = allocate_tcp_port() t.listenOn("tcp:%d:interface=127.0.0.1" % portnum) t.setOption("log-gatherer-furlfile", gatherer_furlfile) # one bug we had was that the log-gatherer was contacted before # setLocation had occurred, so exercise that case d = self.stall(None, 1.0) def _start(res): t.setLocation("127.0.0.1:%d" % portnum) return gatherer.d d.addCallback(_start) d.addCallback(self._emit_messages_and_flush, t) d.addCallback(lambda res: gatherer.do_rotate()) d.addCallback(self._check_gatherer, starting_timestamp, expected_tubid) return d test_log_gatherer_furlfile2.timeout = 20 def test_log_gatherer_furlfile_multiple(self): basedir = "logging/Gatherer/log_gatherer_furlfile_multiple" os.makedirs(basedir) gatherer1_basedir = os.path.join(basedir, "gatherer1") os.makedirs(gatherer1_basedir) gatherer1 = MyGatherer(None, False, gatherer1_basedir) gatherer1.d = defer.Deferred() gatherer1.setServiceParent(self.parent) # that will start the gatherer gatherer1_furl = gatherer1.my_furl starting_timestamp1 = gatherer1._starting_timestamp gatherer2_basedir = os.path.join(basedir, "gatherer2") os.makedirs(gatherer2_basedir) gatherer2 = MyGatherer(None, False, gatherer2_basedir) gatherer2.d = defer.Deferred() gatherer2.setServiceParent(self.parent) # that will start the gatherer gatherer2_furl = gatherer2.my_furl starting_timestamp2 = gatherer2._starting_timestamp gatherer3_basedir = os.path.join(basedir, "gatherer3") os.makedirs(gatherer3_basedir) gatherer3 = MyGatherer(None, False, gatherer3_basedir) gatherer3.d = defer.Deferred() gatherer3.setServiceParent(self.parent) # that will start the gatherer gatherer3_furl = gatherer3.my_furl starting_timestamp3 = gatherer3._starting_timestamp gatherer_furlfile = os.path.join(basedir, "log_gatherer.furl") f = open(gatherer_furlfile, "w") f.write(gatherer1_furl + "\n") f.write(gatherer2_furl + "\n") f.close() t = Tub() expected_tubid = t.tubID assert t.tubID is not None t.setOption("log-gatherer-furl", gatherer3_furl) t.setServiceParent(self.parent) portnum = allocate_tcp_port() t.listenOn("tcp:%d:interface=127.0.0.1" % portnum) t.setLocation("127.0.0.1:%d" % portnum) t.setOption("log-gatherer-furlfile", gatherer_furlfile) # now both log gatherer connections will be being established d = defer.DeferredList([gatherer1.d, gatherer2.d, gatherer3.d], fireOnOneErrback=True) d.addCallback(self._emit_messages_and_flush, t) d.addCallback(lambda res: gatherer1.do_rotate()) d.addCallback(self._check_gatherer, starting_timestamp1, expected_tubid) d.addCallback(lambda res: gatherer2.do_rotate()) d.addCallback(self._check_gatherer, starting_timestamp2, expected_tubid) d.addCallback(lambda res: gatherer3.do_rotate()) d.addCallback(self._check_gatherer, starting_timestamp3, expected_tubid) return d test_log_gatherer_furlfile_multiple.timeout = 20 def test_log_gatherer_empty_furlfile(self): basedir = "logging/Gatherer/log_gatherer_empty_furlfile" os.makedirs(basedir) gatherer_fn = os.path.join(basedir, "lg.furl") open(gatherer_fn, "w").close() # leave the furlfile empty: use no gatherer t = Tub() t.setServiceParent(self.parent) portnum = allocate_tcp_port() t.listenOn("tcp:%d:interface=127.0.0.1" % portnum) t.setLocation("127.0.0.1:%d" % portnum) t.setOption("log-gatherer-furlfile", gatherer_fn) lp_furl = t.getLogPortFURL() del lp_furl t.log("this message shouldn't make anything explode") test_log_gatherer_empty_furlfile.timeout = 20 def test_log_gatherer_missing_furlfile(self): basedir = "logging/Gatherer/log_gatherer_missing_furlfile" os.makedirs(basedir) gatherer_fn = os.path.join(basedir, "missing_lg.furl") open(gatherer_fn, "w").close() # leave the furlfile missing: use no gatherer t = Tub() t.setServiceParent(self.parent) portnum = allocate_tcp_port() t.listenOn("tcp:%d:interface=127.0.0.1" % portnum) t.setLocation("127.0.0.1:%d" % portnum) t.setOption("log-gatherer-furlfile", gatherer_fn) lp_furl = t.getLogPortFURL() del lp_furl t.log("this message shouldn't make anything explode") test_log_gatherer_missing_furlfile.timeout = 20 class Tail(unittest.TestCase): def test_logprinter(self): target_tubid_s = "jiijpvbge2e3c3botuzzz7la3utpl67v" options1 = {"save-to": None, "verbose": None, "timestamps": "short-local"} out = StringIO() lp = tail.LogPrinter(options1, target_tubid_s[:8], out) lp.got_versions({}) lp.remote_msg({"time": 1207005906.527782, "level": 25, "num": 123, "message": "howdy", }) outmsg = out.getvalue() # this contains a localtime string, so don't check the hour self.failUnless(":06.527 L25 []#123 howdy" in outmsg) lp.remote_msg({"time": 1207005907.527782, "level": 25, "num": 124, "format": "howdy %(there)s", "there": "pardner", }) outmsg = out.getvalue() # this contains a localtime string, so don't check the hour self.failUnless(":07.527 L25 []#124 howdy pardner" in outmsg) try: raise RuntimeError("fake error") except RuntimeError: f = failure.Failure() lp.remote_msg({"time": 1207005950.002, "level": 30, "num": 125, "message": "oops", "failure": f, }) outmsg = out.getvalue() self.failUnless(":50.002 L30 []#125 oops\n FAILURE:\n" in outmsg, outmsg) self.failUnless("exceptions.RuntimeError" in outmsg, outmsg) self.failUnless(": fake error" in outmsg, outmsg) self.failUnless("--- ---\n" in outmsg, outmsg) def test_logprinter_verbose(self): target_tubid_s = "jiijpvbge2e3c3botuzzz7la3utpl67v" options1 = {"save-to": None, "verbose": True, "timestamps": "short-local"} out = StringIO() lp = tail.LogPrinter(options1, target_tubid_s[:8], out) lp.got_versions({}) lp.remote_msg({"time": 1207005906.527782, "level": 25, "num": 123, "message": "howdy", }) outmsg = out.getvalue() self.failUnless("'message': 'howdy'" in outmsg, outmsg) self.failUnless("'time': 1207005906.527782" in outmsg, outmsg) self.failUnless("'level': 25" in outmsg, outmsg) self.failUnless("{" in outmsg, outmsg) def test_logprinter_saveto(self): target_tubid_s = "jiijpvbge2e3c3botuzzz7la3utpl67v" saveto_filename = "test_logprinter_saveto.flog" options = {"save-to": saveto_filename, "verbose": False, "timestamps": "short-local"} out = StringIO() lp = tail.LogPrinter(options, target_tubid_s[:8], out) lp.got_versions({}) lp.remote_msg({"time": 1207005906.527782, "level": 25, "num": 123, "message": "howdy", }) outmsg = out.getvalue() del outmsg lp.saver.disconnected() # cause the file to be closed f = open(saveto_filename, "rb") expected_magic = f.read(len(flogfile.MAGIC)) self.failUnlessEqual(expected_magic, flogfile.MAGIC) data = json.loads(f.readline()) # header self.failUnlessEqual(data["header"]["type"], "tail") data = json.loads(f.readline()) # event self.failUnlessEqual(data["from"], "jiijpvbg") self.failUnlessEqual(data["d"]["message"], "howdy") self.failUnlessEqual(data["d"]["num"], 123) def test_options(self): basedir = "logging/Tail/options" os.makedirs(basedir) fn = os.path.join(basedir, "foo") f = open(fn, "w") f.write("pretend this is a furl") f.close() f = open(os.path.join(basedir, "logport.furl"), "w") f.write("this too") f.close() to = tail.TailOptions() to.parseOptions(["pb:pretend-furl"]) self.failIf(to["verbose"]) self.failIf(to["catch-up"]) self.failUnlessEqual(to.target_furl, "pb:pretend-furl") to = tail.TailOptions() to.parseOptions(["--verbose", "--catch-up", basedir]) self.failUnless(to["verbose"]) self.failUnless(to["catch-up"]) self.failUnlessEqual(to.target_furl, "this too") to = tail.TailOptions() to.parseOptions(["--save-to", "save.flog", fn]) self.failIf(to["verbose"]) self.failIf(to["catch-up"]) self.failUnlessEqual(to["save-to"], "save.flog") self.failUnlessEqual(to.target_furl, "pretend this is a furl") to = tail.TailOptions() self.failUnlessRaises(RuntimeError, to.parseOptions, ["bogus.txt"]) # applications that provide a command-line tool may find it useful to include # a "flogtool" subcommand, using something like this: class WrapperOptions(usage.Options): synopsis = "Usage: wrapper flogtool " subCommands = [("flogtool", None, cli.Options, "foolscap log tool")] def run_wrapper(argv): config = WrapperOptions() config.parseOptions(argv) command = config.subCommand if command == "flogtool": return cli.run_flogtool(argv[1:], run_by_human=False) class CLI(unittest.TestCase): def test_create_gatherer(self): basedir = "logging/CLI/create_gatherer" argv = ["flogtool", "create-gatherer", "--port", "tcp:3117", "--location", "tcp:localhost:3117", "--quiet", basedir] cli.run_flogtool(argv[1:], run_by_human=False) self.failUnless(os.path.exists(basedir)) basedir = "logging/CLI/create_gatherer2" argv = ["flogtool", "create-gatherer", "--rotate", "3600", "--port", "tcp:3117", "--location", "tcp:localhost:3117", "--quiet", basedir] cli.run_flogtool(argv[1:], run_by_human=False) self.failUnless(os.path.exists(basedir)) basedir = "logging/CLI/create_gatherer3" argv = ["flogtool", "create-gatherer", "--port", "tcp:3117", "--location", "tcp:localhost:3117", basedir] (out,err) = cli.run_flogtool(argv[1:], run_by_human=False) self.failUnless(os.path.exists(basedir)) self.failUnless(("Gatherer created in directory %s" % basedir) in out, out) self.failUnless("Now run" in out, out) self.failUnless("to launch the daemon" in out, out) def test_create_gatherer_badly(self): #basedir = "logging/CLI/create_gatherer" argv = ["flogtool", "create-gatherer", "--bogus-arg"] self.failUnlessRaises(usage.UsageError, cli.run_flogtool, argv[1:], run_by_human=False) def test_create_gatherer_no_location(self): basedir = "logging/CLI/create_gatherer_no_location" argv = ["flogtool", "create-gatherer", basedir] e = self.failUnlessRaises(usage.UsageError, cli.run_flogtool, argv[1:], run_by_human=False) self.failUnlessIn("--location= is mandatory", str(e)) def test_wrapper(self): basedir = "logging/CLI/wrapper" argv = ["wrapper", "flogtool", "create-gatherer", "--port", "tcp:3117", "--location", "tcp:localhost:3117", "--quiet", basedir] run_wrapper(argv[1:]) self.failUnless(os.path.exists(basedir)) def test_create_incident_gatherer(self): basedir = "logging/CLI/create_incident_gatherer" argv = ["flogtool", "create-incident-gatherer", "--port", "tcp:3118", "--location", "tcp:localhost:3118", "--quiet", basedir] cli.run_flogtool(argv[1:], run_by_human=False) self.failUnless(os.path.exists(basedir)) basedir = "logging/CLI/create_incident_gatherer2" argv = ["flogtool", "create-incident-gatherer", "--port", "tcp:3118", "--location", "tcp:localhost:3118", basedir] (out,err) = cli.run_flogtool(argv[1:], run_by_human=False) self.failUnless(os.path.exists(basedir)) self.failUnless(("Incident Gatherer created in directory %s" % basedir) in out, out) self.failUnless("Now run" in out, out) self.failUnless("to launch the daemon" in out, out) class LogfileWriterMixin: def create_logfile(self): if not os.path.exists(self.basedir): os.makedirs(self.basedir) fn = os.path.join(self.basedir, "dump.flog") l = log.FoolscapLogger() lfo = log.LogFileObserver(fn, level=0) l.addObserver(lfo.msg) l.msg("one", facility="big.facility") time.sleep(0.2) # give filter --after something to work with l.msg("two", level=log.OPERATIONAL-1) try: raise SampleError("whoops1") except: l.err(message="three") l.msg("four") d = fireEventually() def _done(res): lfo._stop() #events = self._read_logfile(fn) #self.failUnlessEqual(len(events), 1+3) return fn d.addCallback(_done) return d def create_incident(self): if not os.path.exists(self.basedir): os.makedirs(self.basedir) l = log.FoolscapLogger() l.setLogDir(self.basedir) l.setIncidentReporterFactory(NoFollowUpReporter) d = defer.Deferred() def _done(name, trigger): d.callback( (name,trigger) ) l.addImmediateIncidentObserver(_done) l.msg("one") l.msg("two") l.msg("boom", level=log.WEIRD) l.msg("four") d.addCallback(lambda (name,trigger): os.path.join(self.basedir, name+".flog.bz2")) return d class Dumper(unittest.TestCase, LogfileWriterMixin, LogfileReaderMixin): # create a logfile, then dump it, and examine the output to make sure it # worked right. def test_dump(self): self.basedir = "logging/Dumper/dump" d = self.create_logfile() def _check(fn): events = self._read_logfile(fn) d = dumper.LogDumper() # initialize the LogDumper() timestamp mode d.options = dumper.DumpOptions() d.options.parseOptions([fn]) tmode = d.options["timestamps"] argv = ["flogtool", "dump", fn] (out,err) = cli.run_flogtool(argv[1:], run_by_human=False) self.failUnlessEqual(err, "") lines = list(StringIO(out).readlines()) self.failUnless(lines[0].strip().startswith("Application versions"), lines[0]) mypid = os.getpid() self.failUnlessEqual(lines[3].strip(), "PID: %s" % mypid, lines[3]) lines = lines[5:] line0 = "local#%d %s: one" % (events[1]["d"]["num"], format_time(events[1]["d"]["time"], tmode)) self.failUnlessEqual(lines[0].strip(), line0) self.failUnless("FAILURE:" in lines[3]) self.failUnlessIn("test_logging.SampleError", lines[4]) self.failUnlessIn(": whoops1", lines[4]) self.failUnless(lines[-1].startswith("local#3 ")) argv = ["flogtool", "dump", "--just-numbers", fn] (out,err) = cli.run_flogtool(argv[1:], run_by_human=False) self.failUnlessEqual(err, "") lines = list(StringIO(out).readlines()) line0 = "%s %d" % (format_time(events[1]["d"]["time"], tmode), events[1]["d"]["num"]) self.failUnlessEqual(lines[0].strip(), line0) self.failUnless(lines[1].strip().endswith(" 1")) self.failUnless(lines[-1].strip().endswith(" 3")) # failures are not dumped in --just-numbers self.failUnlessEqual(len(lines), 1+3) argv = ["flogtool", "dump", "--rx-time", fn] (out,err) = cli.run_flogtool(argv[1:], run_by_human=False) self.failUnlessEqual(err, "") lines = list(StringIO(out).readlines()) self.failUnless(lines[0].strip().startswith("Application versions"), lines[0]) mypid = os.getpid() self.failUnlessEqual(lines[3].strip(), "PID: %s" % mypid, lines[3]) lines = lines[5:] line0 = "local#%d rx(%s) emit(%s): one" % \ (events[1]["d"]["num"], format_time(events[1]["rx_time"], tmode), format_time(events[1]["d"]["time"], tmode)) self.failUnlessEqual(lines[0].strip(), line0) self.failUnless(lines[-1].strip().endswith(" four")) argv = ["flogtool", "dump", "--verbose", fn] (out,err) = cli.run_flogtool(argv[1:], run_by_human=False) self.failUnlessEqual(err, "") lines = list(StringIO(out).readlines()) self.failUnless("header" in lines[0]) self.failUnless(re.search(r"u?'message': u?'one'", lines[1]), lines[1]) self.failUnless("'level': 20" in lines[1]) self.failUnless(": four: {" in lines[-1]) d.addCallback(_check) return d def test_incident(self): self.basedir = "logging/Dumper/incident" d = self.create_incident() def _check(fn): events = self._read_logfile(fn) # for sanity, make sure we created the incident correctly assert events[0]["header"]["type"] == "incident" assert events[0]["header"]["trigger"]["num"] == 2 argv = ["flogtool", "dump", fn] (out,err) = cli.run_flogtool(argv[1:], run_by_human=False) self.failUnlessEqual(err, "") lines = list(StringIO(out).readlines()) self.failUnlessEqual(len(lines), 8) self.failUnlessEqual(lines[0].strip(), "Application versions (embedded in logfile):") self.failUnless(lines[1].strip().startswith("foolscap:"), lines[1]) self.failUnless(lines[2].strip().startswith("twisted:"), lines[2]) mypid = os.getpid() self.failUnlessEqual(lines[3].strip(), "PID: %s" % mypid, lines[3]) self.failUnlessEqual(lines[4].strip(), "") self.failIf("[INCIDENT-TRIGGER]" in lines[5]) self.failIf("[INCIDENT-TRIGGER]" in lines[6]) self.failUnless(lines[7].strip().endswith(": boom [INCIDENT-TRIGGER]")) d.addCallback(_check) return d def test_oops_furl(self): self.basedir = os.path.join("logging", "Dumper", "oops_furl") if not os.path.exists(self.basedir): os.makedirs(self.basedir) fn = os.path.join(self.basedir, "logport.furl") f = open(fn, "w") f.write("pb://TUBID@HINTS/SWISSNUM\n") f.close() d = dumper.LogDumper() # initialize the LogDumper() timestamp mode d.options = dumper.DumpOptions() d.options.parseOptions([fn]) argv = ["flogtool", "dump", fn] (out,err) = cli.run_flogtool(argv[1:], run_by_human=False) self.failUnlessEqual(err, "Error: %s appears to be a FURL file.\nPerhaps you meant to run 'flogtool tail' instead of 'flogtool dump'?\n" % fn) PICKLE_DUMPFILE_B64 = """ KGRwMApTJ2hlYWRlcicKcDEKKGRwMgpTJ3RocmVzaG9sZCcKcDMKSTAKc1MncGlkJwpwNA pJMTg3MjgKc1MndHlwZScKcDUKUydsb2ctZmlsZS1vYnNlcnZlcicKcDYKc1MndmVyc2lv bnMnCnA3CihkcDgKUydmb29sc2NhcCcKcDkKUycwLjkuMSsyMi5nNzhlNWEzZC5kaXJ0eS cKcDEwCnNTJ3R3aXN0ZWQnCnAxMQpTJzE1LjUuMCcKcDEyCnNzcy6AAn1xAChVBGZyb21x AVUFbG9jYWxxAlUHcnhfdGltZXEDR0HVmqGrUXpjVQFkcQR9cQUoVQVsZXZlbHEGSxRVC2 luY2FybmF0aW9ucQdVCMZQLsaodzvDcQhOhnEJVQhmYWNpbGl0eXEKVQxiaWcuZmFjaWxp dHlxC1UDbnVtcQxLAFUEdGltZXENR0HVmqGrRFtbVQdtZXNzYWdlcQ5VA29uZXEPdXUugA J9cQAoVQRmcm9tcQFVBWxvY2FscQJVB3J4X3RpbWVxA0dB1Zqhq1F+s1UBZHEEfXEFKFUH bWVzc2FnZXEGVQN0d29xB1UDbnVtcQhLAVUEdGltZXEJR0HVmqGrUU6cVQtpbmNhcm5hdG lvbnEKVQjGUC7GqHc7w3ELToZxDFUFbGV2ZWxxDUsTdXUugAJ9cQAoVQRmcm9tcQFVBWxv Y2FscQJVB3J4X3RpbWVxA0dB1Zqhq1GAiFUBZHEEfXEFKFUFbGV2ZWxxBksUVQtpbmNhcm 5hdGlvbnEHVQjGUC7GqHc7w3EIToZxCVUDd2h5cQpOVQdmYWlsdXJlcQsoY2Zvb2xzY2Fw LmNhbGwKQ29waWVkRmFpbHVyZQpxDG9xDX1xDyhVAnRicRBOVQl0cmFjZWJhY2txEVSBAw AAVHJhY2ViYWNrIChtb3N0IHJlY2VudCBjYWxsIGxhc3QpOgogIEZpbGUgIi9Vc2Vycy93 YXJuZXIvc3R1ZmYvcHl0aG9uL2Zvb2xzY2FwL3ZlL2xpYi9weXRob24yLjcvc2l0ZS1wYW NrYWdlcy90d2lzdGVkL3RyaWFsL19hc3luY3Rlc3QucHkiLCBsaW5lIDExMiwgaW4gX3J1 bgogICAgdXRpbHMucnVuV2l0aFdhcm5pbmdzU3VwcHJlc3NlZCwgc2VsZi5fZ2V0U3VwcH Jlc3MoKSwgbWV0aG9kKQogIEZpbGUgIi9Vc2Vycy93YXJuZXIvc3R1ZmYvcHl0aG9uL2Zv b2xzY2FwL3ZlL2xpYi9weXRob24yLjcvc2l0ZS1wYWNrYWdlcy90d2lzdGVkL2ludGVybm V0L2RlZmVyLnB5IiwgbGluZSAxNTAsIGluIG1heWJlRGVmZXJyZWQKICAgIHJlc3VsdCA9 IGYoKmFyZ3MsICoqa3cpCiAgRmlsZSAiL1VzZXJzL3dhcm5lci9zdHVmZi9weXRob24vZm 9vbHNjYXAvdmUvbGliL3B5dGhvbjIuNy9zaXRlLXBhY2thZ2VzL3R3aXN0ZWQvaW50ZXJu ZXQvdXRpbHMucHkiLCBsaW5lIDE5NywgaW4gcnVuV2l0aFdhcm5pbmdzU3VwcHJlc3NlZA ogICAgcmVzdWx0ID0gZigqYSwgKiprdykKICBGaWxlICIvVXNlcnMvd2FybmVyL3N0dWZm L3B5dGhvbi9mb29sc2NhcC9mb29sc2NhcC90ZXN0L3Rlc3RfbG9nZ2luZy5weSIsIGxpbm UgMTg4MywgaW4gdGVzdF9kdW1wCiAgICBkID0gc2VsZi5jcmVhdGVfbG9nZmlsZSgpCi0t LSA8ZXhjZXB0aW9uIGNhdWdodCBoZXJlPiAtLS0KICBGaWxlICIvVXNlcnMvd2FybmVyL3 N0dWZmL3B5dGhvbi9mb29sc2NhcC9mb29sc2NhcC90ZXN0L3Rlc3RfbG9nZ2luZy5weSIs IGxpbmUgMTg0MiwgaW4gY3JlYXRlX2xvZ2ZpbGUKICAgIHJhaXNlIFNhbXBsZUVycm9yKC J3aG9vcHMxIikKZm9vbHNjYXAudGVzdC50ZXN0X2xvZ2dpbmcuU2FtcGxlRXJyb3I6IHdo b29wczEKcRJVBXZhbHVlcRNVB3dob29wczFxFFUHcGFyZW50c3EVXXEWKFUmZm9vbHNjYX AudGVzdC50ZXN0X2xvZ2dpbmcuU2FtcGxlRXJyb3JxF1UUZXhjZXB0aW9ucy5FeGNlcHRp b25xGFUYZXhjZXB0aW9ucy5CYXNlRXhjZXB0aW9ucRlVEl9fYnVpbHRpbl9fLm9iamVjdH EaZVUGZnJhbWVzcRtdcRxVBHR5cGVxHVUmZm9vbHNjYXAudGVzdC50ZXN0X2xvZ2dpbmcu U2FtcGxlRXJyb3JxHlUFc3RhY2txH11xIHViVQNudW1xIUsCVQR0aW1lcSJHQdWaoatRVt JVB21lc3NhZ2VxI1UFdGhyZWVxJFUHaXNFcnJvcnElSwF1dS6AAn1xAChVBGZyb21xAVUF bG9jYWxxAlUHcnhfdGltZXEDR0HVmqGrUYXkVQFkcQR9cQUoVQdtZXNzYWdlcQZVBGZvdX JxB1UDbnVtcQhLA1UEdGltZXEJR0HVmqGrUXU2VQtpbmNhcm5hdGlvbnEKVQjGUC7GqHc7 w3ELToZxDFUFbGV2ZWxxDUsUdXUu """ PICKLE_INCIDENT_B64 = """ QlpoOTFBWSZTWUOW3hEAAHjfgAAQAcl/4QkhCAS/59/iQAGdWS2BJRTNNQ2oB6gGgPU9T1 BoEkintKGIABiAAaAwANGhowjJoNGmgMCpJDSaNGqbSMnqGhoaAP1S5rw5GxrNlUoxLXu2 sZ5TYy2rVCVNHMKgeDE97TBiw1hXtCfdSCISDpSlL61KFiacqWj9apY80J2PIpO7mde+vd Jz18Myu4+djYU10JPMGU5vFAcUmmyk0kmcGUSMIDUJcKkog4W2EyyQStwwSYUEohGpr6Wm F4KU7qccsjPJf8dTIv3ydZM5hpkW41JjJ8j0PETxlRRVFSeZYsqFU+hufU3n5O3hmYASDC DhWMHFPJE7nXCYRsz5BGjktwUQCu6d4cixrgmGYLYA7JVCM7UqkMDVD9EMaclrFuayYGBR xMIwXxM9pjeUuZVv2ceR5E6FSWpVRKKD98ObK5wmGmU9vqNBKqjp0wwqZlZ3x3nA4n+LTS rmhbVjNyWeh/xdyRThQkEOW3hE """ class OldPickleDumper(unittest.TestCase): def test_dump(self): self.basedir = "logging/OldPickleDumper/dump" if not os.path.exists(self.basedir): os.makedirs(self.basedir) fn = os.path.join(self.basedir, "dump.flog") with open(fn, "wb") as f: f.write(base64.b64decode(PICKLE_DUMPFILE_B64)) argv = ["flogtool", "dump", fn] (out,err) = cli.run_flogtool(argv[1:], run_by_human=False) self.failUnlessEqual(out, "") self.failUnlessIn("which cannot be loaded safely", err) def test_incident(self): self.basedir = "logging/OldPickleDumper/incident" if not os.path.exists(self.basedir): os.makedirs(self.basedir) fn = os.path.join(self.basedir, "incident-2015-12-11--08-18-28Z-uqyuiea.flog.bz2") with open(fn, "wb") as f: f.write(base64.b64decode(PICKLE_INCIDENT_B64)) argv = ["flogtool", "dump", fn] (out,err) = cli.run_flogtool(argv[1:], run_by_human=False) self.failUnlessEqual(out, "") self.failUnlessIn("which cannot be loaded safely", err) class Filter(unittest.TestCase, LogfileWriterMixin, LogfileReaderMixin): def compare_events(self, a, b): ## # cmp(a,b) won't quite work, because two instances of CopiedFailure ## # loaded from the same pickle don't compare as equal # in fact we no longer create CopiedFailure instances in logs, so a # simple failUnlessEqual will now suffice self.failUnlessEqual(a, b) def test_basic(self): self.basedir = "logging/Filter/basic" d = self.create_logfile() def _check(fn): events = self._read_logfile(fn) count = len(events) assert count == 5 dirname,filename = os.path.split(fn) fn2 = os.path.join(dirname, "filtered-" + filename) # pass-through argv = ["flogtool", "filter", fn, fn2] (out,err) = cli.run_flogtool(argv[1:], run_by_human=False) self.failUnless("copied 5 of 5 events into new file" in out, out) self.compare_events(events, self._read_logfile(fn2)) # convert to .bz2 while we're at it fn2bz2 = fn2 + ".bz2" argv = ["flogtool", "filter", fn, fn2bz2] (out,err) = cli.run_flogtool(argv[1:], run_by_human=False) self.failUnless("copied 5 of 5 events into new file" in out, out) self.compare_events(events, self._read_logfile(fn2bz2)) # modify the file in place argv = ["flogtool", "filter", "--above", "20", fn2] (out,err) = cli.run_flogtool(argv[1:], run_by_human=False) self.failUnless("modifying event file in place" in out, out) self.failUnless("--above: removing events below level 20" in out, out) self.failUnless("copied 4 of 5 events into new file" in out, out) self.compare_events([events[0], events[1], events[3], events[4]], self._read_logfile(fn2)) # modify the file in place, two-argument version argv = ["flogtool", "filter", fn2, fn2] (out,err) = cli.run_flogtool(argv[1:], run_by_human=False) self.failUnless("modifying event file in place" in out, out) self.failUnless("copied 4 of 4 events into new file" in out, out) self.compare_events([events[0], events[1], events[3], events[4]], self._read_logfile(fn2)) # --above with a string argument argv = ["flogtool", "filter", "--above", "OPERATIONAL", fn, fn2] (out,err) = cli.run_flogtool(argv[1:], run_by_human=False) self.failUnless("--above: removing events below level 20" in out, out) self.failUnless("copied 4 of 5 events into new file" in out, out) self.compare_events([events[0], events[1], events[3], events[4]], self._read_logfile(fn2)) t_one = events[1]["d"]["time"] # we can only pass integers into --before and --after, so we'll # just test that we get all or nothing argv = ["flogtool", "filter", "--before", str(int(t_one - 10)), fn, fn2] (out,err) = cli.run_flogtool(argv[1:], run_by_human=False) self.failUnless("copied 1 of 5 events into new file" in out, out) # we always get the header, so it's 1 instead of 0 self.compare_events(events[:1], self._read_logfile(fn2)) argv = ["flogtool", "filter", "--after", str(int(t_one + 10)), fn, fn2] (out,err) = cli.run_flogtool(argv[1:], run_by_human=False) self.failUnless("copied 1 of 5 events into new file" in out, out) self.compare_events(events[:1], self._read_logfile(fn2)) # --facility argv = ["flogtool", "filter", "--strip-facility", "big", fn, fn2] (out,err) = cli.run_flogtool(argv[1:], run_by_human=False) self.failUnless("--strip-facility: removing events for big and children" in out, out) self.failUnless("copied 4 of 5 events into new file" in out, out) self.compare_events([events[0],events[2],events[3],events[4]], self._read_logfile(fn2)) # pass-through, --verbose, read from .bz2 argv = ["flogtool", "filter", "--verbose", fn2bz2, fn2] (out,err) = cli.run_flogtool(argv[1:], run_by_human=False) self.failUnless("copied 5 of 5 events into new file" in out, out) lines = [l.strip() for l in StringIO(out).readlines()] self.failUnlessEqual(lines, ["HEADER", "0", "1", "2", "3", "copied 5 of 5 events into new file"]) self.compare_events(events, self._read_logfile(fn2)) # --from . This normally takes a base32 tubid prefix, but the # things we've logged all say ["from"]="local". So just test # all-or-nothing. argv = ["flogtool", "filter", "--from", "local", fn, fn2] (out,err) = cli.run_flogtool(argv[1:], run_by_human=False) self.failUnless("--from: retaining events only from tubid prefix local" in out, out) self.failUnless("copied 5 of 5 events into new file" in out, out) self.compare_events(events, self._read_logfile(fn2)) argv = ["flogtool", "filter", "--from", "NOTlocal", fn, fn2] (out,err) = cli.run_flogtool(argv[1:], run_by_human=False) self.failUnless("--from: retaining events only from tubid prefix NOTlocal" in out, out) self.failUnless("copied 1 of 5 events into new file" in out, out) self.compare_events(events[:1], self._read_logfile(fn2)) d.addCallback(_check) return d @inlineCallbacks def getPage(url): a = client.Agent(reactor) response = yield a.request("GET", url) import warnings with warnings.catch_warnings(): warnings.simplefilter("ignore") # Twisted can emit a spurious internal warning here ("Using readBody # with a transport that does not have an abortConnection method") # which seems to be https://twistedmatrix.com/trac/ticket/8227 page = yield client.readBody(response) if response.code != 200: raise ValueError("request failed (%d), page contents were: %s" % ( response.code, page)) returnValue(page) class Web(unittest.TestCase): def setUp(self): self.viewer = None def tearDown(self): d = defer.maybeDeferred(unittest.TestCase.tearDown, self) if self.viewer: d.addCallback(lambda res: self.viewer.stop()) return d @inlineCallbacks def test_basic(self): basedir = "logging/Web/basic" os.makedirs(basedir) l = log.FoolscapLogger() fn = os.path.join(basedir, "flog.out") ob = log.LogFileObserver(fn) l.addObserver(ob.msg) l.msg("one") lp = l.msg("two") l.msg("three", parent=lp, failure=failure.Failure(RuntimeError("yo"))) l.msg("four", level=log.UNUSUAL) yield fireEventually() l.removeObserver(ob.msg) ob._stop() portnum = allocate_tcp_port() argv = ["-p", "tcp:%d:interface=127.0.0.1" % portnum, "--quiet", fn] options = web.WebViewerOptions() options.parseOptions(argv) self.viewer = web.WebViewer() self.url = yield self.viewer.start(options) self.baseurl = self.url[:self.url.rfind("/")] + "/" page = yield getPage(self.url) mypid = os.getpid() self.failUnless("PID %s" % mypid in page, "didn't see 'PID %s' in '%s'" % (mypid, page)) self.failUnless("Application Versions:" in page, page) self.failUnless("foolscap: %s" % foolscap.__version__ in page, page) self.failUnless("4 events covering" in page) self.failUnless('href="summary/0-20">3 events at level 20' in page) page = yield getPage(self.baseurl + "summary/0-20") self.failUnless("Events at level 20" in page) self.failUnless(": two" in page) self.failIf("four" in page) def check_all_events(page): self.failUnless("3 root events" in page) self.failUnless(": one" in page) self.failUnless(": two" in page) self.failUnless(": three FAILURE:" in page) self.failUnless(": UNUSUAL four" in page) page = yield getPage(self.baseurl + "all-events") check_all_events(page) page = yield getPage(self.baseurl + "all-events?sort=number") check_all_events(page) page = yield getPage(self.baseurl + "all-events?sort=time") check_all_events(page) page = yield getPage(self.baseurl + "all-events?sort=nested") check_all_events(page) page = yield getPage(self.baseurl + "all-events?timestamps=short-local") check_all_events(page) page = yield getPage(self.baseurl + "all-events?timestamps=utc") check_all_events(page) class Bridge(unittest.TestCase): def test_foolscap_to_twisted(self): fl = log.FoolscapLogger() tw = twisted_log.LogPublisher() log.bridgeLogsToTwisted(None, fl, tw) tw_out = [] tw.addObserver(tw_out.append) fl_out = [] fl.addObserver(fl_out.append) fl.msg("one") fl.msg(format="two %(two)d", two=2) fl.msg("three", level=log.NOISY) # should be removed d = flushEventualQueue() def _check(res): self.failUnlessEqual(len(fl_out), 3) self.failUnlessEqual(fl_out[0]["message"], "one") self.failUnlessEqual(fl_out[1]["format"], "two %(two)d") self.failUnlessEqual(fl_out[2]["message"], "three") self.failUnlessEqual(len(tw_out), 2) self.failUnlessEqual(tw_out[0]["message"], ("one",)) self.failUnless(tw_out[0]["from-foolscap"]) self.failUnlessEqual(tw_out[1]["message"], ("two 2",)) self.failUnless(tw_out[1]["from-foolscap"]) d.addCallback(_check) return d def test_twisted_to_foolscap(self): fl = log.FoolscapLogger() tw = twisted_log.LogPublisher() log.bridgeLogsFromTwisted(None, tw, fl) tw_out = [] tw.addObserver(tw_out.append) fl_out = [] fl.addObserver(fl_out.append) tw.msg("one") tw.msg(format="two %(two)d", two=2) # twisted now has places (e.g. Factory.doStart) where the new # Logger.info() is called with arbitrary (unserializable) kwargs for # string formatting, which are passed into the old LogPublisher(), # from which they arrive in foolscap. Make sure we can tolerate that. # The rule is that foolscap immediately stringifies all events it # gets from twisted (with log.textFromEventDict), and doesn't store # the additional arguments. So it's ok to put an *unserializable* # argument into the log.msg() call, as long as it's still # *stringifyable*. unserializable = lambda: "unserializable" tw.msg(format="three is %(evil)s", evil=unserializable) d = flushEventualQueue() def _check(res): self.failUnlessEqual(len(tw_out), 3) self.failUnlessEqual(tw_out[0]["message"], ("one",)) self.failUnlessEqual(tw_out[1]["format"], "two %(two)d") self.failUnlessEqual(tw_out[1]["two"], 2) self.failUnlessEqual(tw_out[2]["format"], "three is %(evil)s") self.failUnlessEqual(tw_out[2]["evil"], unserializable) self.failUnlessEqual(len(fl_out), 3) self.failUnlessEqual(fl_out[0]["message"], "one") self.failUnless(fl_out[0]["from-twisted"]) self.failUnlessEqual(fl_out[1]["message"], "two 2") self.failUnless(fl_out[1]["from-twisted"]) # str(unserializable) is like " at 0xblahblah>" self.failUnlessEqual(fl_out[2]["message"], "three is " + str(unserializable)) self.failUnless(fl_out[2]["from-twisted"]) d.addCallback(_check) return d def test_twisted_logger_to_foolscap(self): if not twisted_logger: raise unittest.SkipTest("needs twisted.logger from Twisted>=15.2.0") new_pub = twisted_logger.LogPublisher() old_pub = twisted_log.LogPublisher(observerPublisher=new_pub, publishPublisher=new_pub) fl = log.FoolscapLogger() log.bridgeLogsFromTwisted(None, old_pub, fl) tw_out = [] old_pub.addObserver(tw_out.append) fl_out = [] fl.addObserver(fl_out.append) tl = twisted_logger.Logger(observer=new_pub) tl.info("one") # note: new twisted logger wants PEP3101 format strings, {} not % tl.info(format="two {two}", two=2) # twisted's new Logger.info() takes arbitrary (unserializable) kwargs # for string formatting, and passes them into the old LogPublisher(), # so make sure we can tolerate that. The rule is that foolscap # stringifies all events it gets from twisted, and doesn't store the # additional arguments. unserializable = lambda: "unserializable" tl.info("three is {evil!s}", evil=unserializable) d = flushEventualQueue() def _check(res): self.failUnlessEqual(len(fl_out), 3) self.failUnlessEqual(fl_out[0]["message"], "one") self.failUnless(fl_out[0]["from-twisted"]) self.failUnlessEqual(fl_out[1]["message"], "two 2") self.failIf("two" in fl_out[1]) self.failUnless(fl_out[1]["from-twisted"]) # str(unserializable) is like " at 0xblahblah>" self.failUnlessEqual(fl_out[2]["message"], "three is " + str(unserializable)) self.failUnless(fl_out[2]["from-twisted"]) d.addCallback(_check) return d def test_no_loops(self): fl = log.FoolscapLogger() tw = twisted_log.LogPublisher() log.bridgeLogsFromTwisted(None, tw, fl) log.bridgeLogsToTwisted(None, fl, tw) tw_out = [] tw.addObserver(tw_out.append) fl_out = [] fl.addObserver(fl_out.append) tw.msg("one") fl.msg("two") d = flushEventualQueue() def _check(res): self.failUnlessEqual(len(tw_out), 2) self.failUnlessEqual(tw_out[0]["message"], ("one",)) self.failUnlessEqual(tw_out[1]["message"], ("two",)) self.failUnlessEqual(len(fl_out), 2) self.failUnlessEqual(fl_out[0]["message"], "one") self.failUnlessEqual(fl_out[1]["message"], "two") d.addCallback(_check) return d foolscap-0.13.1/src/foolscap/test/test_loopback.py0000644000076500000240000000217412766553111022615 0ustar warnerstaff00000000000000 from twisted.trial import unittest from twisted.internet import defer from foolscap.test.common import HelperTarget, MakeTubsMixin from foolscap.eventual import flushEventualQueue class ConnectToSelf(MakeTubsMixin, unittest.TestCase): def setUp(self): self.makeTubs(1) def tearDown(self): d = defer.DeferredList([s.stopService() for s in self.services]) d.addCallback(flushEventualQueue) return d def testConnectAuthenticated(self): tub = self.services[0] target = HelperTarget("bob") target.obj = "unset" url = tub.registerReference(target) # can we connect to a reference on our own Tub? d = tub.getReference(url) def _connected(ref): return ref.callRemote("set", 12) d.addCallback(_connected) def _check(res): self.failUnlessEqual(target.obj, 12) d.addCallback(_check) def _connect_again(res): target.obj = None return tub.getReference(url) d.addCallback(_connect_again) d.addCallback(_connected) d.addCallback(_check) return d foolscap-0.13.1/src/foolscap/test/test_negotiate.py0000644000076500000240000011633713204160675023006 0ustar warnerstaff00000000000000 from twisted.trial import unittest from twisted.internet import protocol, defer, reactor from twisted.internet.defer import inlineCallbacks, returnValue from twisted.application import internet from twisted.web.client import Agent from foolscap import negotiate, tokens from foolscap.api import Referenceable, Tub, BananaError from foolscap.util import allocate_tcp_port from foolscap.test.common import (BaseMixin, PollMixin, tubid_low, certData_low, certData_high) class Target(Referenceable): def __init__(self): self.calls = 0 def remote_call(self): self.calls += 1 class OneTimeDeferred(defer.Deferred): def callback(self, res): if self.called: return return defer.Deferred.callback(self, res) class Basic(BaseMixin, unittest.TestCase): def testOptions(self): url, portnum = self.makeServer({'opt': 12}) self.failUnlessEqual(self.tub._test_options['opt'], 12) def testAuthenticated(self): url, portnum = self.makeServer() client = Tub() client.startService() self.services.append(client) d = client.getReference(url) return d testAuthenticated.timeout = 10 class Versus(BaseMixin, unittest.TestCase): def testVersusHTTPServerAuthenticated(self): portnum = self.makeHTTPServer() client = Tub() client.startService() self.services.append(client) url = "pb://%s@127.0.0.1:%d/target" % (tubid_low, portnum) d = client.getReference(url) d.addCallbacks(lambda res: self.fail("this is supposed to fail"), lambda f: f.trap(BananaError)) # the HTTP server needs a moment to notice that the connection has # gone away. Without this, trial flunks the test because of the # leftover HTTP server socket. d.addCallback(self.stall, 1) return d testVersusHTTPServerAuthenticated.timeout = 10 @inlineCallbacks def testVersusHTTPClientAuthenticated(self): url, portnum = self.makeServer() a = Agent(reactor) response = yield a.request("GET", "http://127.0.0.1:%d/foo" % portnum) self.assertEqual(response.code, 500) testVersusHTTPClientAuthenticated.timeout = 10 def testNoConnection(self): url, portnum = self.makeServer() d = self.tub.stopService() d.addCallback(self._testNoConnection_1, url) return d testNoConnection.timeout = 10 def _testNoConnection_1(self, res, url): self.services.remove(self.tub) client = Tub() client.startService() self.services.append(client) d = client.getReference(url) d.addCallbacks(lambda res: self.fail("this is supposed to fail"), self._testNoConnection_fail) return d def _testNoConnection_fail(self, why): from twisted.internet import error self.failUnless(why.check(error.ConnectionRefusedError)) def testClientTimeout(self): portnum = self.makeNullServer() # lower the connection timeout to 2 seconds client = Tub(_test_options={'connect_timeout': 1}) client.startService() self.services.append(client) url = "pb://faketubid@127.0.0.1:%d/target" % portnum d = client.getReference(url) d.addCallbacks(lambda res: self.fail("hey! this is supposed to fail"), lambda f: f.trap(tokens.NegotiationError)) return d testClientTimeout.timeout = 10 def testServerTimeout(self): # lower the connection timeout to 1 seconds # the debug callback gets fired each time Negotiate.negotiationFailed # is fired, which happens twice (once for the timeout, once for the # resulting connectionLost), so we have to make sure the Deferred is # only fired once. d = OneTimeDeferred() options = {'server_timeout': 1, 'debug_negotiationFailed_cb': d.callback } url, portnum = self.makeServer(listenerOptions=options) f = protocol.ClientFactory() f.protocol = protocol.Protocol # discards everything s = internet.TCPClient("127.0.0.1", portnum, f) s.startService() self.services.append(s) d.addCallbacks(lambda res: self.fail("hey! this is supposed to fail"), lambda f: self._testServerTimeout_1) return d testServerTimeout.timeout = 10 def _testServerTimeout_1(self, f): self.failUnless(f.check(tokens.NegotiationError)) self.failUnlessEqual(f.value.args[0], "negotiation timeout") class Parallel(BaseMixin, unittest.TestCase): # testParallel*: listen on two separate ports, set up a URL with both # ports in the locationHints field, the connect. PB is supposed to # connect to both ports at the same time, using whichever one completes # negotiation first. The other connection is supposed to be dropped # silently. # the cases we need to cover are enumerated by the possible states that # connection[1] can be in when connection[0] (the winning connection) # completes negotiation. Those states are: # 1: connectTCP initiated and failed # 2: connectTCP initiated, but not yet established # 3: connection established, but still in the PLAINTEXT phase # (sent GET, waiting for the 101 Switching Protocols) # 4: still in ENCRYPTED phase: sent Hello, waiting for their Hello # 5: in DECIDING phase (non-master), waiting for their decision # def makeServers(self, tubopts={}, lo1={}, lo2={}): self.tub = tub = Tub(certData=certData_high, _test_options=tubopts) tub.startService() self.services.append(tub) self.p1, self.p2 = allocate_tcp_port(), allocate_tcp_port() tub.listenOn("tcp:%d:interface=127.0.0.1" % self.p1, lo1) tub.listenOn("tcp:%d:interface=127.0.0.1" % self.p2, lo2) tub.setLocation("127.0.0.1:%d" % self.p1, "127.0.0.1:%d" % self.p2) self.target = Target() return tub.registerReference(self.target) def connect(self, url): self.clientPhases = [] opts = {"debug_stall_second_connection": True, "debug_gatherPhases": self.clientPhases} self.client = client = Tub(certData_low, _test_options=opts) client.startService() self.services.append(client) d = client.getReference(url) return d def checkConnectedToFirstListener(self, rr, targetPhases): # verify that we connected to the first listener, and not the second self.failUnlessEqual(rr.tracker.broker.transport.getPeer().port, self.p1) # then pause a moment for the other connection to finish giving up d = self.stall(rr, 0.5) # and verify that we finished during the phase that we meant to test d.addCallback(lambda res: self.failUnlessEqual(self.clientPhases, targetPhases, "negotiation was abandoned in " "the wrong phase")) return d def test1(self): # in this test, we stop listening on the second port, so the second # connection will terminate with an ECONNREFUSED before the first one # completes. We also slow down the first port so we're sure to # recognize the failed second connection before starting negotiation # on the first. url = self.makeServers(lo1={'debug_slow_connectionMade': True}) d = self.tub.stopListeningOn(self.tub.getListeners()[1]) d.addCallback(self._test1_1, url) return d def _test1_1(self, res, url): d = self.connect(url) d.addCallback(self.checkConnectedToFirstListener, []) #d.addCallback(self.stall, 1) return d test1.timeout = 10 def test2(self): # slow down the second listener so that the first one is used. The # second listener will be connected but it will not respond to # negotiation for a moment, allowing the first connection to # complete. url = self.makeServers(lo2={'debug_slow_connectionMade': True}) d = self.connect(url) d.addCallback(self.checkConnectedToFirstListener, [negotiate.PLAINTEXT]) #d.addCallback(self.stall, 1) return d test2.timeout = 10 def test3(self): # have the second listener stall just before it does # sendPlaintextServer(). This insures the second connection will be # waiting in the PLAINTEXT phase when the first connection completes. url = self.makeServers(lo2={'debug_slow_sendPlaintextServer': True}) d = self.connect(url) d.addCallback(self.checkConnectedToFirstListener, [negotiate.PLAINTEXT]) return d test3.timeout = 10 def test4(self): # stall the second listener just before it sends the Hello. # This insures the second connection will be waiting in the ENCRYPTED # phase when the first connection completes. url = self.makeServers(lo2={'debug_slow_sendHello': True}) d = self.connect(url) d.addCallback(self.checkConnectedToFirstListener, [negotiate.ENCRYPTED]) #d.addCallback(self.stall, 1) return d test4.timeout = 10 def test5(self): # stall the second listener just before it sends the decision. This # insures the second connection will be waiting in the DECIDING phase # when the first connection completes. # note: this requires that the listener winds up as the master. We # force this by ensuring that the server uses a stable certificate # with a pre-calculated tubid sort order. url = self.makeServers(lo2={'debug_slow_sendDecision': True}) d = self.connect(url) d.addCallback(self.checkConnectedToFirstListener, [negotiate.DECIDING]) return d test5.timeout = 10 class ThreeInParallel(BaseMixin, unittest.TestCase): # Reentrancy bugs can appear when multiple connections are abandoned at # the same time. Create three hints, of which one wins, to exercise two # being abandoned together. def makeServers(self, tubopts={}): self.tub = tub = Tub(certData=certData_high, _test_options=tubopts) tub.startService() self.services.append(tub) port = allocate_tcp_port() tub.listenOn("tcp:%d:interface=127.0.0.1" % port) tub.setLocation("127.0.0.1:%d" % port, "127.0.0.2:%d" % port, "127.0.0.3:%d" % port) self.target = Target() return tub.registerReference(self.target) def connect(self, url): self.client = client = Tub(certData_low) client.startService() self.services.append(client) d = client.getReference(url) return d def test1(self): url = self.makeServers() d = self.connect(url) # We stall to notice any errors that might happen when the losing # connections get cancelled. It doesn't matter so much for # TCP4ClientEndpoint, but HostnameEndpoint throws them (see # twisted#8014), so this will improve testing when we switch. d.addCallback(self.stall, 0.5) return d test1.timeout = 10 class SharedConnections(BaseMixin, unittest.TestCase): def makeServers(self): self.tub = tub = Tub(certData=certData_high) tub.startService() self.services.append(tub) port = allocate_tcp_port() tub.listenOn("tcp:%d:interface=127.0.0.1" % port) tub.setLocation("127.0.0.1:%d" % port) furl1 = tub.registerReference(Target()) furl2 = tub.registerReference(Target()) return furl1, furl2 def test1(self): # Two FURLs pointing at the same Tub should share a connection. This # basically exercises TubRef.__cmp__ . furl1, furl2 = self.makeServers() client = Tub(certData_low) client.startService() self.services.append(client) d = client.getReference(furl1) rrefs = [] d.addCallback(rrefs.append) d.addCallback(lambda _: client.getReference(furl2)) d.addCallback(rrefs.append) def _check(_): self.failUnlessIdentical(rrefs[0].tracker.broker, rrefs[1].tracker.broker) d.addCallback(_check) return d class CrossfireMixin(BaseMixin, PollMixin): # testSimultaneous*: similar to Parallel, but connection[0] is initiated # in the opposite direction. This is the case when two Tubs initiate # connections to each other at the same time. tub1IsMaster = False def makeServers(self, t1opts={}, t2opts={}, lo1={}, lo2={}): # first we create two Tubs a = Tub(_test_options=t1opts) b = Tub(_test_options=t1opts) # then we figure out which one will be the master, and call it tub1 if a.tubID > b.tubID: # a is the master tub1,tub2 = a,b else: tub1,tub2 = b,a if not self.tub1IsMaster: tub1,tub2 = tub2,tub1 self.tub1 = tub1 self.tub2 = tub2 # now fix up the options and everything else self.tub1phases = [] t1opts['debug_gatherPhases'] = self.tub1phases tub1._test_options = t1opts self.tub2phases = [] t2opts['debug_gatherPhases'] = self.tub2phases tub2._test_options = t2opts # connection[0], the winning connection, will be from tub1 to tub2 tub1.startService() self.services.append(tub1) self.portnum1 = allocate_tcp_port() tub1.listenOn("tcp:%d:interface=127.0.0.1" % self.portnum1, lo1) tub1.setLocation("127.0.0.1:%d" % self.portnum1) self.target1 = Target() self.url1 = tub1.registerReference(self.target1) # connection[1], the abandoned connection, will be from tub2 to tub1 tub2.startService() self.services.append(tub2) self.portnum2 = allocate_tcp_port() tub2.listenOn("tcp:%d:interface=127.0.0.1" % self.portnum2, lo2) tub2.setLocation("127.0.0.1:%d" % self.portnum2) self.target2 = Target() self.url2 = tub2.registerReference(self.target2) def connect(self): # initiate connection[1] from tub2 to tub1, which will stall (but the # actual getReference will eventually succeed once the # reverse-direction connection is established) d1 = self.tub2.getReference(self.url1) # give it a moment to get to the point where it stalls d = self.stall(None, 0.1) d.addCallback(self._connect, d1) return d, d1 def _connect(self, res, d1): # now initiate connection[0], from tub1 to tub2 d2 = self.tub1.getReference(self.url2) return d2 @inlineCallbacks def connect2(self): # start both connections at the same time d1 = self.tub1.getReference(self.url2) # tub2->tub1 will pause because listener1 is blocked somehow d2 = self.tub2.getReference(self.url1) # so the tub1->tub2 connection will win, firing both rrefs rref1 = yield d1 rref2 = yield d2 del rref2 # We never unblock listener1, but we need to wait for the tub2->tub1 # connection to be dropped, which will happen shortly after the # forward direction is established (but after some network traffic). yield self.poll(lambda: self.tub2phases) returnValue(rref1) def checkConnectedViaReverse(self, rref, targetPhases): # assert that connection[0] (from tub1 to tub2) is actually in use. # This connection uses a per-client allocated port number for the # tub1 side, and the tub2 Listener's port for the tub2 side. # Therefore tub1's Broker (as used by its RemoteReference) will have # a far-end port number that should match tub2's Listener. self.failUnlessEqual(rref.tracker.broker.transport.getPeer().port, self.portnum2) # in addition, connection[1] should have been abandoned during a # specific phase. self.failUnlessEqual(self.tub2phases, targetPhases) class CrossfireReverse(CrossfireMixin, unittest.TestCase): # just like the following Crossfire except that tub2 is the master, just # in case it makes a difference somewhere tub1IsMaster = False def test1(self): # in this test, tub2 isn't listening at all. So not only will # connection[1] fail, the tub2.getReference that uses it will fail # too (whereas in all other tests, connection[1] is abandoned but # tub2.getReference succeeds) self.makeServers(lo1={'debug_slow_connectionMade': True}) d = self.tub2.stopListeningOn(self.tub2.getListeners()[0]) d.addCallback(self._test1_1) return d def _test1_1(self, res): d,d1 = self.connect() d.addCallback(self.insert_turns, 4) d.addCallbacks(lambda res: self.fail("hey! this is supposed to fail"), self._test1_2, errbackArgs=(d1,)) return d def _test1_2(self, why, d1): from twisted.internet import error self.failUnless(why.check(error.ConnectionRefusedError)) # but now the other getReference should succeed return d1 test1.timeout = 10 def test2(self): self.makeServers(lo1={'debug_slow_connectionMade': True}) d,d1 = self.connect() d.addCallback(self.insert_turns, 4) d.addCallback(self.checkConnectedViaReverse, [negotiate.PLAINTEXT]) d.addCallback(lambda res: d1) # other getReference should work too return d test2.timeout = 10 def test3(self): self.makeServers(lo1={'debug_slow_sendPlaintextServer': True}) d,d1 = self.connect() d.addCallback(self.insert_turns, 4) d.addCallback(self.checkConnectedViaReverse, [negotiate.PLAINTEXT]) d.addCallback(lambda res: d1) # other getReference should work too return d test3.timeout = 10 @inlineCallbacks def test4(self): # prevent listener1 from doing sendHello by dropping the Deferred self.makeServers(lo1={'debug_pause_sendHello': lambda d: None}) rref1 = yield self.connect2() self.checkConnectedViaReverse(rref1, [negotiate.ENCRYPTED]) test4.timeout = 10 class Crossfire(CrossfireReverse): tub1IsMaster = True def test5(self): # this is the only test where we rely upon the fact that # makeServers() always puts the higher-numbered Tub (which will be # the master) in self.tub1 # connection[1] (the abandoned connection) is started from tub2 to # tub1. It connects, begins negotiation (tub1 is the master), but # then is stalled because we've added the debug_slow_sendDecision # flag to tub1's Listener. That allows connection[0] to begin from # tub1 to tub2, which is *not* stalled (because we added the slowdown # flag to the Listener's options, not tub1.options), so it completes # normally. When connection[1] is unpaused and hits switchToBanana, # it discovers that it already has a Broker in place, and the # connection is abandoned. self.makeServers(lo1={'debug_slow_sendDecision': True}) d,d1 = self.connect() d.addCallback(self.insert_turns, 4) d.addCallback(self.checkConnectedViaReverse, [negotiate.DECIDING]) d.addCallback(lambda res: d1) # other getReference should work too return d test5.timeout = 10 # TODO: some of these tests cause the TLS connection to be abandoned, and it # looks like TLS sockets don't shut down very cleanly. I connectionLost # getting called with the following error (instead of a normal ConnectionDone # exception): # 2005/10/10 19:56 PDT [Negotiation,0,127.0.0.1] # Negotiation.negotiationFailed: [Failure instance: Traceback: # exceptions.AttributeError: TLSConnection instance has no attribute 'socket' # twisted/internet/tcp.py:402:connectionLost # twisted/pb/negotiate.py:366:connectionLost # twisted/pb/negotiate.py:205:debug_forceTimer # twisted/pb/negotiate.py:223:debug_fireTimer # --- --- # twisted/pb/negotiate.py:324:dataReceived # twisted/pb/negotiate.py:432:handlePLAINTEXTServer # twisted/pb/negotiate.py:457:sendPlaintextServerAndStartENCRYPTED # twisted/pb/negotiate.py:494:startENCRYPTED # twisted/pb/negotiate.py:768:startTLS # twisted/internet/tcp.py:693:startTLS # twisted/internet/tcp.py:314:startTLS # ] # # specifically, I saw this happen for CrossfireReverse.test2, Parallel.test2 # other tests don't do quite what I want: closing a connection (say, due to a # duplicate broker) should send a sensible error message to the other side, # rather than triggering a low-level protocol error. class Existing(CrossfireMixin, unittest.TestCase): def checkNumBrokers(self, res, expected, dummy): if type(expected) not in (tuple,list): expected = [expected] self.failUnless(len(self.tub1.brokers) in expected) self.failUnless(len(self.tub2.brokers) in expected) def testAuthenticated(self): # When two Tubs connect, that connection should be used in the # reverse connection too self.makeServers() d = self.tub1.getReference(self.url2) d.addCallback(self._testAuthenticated_1) return d def _testAuthenticated_1(self, r12): # this should use the existing connection d = self.tub2.getReference(self.url1) d.addCallback(self.checkNumBrokers, 1, (r12,)) return d # this test will have to change when the regular Negotiation starts using # different decision blocks. The version numbers must be updated each time # the negotiation version is changed. assert negotiate.Negotiation.maxVersion == 3 MAX_HANDLED_VERSION = negotiate.Negotiation.maxVersion UNHANDLED_VERSION = 4 class NegotiationVbig(negotiate.Negotiation): maxVersion = UNHANDLED_VERSION def __init__(self, logparent): negotiate.Negotiation.__init__(self, logparent) self.negotiationOffer["extra"] = "new value" def evaluateNegotiationVersion4(self, offer): # just like v1, but different return self.evaluateNegotiationVersion1(offer) def acceptDecisionVersion4(self, decision): return self.acceptDecisionVersion1(decision) class NegotiationVbigOnly(NegotiationVbig): minVersion = UNHANDLED_VERSION class Future(BaseMixin, unittest.TestCase): def testFuture1(self): # when a peer that understands version=[1] that connects to a peer # that understands version=[1,2], they should pick version=1 # the listening Tub will have the higher tubID, and thus make the # negotiation decision url, portnum = self.makeSpecificServer(certData_high) # the client client = Tub(certData=certData_low) client.negotiationClass = NegotiationVbig client.startService() self.services.append(client) d = client.getReference(url) def _check_version(rref): ver = rref.tracker.broker._banana_decision_version self.failUnlessEqual(ver, MAX_HANDLED_VERSION) d.addCallback(_check_version) return d testFuture1.timeout = 10 def testFuture2(self): # same as before, but the connecting Tub will have the higher tubID, # and thus make the negotiation decision url, portnum = self.makeSpecificServer(certData_low) # the client client = Tub(certData=certData_high) client.negotiationClass = NegotiationVbig client.startService() self.services.append(client) d = client.getReference(url) def _check_version(rref): ver = rref.tracker.broker._banana_decision_version self.failUnlessEqual(ver, MAX_HANDLED_VERSION) d.addCallback(_check_version) return d testFuture2.timeout = 10 def testFuture3(self): # same as testFuture1, but it is the listening server that # understands [1,2] url, portnum = self.makeSpecificServer(certData_high, NegotiationVbig) client = Tub(certData=certData_low) client.startService() self.services.append(client) d = client.getReference(url) def _check_version(rref): ver = rref.tracker.broker._banana_decision_version self.failUnlessEqual(ver, MAX_HANDLED_VERSION) d.addCallback(_check_version) return d testFuture3.timeout = 10 def testFuture4(self): # same as testFuture2, but it is the listening server that # understands [1,2] url, portnum = self.makeSpecificServer(certData_low, NegotiationVbig) # the client client = Tub(certData=certData_high) client.startService() self.services.append(client) d = client.getReference(url) def _check_version(rref): ver = rref.tracker.broker._banana_decision_version self.failUnlessEqual(ver, MAX_HANDLED_VERSION) d.addCallback(_check_version) return d testFuture4.timeout = 10 def testTooFarInFuture1(self): # when a peer that understands version=[1] that connects to a peer # that only understands version=[2], they should fail to negotiate # the listening Tub will have the higher tubID, and thus make the # negotiation decision url, portnum = self.makeSpecificServer(certData_high) # the client client = Tub(certData=certData_low) client.negotiationClass = NegotiationVbigOnly client.startService() self.services.append(client) d = client.getReference(url) def _oops_succeeded(rref): self.fail("hey! this is supposed to fail") def _check_failure(f): f.trap(tokens.NegotiationError, tokens.RemoteNegotiationError) d.addCallbacks(_oops_succeeded, _check_failure) return d testTooFarInFuture1.timeout = 10 def testTooFarInFuture2(self): # same as before, but the connecting Tub will have the higher tubID, # and thus make the negotiation decision url, portnum = self.makeSpecificServer(certData_low) client = Tub(certData=certData_high) client.negotiationClass = NegotiationVbigOnly client.startService() self.services.append(client) d = client.getReference(url) def _oops_succeeded(rref): self.fail("hey! this is supposed to fail") def _check_failure(f): f.trap(tokens.NegotiationError, tokens.RemoteNegotiationError) d.addCallbacks(_oops_succeeded, _check_failure) return d testTooFarInFuture1.timeout = 10 def testTooFarInFuture3(self): # same as testTooFarInFuture1, but it is the listening server which # only understands [2] url, portnum = self.makeSpecificServer(certData_high, NegotiationVbigOnly) client = Tub(certData=certData_low) client.startService() self.services.append(client) d = client.getReference(url) def _oops_succeeded(rref): self.fail("hey! this is supposed to fail") def _check_failure(f): f.trap(tokens.NegotiationError, tokens.RemoteNegotiationError) d.addCallbacks(_oops_succeeded, _check_failure) return d testTooFarInFuture3.timeout = 10 def testTooFarInFuture4(self): # same as testTooFarInFuture2, but it is the listening server which # only understands [2] url, portnum = self.makeSpecificServer(certData_low, NegotiationVbigOnly) client = Tub(certData=certData_high) client.startService() self.services.append(client) d = client.getReference(url) def _oops_succeeded(rref): self.fail("hey! this is supposed to fail") def _check_failure(f): f.trap(tokens.NegotiationError, tokens.RemoteNegotiationError) d.addCallbacks(_oops_succeeded, _check_failure) return d testTooFarInFuture4.timeout = 10 class Replacement(BaseMixin, unittest.TestCase): # in certain circumstances, a new connection is supposed to replace an # existing one. def createDuplicateServer(self, oldtub): tub = Tub(certData=oldtub.getCertData()) tub.startService() self.services.append(tub) tub.incarnation = oldtub.incarnation tub.incarnation_string = oldtub.incarnation_string tub.slave_table = oldtub.slave_table.copy() tub.master_table = oldtub.master_table.copy() portnum = allocate_tcp_port() tub.listenOn("tcp:%d:interface=127.0.0.1" % portnum) tub.setLocation("127.0.0.1:%d" % portnum) target = Target() return tub, target, tub.registerReference(target), portnum def setUp(self): BaseMixin.setUp(self) (self.tub1, self.target1, self.furl1, l1) = \ self.createSpecificServer(certData_low) (self.tub2, self.target2, self.furl2, l2) = \ self.createSpecificServer(certData_high) # self.tub1 is the slave, self.tub2 is the master assert self.tub2.tubID > self.tub1.tubID def clone_servers(self): (self.tub1a, self.target1a, self.furl1a, l1a) = \ self.createDuplicateServer(self.tub1) (self.tub2a, self.target2a, self.furl2a, l2a) = \ self.createDuplicateServer(self.tub2) def testBouncedClient(self): # self.tub1 is the slave, self.tub2 is the master d = self.tub1.getReference(self.furl2) d2 = defer.Deferred() def _connected(rref): self.clone_servers() # our tub1a is not the same incarnation as tub1 self.tub1a.make_incarnation() # a new incarnation of the slave should replace the old connection rref.notifyOnDisconnect(d2.callback, None) return self.tub1a.getReference(self.furl2) d.addCallback(_connected) # the old rref should be broken (eventually) d.addCallback(lambda res: d2) return d def testAncientClient(self): disconnects = [] d = self.tub1.getReference(self.furl2) def _connected(rref): self.clone_servers() # old clients (foolscap-0.1.7 or earlier) don't send a # my-incarnation header, so we're supposed to reject their # connection offer self.tub1a.incarnation_string = "" # this new connection attempt will be rejected rref.notifyOnDisconnect(disconnects.append, 1) return self.shouldFail(tokens.RemoteNegotiationError, "testAncientClient", "Duplicate connection", self.tub1a.getReference, self.furl2) d.addCallback(_connected) d.addCallback(self.insert_turns, 1) def _check(res): self.failIf(disconnects) d.addCallback(_check) return d def testAncientClientWorkaround(self): self.tub2.setOption("handle-old-duplicate-connections", True) # the second connection will be dropped, because it shows up too # quickly. disconnects = [] d2 = defer.Deferred() d = self.tub1.getReference(self.furl2) def _connected(rref): self.clone_servers() # old clients (foolscap-0.1.7 or earlier) don't send a # my-incarnation header, so we're supposed to reject their # connection offer self.tub1a.incarnation_string = "" # this new connection attempt will be rejected rref.notifyOnDisconnect(disconnects.append, 1) rref.notifyOnDisconnect(d2.callback, None) return self.shouldFail(tokens.RemoteNegotiationError, "testAncientClientWorkaround", "Duplicate connection", self.tub1a.getReference, self.furl2) d.addCallback(_connected) d.addCallback(self.insert_turns, 1) def _check(res): self.failIf(disconnects) d.addCallback(_check) # now we tweak the connection-is-old threshold to allow the third # connection to succeed. def _reconnect(rref): self.tub2._handle_old_duplicate_connections = -10 return self.tub1a.getReference(self.furl2) d.addCallback(_reconnect) # the old rref should be broken (eventually) d.addCallback(lambda res: d2) return d def testLostDecisionMessage_NewServer(self): # doctor the client's memory, make it think that it had a connection # to a different incarnation of the server # this test exercises the offer_master_IR != self.tub..IR case d = self.tub1.getReference(self.furl2) d2 = defer.Deferred() def _connected(rref): # if the slave thinks it was connected to an earlier master, we # accept the new connection self.clone_servers() oldrecord = self.tub1.slave_table[self.tub2.tubID] self.tub1a.slave_table[self.tub2.tubID] = ("figment", oldrecord[1]) rref.notifyOnDisconnect(d2.callback, None) return self.tub1a.getReference(self.furl2) d.addCallback(_connected) # the old rref should be broken (eventually) d.addCallback(lambda res: d2) return d def testTwoLostDecisionMessages(self): # the client connects successfully with seqnum=1. Then the client # thinks the connection is lost, so it tries to reconnect, the server # accepts (seqnum=2), but the decision message gets lost. Then the # client tries to connect a third time: the client says it knows # about seqnum=1, which is older than the current one. We should # reject the third attempt. # we represent this case by connecting once, disconnecting, # reconnecting, then having the second tub connect with an # artificially-decremented seqnum. # this test exercises the offer_master_seqnum < existing_seqnum case disconnects = [] d = self.tub1.getReference(self.furl2) def _connect1(rref): d2 = defer.Deferred() rref.notifyOnDisconnect(d2.callback, None) rref.tracker.broker.transport.loseConnection() return d2 d.addCallback(_connect1) def _reconnect(res): return self.tub1.getReference(self.furl2) d.addCallback(_reconnect) def _connect2(rref): self.clone_servers() old_record = self.tub1a.slave_table[self.tub2.tubID] (old_IR, old_seqnum) = old_record new_record = (old_IR, str(int(old_seqnum)-1)) self.tub1a.slave_table[self.tub2.tubID] = new_record # this new connection attempt will be rejected rref.notifyOnDisconnect(disconnects.append, 1) return self.shouldFail(tokens.RemoteNegotiationError, "testTwoLostDecisionMessages", "Duplicate connection", self.tub1a.getReference, self.furl2) d.addCallback(_connect2) d.addCallback(self.insert_turns, 1) def _check(res): self.failIf(disconnects) d.addCallback(_check) return d def testWeirdSeqnum(self): # if the client sends a seqnum that's too far into the future, # something weird is going on, and we should reject the offer. # this test exercises the offer_master_seqnum > existing_seqnum case disconnects = [] d = self.tub1.getReference(self.furl2) def _connected(rref): self.clone_servers() old_record = self.tub1a.slave_table[self.tub2.tubID] (old_IR, old_seqnum) = old_record new_record = (old_IR, str(int(old_seqnum)+10)) self.tub1a.slave_table[self.tub2.tubID] = new_record # this new connection attempt will be rejected rref.notifyOnDisconnect(disconnects.append, 1) return self.shouldFail(tokens.RemoteNegotiationError, "testSimultaneousClient", "Duplicate connection", self.tub1a.getReference, self.furl2) d.addCallback(_connected) d.addCallback(self.insert_turns, 1) def _check(res): self.failIf(disconnects) d.addCallback(_check) return d def testNATEntryDropped(self): # a client connects successfully, and receives the decision, but then # the connection goes away such that the client sees it but the # server does not. The new connection should be accepted. # this test exercises the offer_master_seqnum == existing_seqnum case d = self.tub1.getReference(self.furl2) d2 = defer.Deferred() def _connected(rref): self.clone_servers() # leave the slave_table entry intact rref.notifyOnDisconnect(d2.callback, None) return self.tub1a.getReference(self.furl2) d.addCallback(_connected) # the old rref should be broken (eventually) d.addCallback(lambda res: d2) return d def testConnectionHintRace(self): # doctor the client to make the second connection look like it came # from the same batch as the existing one. This should be rejected: # this is the multiple-connection-hints case. # This is also what happens when a decision message is droped. # since this is the first time the slave tried to connect, this test # exercises the offer_master_IR == "none" case disconnects = [] d = self.tub1.getReference(self.furl2) def _connected(rref): self.clone_servers() del self.tub1a.slave_table[self.tub2.tubID] # this new connection attempt will be rejected rref.notifyOnDisconnect(disconnects.append, 1) return self.shouldFail(tokens.RemoteNegotiationError, "testSimultaneousClient", "Duplicate connection", self.tub1a.getReference, self.furl2) d.addCallback(_connected) d.addCallback(self.insert_turns, 1) def _check(res): self.failIf(disconnects) d.addCallback(_check) return d def testBouncedClient_Reverse(self): # self.tub1 is the master, self.tub2 is the slave d = self.tub2.getReference(self.furl1) d2 = defer.Deferred() def _connected(rref): self.clone_servers() # our tub2a is not the same incarnation as tub2 self.tub2a.make_incarnation() # a new incarnation of the master should replace the old connection rref.notifyOnDisconnect(d2.callback, None) return self.tub2a.getReference(self.furl1) d.addCallback(_connected) # the old rref should be broken (eventually) d.addCallback(lambda res: d2) return d foolscap-0.13.1/src/foolscap/test/test_observer.py0000644000076500000240000000131112766553111022642 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test_observer -*- from twisted.trial import unittest from twisted.internet import defer from foolscap import observer class Observer(unittest.TestCase): def test_oneshot(self): ol = observer.OneShotObserverList() rep = repr(ol) d1 = ol.whenFired() d2 = ol.whenFired() def _addmore(res): self.failUnlessEqual(res, "result") d3 = ol.whenFired() d3.addCallback(self.failUnlessEqual, "result") return d3 d1.addCallback(_addmore) ol.fire("result") rep = repr(ol) del rep d4 = ol.whenFired() dl = defer.DeferredList([d1,d2,d4]) return dl foolscap-0.13.1/src/foolscap/test/test_pb.py0000644000076500000240000006437613204511065021426 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test.test_pb -*- import re if False: import sys from twisted.python import log log.startLogging(sys.stderr) from twisted.python import failure, reflect from twisted.internet import defer from twisted.internet.interfaces import IAddress from twisted.trial import unittest from foolscap import referenceable from foolscap.tokens import BananaError, Violation, INT, STRING, OPEN from foolscap.tokens import BananaFailure from foolscap import broker, call from foolscap.constraint import IConstraint from foolscap.logging import log from foolscap.api import Tub from foolscap.util import allocate_tcp_port from foolscap.test.common import HelperTarget, TargetMixin, \ Target, TargetWithoutInterfaces, MakeTubsMixin from foolscap.eventual import fireEventually, flushEventualQueue class TestRequest(call.PendingRequest): def __init__(self, reqID, rref=None): self.answers = [] call.PendingRequest.__init__(self, reqID, rref, None, None) def complete(self, res): self.answers.append((True, res)) def fail(self, why): self.answers.append((False, why)) class NullTransport: def write(self, data): pass def loseConnection(self, why=None): pass class TestReferenceUnslicer(unittest.TestCase): # OPEN(reference), INT(refid), [STR(interfacename), INT(version)]... CLOSE def setUp(self): self.broker = broker.Broker(None) self.broker.transport = NullTransport() self.broker.connectionMade() def tearDown(self): return flushEventualQueue() def newUnslicer(self): unslicer = referenceable.ReferenceUnslicer() unslicer.broker = self.broker unslicer.opener = self.broker.rootUnslicer return unslicer def testReject(self): u = self.newUnslicer() self.failUnlessRaises(BananaError, u.checkToken, STRING, 10) u = self.newUnslicer() self.failUnlessRaises(BananaError, u.checkToken, OPEN, 0) def testNoInterfaces(self): u = self.newUnslicer() u.checkToken(INT, 0) u.receiveChild(12) rr1,rr1d = u.receiveClose() self.failUnless(rr1d is None) rr2 = self.broker.getTrackerForYourReference(12).getRef() self.failUnless(rr2) self.failUnless(isinstance(rr2, referenceable.RemoteReference)) self.failUnlessEqual(rr2.tracker.broker, self.broker) self.failUnlessEqual(rr2.tracker.clid, 12) self.failUnlessEqual(rr2.tracker.interfaceName, None) def testInterfaces(self): u = self.newUnslicer() u.checkToken(INT, 0) u.receiveChild(12) u.receiveChild("IBar") rr1,rr1d = u.receiveClose() self.failUnless(rr1d is None) rr2 = self.broker.getTrackerForYourReference(12).getRef() self.failUnless(rr2) self.failUnlessIdentical(rr1, rr2) self.failUnless(isinstance(rr2, referenceable.RemoteReference)) self.failUnlessEqual(rr2.tracker.broker, self.broker) self.failUnlessEqual(rr2.tracker.clid, 12) self.failUnlessEqual(rr2.tracker.interfaceName, "IBar") class TestAnswer(unittest.TestCase): # OPEN(answer), INT(reqID), [answer], CLOSE def setUp(self): self.broker = broker.Broker(None) self.broker.transport = NullTransport() self.broker.connectionMade() def tearDown(self): return flushEventualQueue() def newUnslicer(self): unslicer = call.AnswerUnslicer() unslicer.broker = self.broker unslicer.opener = self.broker.rootUnslicer unslicer.protocol = self.broker return unslicer def testAccept1(self): req = TestRequest(12) self.broker.addRequest(req) u = self.newUnslicer() u.start(0) u.checkToken(INT, 0) u.receiveChild(12) # causes broker.getRequest u.checkToken(STRING, 8) u.receiveChild("results") self.failIf(req.answers) u.receiveClose() # causes broker.gotAnswer self.failUnlessEqual(req.answers, [(True, "results")]) def testAccept2(self): req = TestRequest(12) req.setConstraint(IConstraint(str)) self.broker.addRequest(req) u = self.newUnslicer() u.start(0) u.checkToken(INT, 0) u.receiveChild(12) # causes broker.getRequest u.checkToken(STRING, 15) u.receiveChild("results") self.failIf(req.answers) u.receiveClose() # causes broker.gotAnswer self.failUnlessEqual(req.answers, [(True, "results")]) def testReject1(self): # answer a non-existent request req = TestRequest(12) self.broker.addRequest(req) u = self.newUnslicer() u.checkToken(INT, 0) self.failUnlessRaises(Violation, u.receiveChild, 13) def testReject2(self): # answer a request with a result that violates the constraint req = TestRequest(12) req.setConstraint(IConstraint(int)) self.broker.addRequest(req) u = self.newUnslicer() u.checkToken(INT, 0) u.receiveChild(12) self.failUnlessRaises(Violation, u.checkToken, STRING, 42) # this does not yet errback the request self.failIf(req.answers) # it gets errbacked when banana reports the violation v = Violation("icky") v.setLocation("here") u.reportViolation(BananaFailure(v)) self.failUnlessEqual(len(req.answers), 1) err = req.answers[0] self.failIf(err[0]) f = err[1] self.failUnless(f.check(Violation)) class TestReferenceable(TargetMixin, unittest.TestCase): # test how a Referenceable gets transformed into a RemoteReference as it # crosses the wire, then verify that it gets transformed back into the # original Referenceable when it comes back. Also test how shared # references to the same object are handled. def setUp(self): TargetMixin.setUp(self) self.setupBrokers() if 0: print self.callingBroker.doLog = "TX" self.targetBroker.doLog = " rx" def send(self, arg): rr, target = self.setupTarget(HelperTarget()) d = rr.callRemote("set", obj=arg) d.addCallback(self.failUnless) d.addCallback(lambda res: target.obj) return d def send2(self, arg1, arg2): rr, target = self.setupTarget(HelperTarget()) d = rr.callRemote("set2", obj1=arg1, obj2=arg2) d.addCallback(self.failUnless) d.addCallback(lambda res: (target.obj1, target.obj2)) return d def echo(self, arg): rr, target = self.setupTarget(HelperTarget()) d = rr.callRemote("echo", obj=arg) return d def testRef1(self): # Referenceables turn into RemoteReferences r = Target() d = self.send(r) d.addCallback(self._testRef1_1, r) return d def _testRef1_1(self, res, r): self.failUnless(isinstance(res, referenceable.RemoteReference)) rref = res self.failUnless(isinstance(rref.getPeer(), broker.LoopbackAddress)) self.failUnlessEqual(rref.isConnected(), True) self.failUnlessEqual(rref.getLocationHints(), []) # loopback self.failUnlessEqual(rref.getSturdyRef().getURL(), None) # keepalives are disabled self.failUnlessEqual(rref.getDataLastReceivedAt(), None) t = rref.tracker self.failUnlessEqual(t.broker, self.targetBroker) self.failUnless(type(t.clid) is int) self.failUnless(self.callingBroker.getMyReferenceByCLID(t.clid) is r) self.failUnlessEqual(t.interfaceName, 'RIMyTarget') def testRef2(self): # sending a Referenceable over the wire multiple times should result # in equivalent RemoteReferences r = Target() d = self.send(r) d.addCallback(self._testRef2_1, r) return d def _testRef2_1(self, res1, r): d = self.send(r) d.addCallback(self._testRef2_2, res1) return d def _testRef2_2(self, res2, res1): self.failUnless(res1 == res2) self.failUnless(res1 is res2) # newpb does this, oldpb didn't def testRef3(self): # sending the same Referenceable in multiple arguments should result # in equivalent RRs r = Target() d = self.send2(r, r) d.addCallback(self._testRef3_1) return d def _testRef3_1(self, (res1, res2)): self.failUnless(res1 == res2) self.failUnless(res1 is res2) def testRef4(self): # sending the same Referenceable in multiple calls will result in # equivalent RRs r = Target() rr, target = self.setupTarget(HelperTarget()) d = rr.callRemote("set", obj=r) d.addCallback(self._testRef4_1, rr, r, target) return d def _testRef4_1(self, res, rr, r, target): res1 = target.obj d = rr.callRemote("set", obj=r) d.addCallback(self._testRef4_2, target, res1) return d def _testRef4_2(self, res, target, res1): res2 = target.obj self.failUnless(res1 == res2) self.failUnless(res1 is res2) def testRef5(self): # those RemoteReferences can be used to invoke methods on the sender. # 'r' lives on side A. The anonymous target lives on side B. From # side A we invoke B.set(r), and we get the matching RemoteReference # 'rr' which lives on side B. Then we use 'rr' to invoke r.getName # from side A. r = Target() r.name = "ernie" d = self.send(r) d.addCallback(lambda rr: rr.callRemote("getName")) d.addCallback(self.failUnlessEqual, "ernie") return d def testRef6(self): # Referenceables survive round-trips r = Target() d = self.echo(r) d.addCallback(self.failUnlessIdentical, r) return d ## def NOTtestRemoteRef1(self): ## # known URLRemoteReferences turn into Referenceables ## root = Target() ## rr, target = self.setupTarget(HelperTarget()) ## self.targetBroker.factory = pb.PBServerFactory(root) ## urlRRef = self.callingBroker.remoteReferenceForName("", []) ## # urlRRef points at root ## d = rr.callRemote("set", obj=urlRRef) ## self.failUnless(dr(d)) ## self.failUnlessIdentical(target.obj, root) ## def NOTtestRemoteRef2(self): ## # unknown URLRemoteReferences are errors ## root = Target() ## rr, target = self.setupTarget(HelperTarget()) ## self.targetBroker.factory = pb.PBServerFactory(root) ## urlRRef = self.callingBroker.remoteReferenceForName("bogus", []) ## # urlRRef points at nothing ## d = rr.callRemote("set", obj=urlRRef) ## f = de(d) ## #print f ## #self.failUnlessEqual(f.type, tokens.Violation) ## self.failUnlessEqual(type(f.value), str) ## self.failUnless(f.value.find("unknown clid 'bogus'") != -1) def testArgs1(self): # sending the same non-Referenceable object in multiple calls results # in distinct objects, because the serialization scope is bounded by # each method call r = [1,2] rr, target = self.setupTarget(HelperTarget()) d = rr.callRemote("set", obj=r) d.addCallback(self._testArgs1_1, rr, r, target) # TODO: also make sure the original list goes out of scope once the # method call has finished, to guard against a leaky # reference-tracking implementation. return d def _testArgs1_1(self, res, rr, r, target): res1 = target.obj d = rr.callRemote("set", obj=r) d.addCallback(self._testArgs1_2, target, res1) return d def _testArgs1_2(self, res, target, res1): res2 = target.obj self.failUnless(res1 == res2) self.failIf(res1 is res2) def testArgs2(self): # but sending them as multiple arguments of the *same* method call # results in identical objects r = [1,2] rr, target = self.setupTarget(HelperTarget()) d = rr.callRemote("set2", obj1=r, obj2=r) d.addCallback(self._testArgs2_1, rr, target) return d def _testArgs2_1(self, res, rr, target): self.failUnlessIdentical(target.obj1, target.obj2) def testAnswer1(self): # also, shared objects in a return value should be shared r = [1,2] rr, target = self.setupTarget(HelperTarget()) target.obj = (r,r) d = rr.callRemote("get") d.addCallback(lambda res: self.failUnlessIdentical(res[0], res[1])) return d def testAnswer2(self): # but objects returned by separate method calls should be distinct rr, target = self.setupTarget(HelperTarget()) r = [1,2] target.obj = r d = rr.callRemote("get") d.addCallback(self._testAnswer2_1, rr, target) return d def _testAnswer2_1(self, res1, rr, target): d = rr.callRemote("get") d.addCallback(self._testAnswer2_2, res1) return d def _testAnswer2_2(self, res2, res1): self.failUnless(res1 == res2) self.failIf(res1 is res2) class TestFactory(unittest.TestCase): def setUp(self): self.client = None self.server = None def gotReference(self, ref): self.client = ref def tearDown(self): if self.client: self.client.broker.transport.loseConnection() if self.server: d = self.server.stopListening() else: d = defer.succeed(None) d.addCallback(flushEventualQueue) return d class TestCallable(MakeTubsMixin, unittest.TestCase): def setUp(self): self.tubA, self.tubB = self.makeTubs(2) self._log_observers_to_remove = [] def addLogObserver(self, observer): log.theLogger.addObserver(observer) self._log_observers_to_remove.append(observer) def tearDown(self): for lo in self._log_observers_to_remove: log.theLogger.removeObserver(lo) d = defer.DeferredList([s.stopService() for s in self.services]) d.addCallback(flushEventualQueue) return d def testWrongSwiss(self): target = Target() url = self.tubB.registerReference(target) badurl = url + "_wrong" swiss = url[url.rindex("/")+1:] d = self.tubA.getReference(badurl) def _check(f): self.failIf(swiss in str(f), "swissnum revealed") self.failUnless(swiss[:2] in str(f), "swissnum hint not given") d.addErrback(_check) return d def testGetSturdyRef(self): target = Target() url = self.tubB.registerReference(target) d = self.tubA.getReference(url) def _check(rref): sr = rref.getSturdyRef() self.failUnlessEqual(sr.getURL(), url) peer = rref.getPeer() self.failUnless(IAddress.providedBy(peer)) self.failUnlessEqual(peer.type, "TCP") self.failUnlessEqual(peer.host, "127.0.0.1") self.failUnlessEqual(rref.getRemoteTubID(), self.tubB.getTubID()) self.failUnlessEqual(rref.isConnected(), True) self.failUnlessEqual(rref.getLocationHints(), ['tcp:127.0.0.1:%d' % self.tub_ports[1]]) d.addCallback(_check) return d def testLogLocalFailure(self): self.tubB.setOption("logLocalFailures", True) target = Target() logs = [] self.addLogObserver(logs.append) url = self.tubB.registerReference(target) d = self.tubA.getReference(url) d.addCallback(lambda rref: rref.callRemote("fail")) # this will cause some text to be logged with log.msg. TODO: capture # this text and look at it more closely. def _check(res): self.failUnless(isinstance(res, failure.Failure)) res.trap(ValueError) messages = [log.format_message(e) for e in logs] failures = [e['failure'] for e in logs if "failure" in e] text = "\n".join(messages) msg = ("an inbound callRemote that we [%s] executed (on behalf of " "someone else, TubID %s) failed\n" % (self.tubB.getShortTubID(), self.tubA.getShortTubID())) self.failUnless(msg in text, "msg '%s' not in text '%s'" % (msg, text)) self.failUnless("\n reqID=2, rref=, methname=RIMyTarget.fail\n" % url) in text) #self.failUnless("\n args=[]\n" in text) # TODO: log these too #self.failUnless("\n kwargs={}\n" in text) self.failUnlessEqual(len(failures), 1) f = failures[0] self.failUnless("Traceback (most recent call last):\n" in str(f)) self.failUnless("\nexceptions.ValueError: you asked me to fail\n" in str(f)) d.addBoth(_check) return d testLogRemoteFailure.timeout = 2 def testBoundMethod(self): target = Target() meth_url = self.tubB.registerReference(target.remote_add) d = self.tubA.getReference(meth_url) d.addCallback(self._testBoundMethod_1) return d testBoundMethod.timeout = 5 def _testBoundMethod_1(self, ref): self.failUnless(isinstance(ref, referenceable.RemoteMethodReference)) #self.failUnlessEqual(ref.getSchemaName(), # RIMyTarget.__remote_name__ + "/remote_add") d = ref.callRemote(a=1, b=2) d.addCallback(lambda res: self.failUnlessEqual(res, 3)) return d def testFunction(self): l = [] # we need a keyword arg here def append(what): l.append(what) func_url = self.tubB.registerReference(append) d = self.tubA.getReference(func_url) d.addCallback(self._testFunction_1, l) return d testFunction.timeout = 5 def _testFunction_1(self, ref, l): self.failUnless(isinstance(ref, referenceable.RemoteMethodReference)) d = ref.callRemote(what=12) d.addCallback(lambda res: self.failUnlessEqual(l, [12])) return d class TestNotifyOnConnectionLost(unittest.TestCase): """ Tests for L{Broker._notifyOnConnectionLost}. """ def testCalled(self): """ The object passed to L{Broker._notifyOnConnectionLost} is called when the L{Broker} is notify that its connection has been lost. """ transport = NullTransport() protocol = broker.Broker(None) protocol.makeConnection(transport) disconnected = [] protocol._notifyOnConnectionLost(lambda: disconnected.append(1)) protocol._notifyOnConnectionLost(lambda: disconnected.append(2)) protocol.connectionLost(failure.Failure(Exception("Connection lost"))) d = flushEventualQueue() def flushed(ignored): self.assertEqual([1, 2], disconnected) d.addCallback(flushed) return d class TestService(unittest.TestCase): def setUp(self): self.services = [Tub()] self.services[0].startService() def tearDown(self): d = defer.DeferredList([s.stopService() for s in self.services]) d.addCallback(flushEventualQueue) return d def testRegister(self): s = self.services[0] portnum = allocate_tcp_port() s.listenOn("tcp:%d:interface=127.0.0.1" % portnum) s.setLocation("127.0.0.1:%d" % portnum) t1 = Target() public_url = s.registerReference(t1, "target") self.failUnless(public_url.startswith("pb://")) self.failUnless(public_url.endswith("@127.0.0.1:%d/target" % portnum)) self.failUnlessEqual(s.registerReference(t1, "target"), public_url) self.failUnlessIdentical(s.getReferenceForURL(public_url), t1) t2 = Target() private_url = s.registerReference(t2) self.failUnlessEqual(s.registerReference(t2), private_url) self.failUnlessIdentical(s.getReferenceForURL(private_url), t2) s.unregisterURL(public_url) self.failUnlessRaises(KeyError, s.getReferenceForURL, public_url) s.unregisterReference(t2) self.failUnlessRaises(KeyError, s.getReferenceForURL, private_url) # TODO: check what happens when you register the same referenceable # under multiple URLs def getRef(self, target): self.services.append(Tub()) s1 = self.services[0] s2 = self.services[1] s2.startService() portnum = allocate_tcp_port() s1.listenOn("tcp:%d:interface=127.0.0.1" % portnum) s1.setLocation("127.0.0.1:%d" % portnum) public_url = s1.registerReference(target, "target") self.public_url = public_url d = s2.getReference(public_url) return d def testConnect1(self): t1 = TargetWithoutInterfaces() d = self.getRef(t1) d.addCallback(lambda ref: ref.callRemote('add', a=2, b=3)) d.addCallback(self._testConnect1, t1) return d testConnect1.timeout = 5 def _testConnect1(self, res, t1): self.failUnlessEqual(t1.calls, [(2,3)]) self.failUnlessEqual(res, 5) def testConnect2(self): t1 = Target() d = self.getRef(t1) d.addCallback(lambda ref: ref.callRemote('add', a=2, b=3)) d.addCallback(self._testConnect2, t1) return d testConnect2.timeout = 5 def _testConnect2(self, res, t1): self.failUnlessEqual(t1.calls, [(2,3)]) self.failUnlessEqual(res, 5) def testConnect3(self): # test that we can get the reference multiple times t1 = Target() d = self.getRef(t1) d.addCallback(lambda ref: ref.callRemote('add', a=2, b=3)) def _check(res): self.failUnlessEqual(t1.calls, [(2,3)]) self.failUnlessEqual(res, 5) t1.calls = [] d.addCallback(_check) d.addCallback(lambda res: self.services[1].getReference(self.public_url)) d.addCallback(lambda ref: ref.callRemote('add', a=5, b=6)) def _check2(res): self.failUnlessEqual(t1.calls, [(5,6)]) self.failUnlessEqual(res, 11) d.addCallback(_check2) return d testConnect3.timeout = 5 def TODO_testStatic(self): # make sure we can register static data too, at least hashable ones t1 = (1,2,3) d = self.getRef(t1) d.addCallback(lambda ref: self.failUnlessEqual(ref, (1,2,3))) return d #testStatic.timeout = 2 def testBadMethod(self): t1 = Target() d = self.getRef(t1) d.addCallback(lambda ref: ref.callRemote('missing', a=2, b=3)) d.addCallbacks(self._testBadMethod_cb, self._testBadMethod_eb) return d testBadMethod.timeout = 5 def _testBadMethod_cb(self, res): self.fail("method wasn't supposed to work") def _testBadMethod_eb(self, f): #self.failUnlessEqual(f.type, 'foolscap.tokens.Violation') self.failUnlessEqual(f.type, Violation) self.failUnless(re.search(r'RIMyTarget\(.*\) does not offer missing', str(f))) def testBadMethod2(self): t1 = TargetWithoutInterfaces() d = self.getRef(t1) d.addCallback(lambda ref: ref.callRemote('missing', a=2, b=3)) d.addCallbacks(self._testBadMethod_cb, self._testBadMethod2_eb) return d testBadMethod2.timeout = 5 def _testBadMethod2_eb(self, f): self.failUnlessEqual(reflect.qual(f.type), 'exceptions.AttributeError') self.failUnlessSubstring("TargetWithoutInterfaces", f.value) self.failUnlessSubstring(" has no attribute 'remote_missing'", f.value) # TODO: # when the Violation is remote, it is reported in a CopiedFailure, which # means f.type is a string. When it is local, it is reported in a Failure, # and f.type is the tokens.Violation class. I'm not sure how I feel about # these being different. # TODO: tests to port from oldpb suite # testTooManyRefs: sending pb.MAX_BROKER_REFS across the wire should die # testFactoryCopy? # tests which aren't relevant right now but which might be once we port the # corresponding functionality: # # testObserve, testCache (pb.Cacheable) # testViewPoint # testPublishable (spread.publish??) # SpreadUtilTestCase (spread.util) # NewCredTestCase # tests which aren't relevant and aren't like to ever be # # PagingTestCase # ConnectionTestCase (oldcred) # NSPTestCase foolscap-0.13.1/src/foolscap/test/test_promise.py0000644000076500000240000001624012766553111022500 0ustar warnerstaff00000000000000 from twisted.trial import unittest from twisted.python.failure import Failure from foolscap.promise import makePromise, send, sendOnly, when, UsageError from foolscap.eventual import flushEventualQueue, fireEventually class KaboomError(Exception): pass class Target: def __init__(self): self.calls = [] def one(self, a): self.calls.append(("one", a)) return a+1 def two(self, a, b=2, **kwargs): self.calls.append(("two", a, b, kwargs)) def fail(self, arg): raise KaboomError("kaboom!") class Counter: def __init__(self, count=0): self.count = count def add(self, value): self.count += value return self class Send(unittest.TestCase): def tearDown(self): return flushEventualQueue() def testBasic(self): p,r = makePromise() def _check(res, *args, **kwargs): self.failUnlessEqual(res, 1) self.failUnlessEqual(args, ("one",)) self.failUnlessEqual(kwargs, {"two": 2}) p2 = p._then(_check, "one", two=2) self.failUnlessIdentical(p2, p) r(1) def testBasicFailure(self): p,r = makePromise() def _check(res, *args, **kwargs): self.failUnless(isinstance(res, Failure)) self.failUnless(res.check(KaboomError)) self.failUnlessEqual(args, ("one",)) self.failUnlessEqual(kwargs, {"two": 2}) p2 = p._except(_check, "one", two=2) self.failUnlessIdentical(p2, p) r(Failure(KaboomError("oops"))) def testSend(self): t = Target() p = send(t).one(1) self.failIf(t.calls) def _check(res): self.failUnlessEqual(res, 2) self.failUnlessEqual(t.calls, [("one", 1)]) p._then(_check) when(p).addCallback(_check) # check it twice to test both syntaxes def testOrdering(self): t = Target() p1 = send(t).one(1) p2 = send(t).two(3, k="extra") self.failIf(t.calls) def _check1(res): # we can't check t.calls here: the when() clause is not # guaranteed to fire before the second send. self.failUnlessEqual(res, 2) when(p1).addCallback(_check1) def _check2(res): self.failUnlessEqual(res, None) when(p2).addCallback(_check2) def _check3(res): self.failUnlessEqual(t.calls, [("one", 1), ("two", 3, 2, {"k": "extra"}), ]) fireEventually().addCallback(_check3) def testFailure(self): t = Target() p1 = send(t).fail(0) def _check(res): self.failUnless(isinstance(res, Failure)) self.failUnless(res.check(KaboomError)) p1._then(lambda res: self.fail("we were supposed to fail")) p1._except(_check) when(p1).addBoth(_check) def testBadName(self): t = Target() p1 = send(t).missing(0) def _check(res): self.failUnless(isinstance(res, Failure)) self.failUnless(res.check(AttributeError)) when(p1).addBoth(_check) def testDisableDataflowStyle(self): p,r = makePromise() p._useDataflowStyle = False def wrong(p): p.one(12) self.failUnlessRaises(AttributeError, wrong, p) def testNoMultipleResolution(self): p,r = makePromise() r(3) self.failUnlessRaises(UsageError, r, 4) def testResolveBefore(self): t = Target() p,r = makePromise() r(t) p = send(p).one(2) def _check(res): self.failUnlessEqual(res, 3) when(p).addCallback(_check) def testResolveAfter(self): t = Target() p,r = makePromise() p = send(p).one(2) def _check(res): self.failUnlessEqual(res, 3) when(p).addCallback(_check) r(t) def testResolveFailure(self): p,r = makePromise() p = send(p).one(2) def _check(res): self.failUnless(isinstance(res, Failure)) self.failUnless(res.check(KaboomError)) when(p).addBoth(_check) f = Failure(KaboomError("oops")) r(f) class Call(unittest.TestCase): def tearDown(self): return flushEventualQueue() def testResolveBefore(self): t = Target() p1,r = makePromise() r(t) p2 = p1.one(2) def _check(res): self.failUnlessEqual(res, 3) p2._then(_check) def testResolveAfter(self): t = Target() p1,r = makePromise() p2 = p1.one(2) def _check(res): self.failUnlessEqual(res, 3) p2._then(_check) r(t) def testResolveFailure(self): p1,r = makePromise() p2 = p1.one(2) def _check(res): self.failUnless(isinstance(res, Failure)) self.failUnless(res.check(KaboomError)) p2._then(lambda res: self.fail("this was supposed to fail")) p2._except(_check) f = Failure(KaboomError("oops")) r(f) class SendOnly(unittest.TestCase): def testNear(self): t = Target() sendOnly(t).one(1) self.failIf(t.calls) def _check(res): self.failUnlessEqual(t.calls, [("one", 1)]) d = flushEventualQueue() d.addCallback(_check) return d def testResolveBefore(self): t = Target() p,r = makePromise() r(t) sendOnly(p).one(1) d = flushEventualQueue() def _check(res): self.failUnlessEqual(t.calls, [("one", 1)]) d.addCallback(_check) return d def testResolveAfter(self): t = Target() p,r = makePromise() sendOnly(p).one(1) r(t) d = flushEventualQueue() def _check(res): self.failUnlessEqual(t.calls, [("one", 1)]) d.addCallback(_check) return d class Chained(unittest.TestCase): def tearDown(self): return flushEventualQueue() def testResolveToAPromise(self): p1,r1 = makePromise() p2,r2 = makePromise() def _check(res): self.failUnlessEqual(res, 1) p1._then(_check) r1(p2) def _continue(res): r2(1) flushEventualQueue().addCallback(_continue) return when(p1) def testResolveToABrokenPromise(self): p1,r1 = makePromise() p2,r2 = makePromise() r1(p2) def _continue(res): r2(Failure(KaboomError("foom"))) flushEventualQueue().addCallback(_continue) def _check2(res): self.failUnless(isinstance(res, Failure)) self.failUnless(res.check(KaboomError)) d = when(p1) d.addBoth(_check2) return d def testChained1(self): p1,r = makePromise() p2 = p1.add(2) p3 = p2.add(3) def _check(c): self.failUnlessEqual(c.count, 5) p3._then(_check) r(Counter(0)) def testChained2(self): p1,r = makePromise() def _check(c, expected): self.failUnlessEqual(c.count, expected) p1.add(2).add(3)._then(_check, 6) r(Counter(1)) foolscap-0.13.1/src/foolscap/test/test_reconnector.py0000644000076500000240000002662713204746217023354 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test.test_reconnector -*- import time from twisted.trial import unittest from foolscap.api import Tub, eventually, flushEventualQueue from foolscap.test.common import HelperTarget, MakeTubsMixin, PollMixin from foolscap.util import allocate_tcp_port from twisted.internet import defer, reactor, error from foolscap import negotiate, referenceable class AlwaysFailNegotiation(negotiate.Negotiation): def sendHello(self): hello = {"error": "I always fail", 'my-tub-id': self.myTubID, } self.sendBlock(hello) self.receive_phase = negotiate.ABANDONED class Reconnector(MakeTubsMixin, PollMixin, unittest.TestCase): def setUp(self): self.tubA, self.tubB = self.makeTubs(2) def tearDown(self): d = defer.DeferredList([s.stopService() for s in self.services]) d.addCallback(flushEventualQueue) return d def test_try(self): self.count = 0 self.attached = False self.done = defer.Deferred() target = HelperTarget("bob") self.url = self.tubB.registerReference(target) self._time1 = time.time() self.rc = self.tubA.connectTo(self.url, self._got_ref, "arg", kw="kwarg") ri = self.rc.getReconnectionInfo() self.assertEqual(ri.state, "connecting") # at least make sure the stopConnecting method is present, even if we # don't have a real test for it yet self.failUnless(self.rc.stopConnecting) return self.done def _got_ref(self, rref, arg, kw): self.failUnlessEqual(self.attached, False) self.attached = True self.failUnlessEqual(arg, "arg") self.failUnlessEqual(kw, "kwarg") ri = self.rc.getReconnectionInfo() self.assertEqual(ri.state, "connected") time2 = time.time() last = ri.lastAttempt self.assert_(self._time1 <= last <= time2, (self._time1, last, time2)) ci = ri.connectionInfo self.assertEqual(ci.connected, True) hints = referenceable.SturdyRef(self.url).getTubRef().getLocations() expected_hint = hints[0] self.assertEqual(ci.winningHint, expected_hint) self.assertEqual(ci.listenerStatus, (None, None)) self.assertEqual(ci.connectorStatuses, {expected_hint: "successful"}) self.assertEqual(ci.connectionHandlers, {expected_hint: "tcp"}) self.count += 1 rref.notifyOnDisconnect(self._disconnected, self.count) if self.count < 2: # forcibly disconnect it eventually(rref.tracker.broker.transport.loseConnection) else: self.done.callback("done") def _disconnected(self, count): self.failUnlessEqual(self.attached, True) self.failUnlessEqual(count, self.count) self.attached = False ri = self.rc.getReconnectionInfo() self.assertEqual(ri.state, "waiting") # The next connection attempt will be about 1.0s after disconnect. # We'll assert that this is in the future, although on very slow # systems, this may not be true. now = time.time() next_attempt = ri.nextAttempt self.assert_(now <= next_attempt, (now, next_attempt)) def _connected(self, ref, notifiers, accumulate): accumulate.append(ref) if notifiers: notifiers.pop(0).callback(ref) def stall(self, timeout, res=None): d = defer.Deferred() reactor.callLater(timeout, d.callback, res) return d @defer.inlineCallbacks def test_retry(self): tubC = Tub(certData=self.tubB.getCertData()) connects = [] target = HelperTarget("bob") url = self.tubB.registerReference(target, "target") portb = self.tub_ports[1] d1 = defer.Deferred() notifiers = [d1] self.services.remove(self.tubB) # This will fail, since tubB is not listening anymore. Wait until it's # moved to the "waiting" state. yield self.tubB.stopService() rc = self.tubA.connectTo(url, self._connected, notifiers, connects) yield self.poll(lambda: rc.getReconnectionInfo().state == "waiting") self.failUnlessEqual(len(connects), 0) # now start tubC listening on the same port that tubB used to, which # should allow the connection to complete (since they both use the same # certData) self.services.append(tubC) tubC.startService() tubC.listenOn("tcp:%d:interface=127.0.0.1" % portb) tubC.setLocation("tcp:127.0.0.1:%d" % portb) url2 = tubC.registerReference(target, "target") assert url2 == url yield d1 self.failUnlessEqual(len(connects), 1) rc.stopConnecting() @defer.inlineCallbacks def test_negotiate_fails_and_retry(self): connects = [] target = HelperTarget("bob") url = self.tubB.registerReference(target, "target") hint = referenceable.SturdyRef(url).getTubRef().getLocations()[0] l = self.tubB.getListeners()[0] l._negotiationClass = AlwaysFailNegotiation portb = self.tub_ports[1] d1 = defer.Deferred() notifiers = [d1] rc = self.tubA.connectTo(url, self._connected, notifiers, connects) yield self.poll(lambda: rc.getReconnectionInfo().state == "waiting") # the reconnector should have failed once or twice, since the # negotiation would always fail. self.failUnlessEqual(len(connects), 0) ci = rc.getReconnectionInfo().connectionInfo cs = ci.connectorStatuses self.assertEqual(cs, {hint: "negotiation failed: I always fail"}) # Now we fix tubB. We only touched the Listener, so re-doing the # listenOn should clear it. yield self.tubB.stopListeningOn(l) self.tubB.listenOn("tcp:%d:interface=127.0.0.1" % portb) # the next time the reconnector tries, it should succeed yield d1 self.failUnlessEqual(len(connects), 1) rc.stopConnecting() @defer.inlineCallbacks def test_lose_and_retry(self): tubC = Tub(self.tubB.getCertData()) connects = [] d1 = defer.Deferred() d2 = defer.Deferred() notifiers = [d1, d2] target = HelperTarget("bob") url = self.tubB.registerReference(target, "target") portb = self.tub_ports[1] rc = self.tubA.connectTo(url, self._connected, notifiers, connects) yield d1 self.assertEqual(rc.getReconnectionInfo().state, "connected") # we are now connected to tubB. Shut it down to force a disconnect. self.services.remove(self.tubB) yield self.tubB.stopService() # wait for at least one retry yield self.poll(lambda: rc.getReconnectionInfo().state == "waiting") # wait a few seconds more to give the Reconnector a chance to try and # fail a few times. It isn't easy to catch the "connecting" state since # the target is local and the kernel knows that it's not listening. # TODO: add an internal retry counter to the Reconnector that we can # poll for tests. yield self.stall(2) # now start tubC listening on the same port that tubB used to, # which should allow the connection to complete (since they both # use the same certData) self.services.append(tubC) tubC.startService() tubC.listenOn("tcp:%d:interface=127.0.0.1" % portb) tubC.setLocation("tcp:127.0.0.1:%d" % portb) url2 = tubC.registerReference(target, "target") assert url2 == url # this will fire when the second connection has been made yield d2 self.failUnlessEqual(len(connects), 2) rc.stopConnecting() @defer.inlineCallbacks def test_stop_trying(self): connects = [] target = HelperTarget("bob") url = self.tubB.registerReference(target, "target") self.services.remove(self.tubB) # this will fail, since tubB is not listening anymore yield self.tubB.stopService() rc = self.tubA.connectTo(url, self._connected, [], connects) rc.verbose = True # get better code coverage # wait for at least one retry yield self.poll(lambda: rc.getReconnectionInfo().state == "waiting") # and a bit more, for good measure yield self.stall(2) self.failUnlessEqual(len(connects), 0) f = rc.getLastFailure() self.failUnless(f.check(error.ConnectionRefusedError)) delay = rc.getDelayUntilNextAttempt() self.failUnless(delay > 0, delay) self.failUnless(delay < 60, delay) rc.reset() delay = rc.getDelayUntilNextAttempt() self.failUnless(delay < 2) # this stopConnecting occurs while the reconnector's timer is # active rc.stopConnecting() self.failUnlessEqual(rc.getDelayUntilNextAttempt(), None) # if it keeps trying, we'll see a dirty reactor class Unstarted(MakeTubsMixin, unittest.TestCase): def setUp(self): self.tubA, self.tubB = self.makeTubs(2, start=False) def test_unstarted(self): target = HelperTarget("bob") url = self.tubB.registerReference(target) rc = self.tubA.connectTo(url, None) ri = rc.getReconnectionInfo() self.assertEqual(ri.state, "unstarted") # TODO: look at connections that succeed because of a listener, and also # loopback class Failed(PollMixin, unittest.TestCase): def setUp(self): self.services = [] def tearDown(self): d = defer.DeferredList([s.stopService() for s in self.services]) d.addCallback(flushEventualQueue) return d @defer.inlineCallbacks def test_bad_hints(self): self.tubA = Tub() self.tubA.startService() self.services.append(self.tubA) self.tubB = Tub() self.tubB.startService() self.services.append(self.tubB) portnum = allocate_tcp_port() self.tubB.listenOn("tcp:%d:interface=127.0.0.1" % portnum) bad1 = "no-colon" bad2 = "unknown:foo" bad3 = "tcp:300.300.300.300:333" self.tubB.setLocation(bad1, bad2, bad3) target = HelperTarget("bob") url = self.tubB.registerReference(target) rc = self.tubA.connectTo(url, None) ri = rc.getReconnectionInfo() self.assertEqual(ri.state, "connecting") yield self.poll(lambda: rc.getReconnectionInfo().state != "connecting") # now look at the details ri = rc.getReconnectionInfo() self.assertEqual(ri.state, "waiting") ci = ri.connectionInfo self.assertEqual(ci.connected, False) self.assertEqual(ci.winningHint, None) s = ci.connectorStatuses self.assertEqual(set(s.keys()), set([bad1, bad2, bad3])) self.assertEqual(s[bad1], "bad hint: no colon") self.assertEqual(s[bad2], "bad hint: no handler registered") self.assertIn("DNS lookup failed", s[bad3]) ch = ci.connectionHandlers self.assertEqual(ch, {bad2: None, bad3: "tcp"}) # another test: determine the target url early, but don't actually register # the reference yet. Start the reconnector, let it fail once, then register # the reference and make sure the retry succeeds. This will distinguish # between connection/negotiation failures and object-lookup failures, both of # which ought to be handled by Reconnector. I suspect the object-lookup # failures are not yet. # test that Tub shutdown really stops all Reconnectors foolscap-0.13.1/src/foolscap/test/test_reference.py0000644000076500000240000000454212766553111022762 0ustar warnerstaff00000000000000 from zope.interface import implements from twisted.trial import unittest from foolscap.ipb import IRemoteReference from foolscap.test.common import HelperTarget, Target, ShouldFailMixin from foolscap.eventual import flushEventualQueue from foolscap import broker, referenceable, api class Remote: implements(IRemoteReference) pass class LocalReference(unittest.TestCase, ShouldFailMixin): def tearDown(self): return flushEventualQueue() def ignored(self): pass def test_remoteReference(self): r = Remote() rref = IRemoteReference(r) self.failUnlessIdentical(r, rref) def test_callRemote(self): t = HelperTarget() t.obj = None rref = IRemoteReference(t) marker = rref.notifyOnDisconnect(self.ignored, "args", kwargs="foo") rref.dontNotifyOnDisconnect(marker) d = rref.callRemote("set", 12) # the callRemote should be put behind an eventual-send self.failUnlessEqual(t.obj, None) def _check(res): self.failUnlessEqual(t.obj, 12) self.failUnlessEqual(res, True) d.addCallback(_check) return d def test_callRemoteOnly(self): t = HelperTarget() t.obj = None rref = IRemoteReference(t) rc = rref.callRemoteOnly("set", 12) self.failUnlessEqual(rc, None) def test_fail(self): t = Target() rref = IRemoteReference(t) return self.shouldFail(ValueError, "test_fail", "you asked me to fail", rref.callRemote, "fail") class TubID(unittest.TestCase): def test_tubid_must_match(self): good_tubid = "fu2bixsrymp34hwrnukv7hzxc2vrhqqa" bad_tubid = "v5mwmba42j4hu5jxuvgciasvo4aqldkq" good_furl = "pb://" + good_tubid + "@127.0.0.1:1234/swissnum" bad_furl = "pb://" + bad_tubid + "@127.0.0.1:1234/swissnum" ri = "remote_interface_name" good_broker = broker.Broker(referenceable.TubRef(good_tubid)) good_tracker = referenceable.RemoteReferenceTracker(good_broker, 0, good_furl, ri) del good_tracker self.failUnlessRaises(api.BananaError, referenceable.RemoteReferenceTracker, good_broker, 0, bad_furl, ri) foolscap-0.13.1/src/foolscap/test/test_registration.py0000644000076500000240000000433312766553111023534 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test.test_registration -*- from twisted.trial import unittest import os, weakref, gc from foolscap.api import Tub from foolscap.test.common import HelperTarget from foolscap.tokens import WrongNameError class Registration(unittest.TestCase): def testStrong(self): t1 = HelperTarget() tub = Tub() tub.setLocation("bogus:1234567") u1 = tub.registerReference(t1) del u1 results = [] w1 = weakref.ref(t1, results.append) del t1 gc.collect() # t1 should still be alive self.failUnless(w1()) self.failUnlessEqual(results, []) tub.unregisterReference(w1()) gc.collect() # now it should be dead self.failIf(w1()) self.failUnlessEqual(len(results), 1) def testWeak(self): t1 = HelperTarget() tub = Tub() tub.setLocation("bogus:1234567") name = tub._assignName(t1) url = tub.buildURL(name) del url results = [] w1 = weakref.ref(t1, results.append) del t1 gc.collect() # t1 should be dead self.failIf(w1()) self.failUnlessEqual(len(results), 1) def TODO_testNonweakrefable(self): # what happens when we register a non-Referenceable? We don't really # need this yet, but as registerReference() becomes more generalized # into just plain register(), we'll want to provide references to # Copyables and ordinary data structures too. Let's just test that # this doesn't cause an error. target = [] tub = Tub() tub.setLocation("bogus:1234567") url = tub.registerReference(target) del url def test_duplicate(self): basedir = "test_registration" os.makedirs(basedir) ff = os.path.join(basedir, "duplicate.furl") t1 = HelperTarget() tub = Tub() tub.setLocation("bogus:1234567") u1 = tub.registerReference(t1, "name", furlFile=ff) u2 = tub.registerReference(t1, "name", furlFile=ff) self.failUnlessEqual(u1, u2) self.failUnlessRaises(WrongNameError, tub.registerReference, t1, "newname", furlFile=ff) foolscap-0.13.1/src/foolscap/test/test_schema.py0000644000076500000240000004172212766553111022265 0ustar warnerstaff00000000000000 import re from twisted.trial import unittest from foolscap import schema, copyable, broker from foolscap.tokens import Violation, InvalidRemoteInterface from foolscap.constraint import IConstraint from foolscap.remoteinterface import RemoteMethodSchema, \ RemoteInterfaceConstraint, LocalInterfaceConstraint from foolscap.referenceable import RemoteReferenceTracker, \ RemoteReference, Referenceable, TubRef from foolscap.test import common class Dummy: pass HEADER = 64 INTSIZE = HEADER+1 STR10 = HEADER+1+10 class ConformTest(unittest.TestCase): """This tests how Constraints are asserted on outbound objects (where the object already exists). Inbound constraints are checked in test_banana.InboundByteStream in the various testConstrainedFoo methods. """ def conforms(self, c, obj): c.checkObject(obj, False) def violates(self, c, obj): self.assertRaises(schema.Violation, c.checkObject, obj, False) def testInteger(self): # s_int32_t c = schema.IntegerConstraint() self.conforms(c, 123) self.violates(c, 2**64) self.conforms(c, 0) self.conforms(c, 2**31-1) self.violates(c, 2**31) self.conforms(c, -2**31) self.violates(c, -2**31-1) self.violates(c, "123") self.violates(c, Dummy()) self.violates(c, None) def testLargeInteger(self): c = schema.IntegerConstraint(64) self.conforms(c, 123) self.violates(c, "123") self.violates(c, None) self.conforms(c, 2**512-1) self.violates(c, 2**512) self.conforms(c, -2**512+1) self.violates(c, -2**512) def testByteString(self): c = schema.ByteStringConstraint(10) self.conforms(c, "I'm short") self.violates(c, "I am too long") self.conforms(c, "a" * 10) self.violates(c, "a" * 11) self.violates(c, 123) self.violates(c, Dummy()) self.violates(c, None) c2 = schema.ByteStringConstraint(15, 10) self.violates(c2, "too short") self.conforms(c2, "long enough") self.violates(c2, "this is too long") self.violates(c2, u"I am unicode") c3 = schema.ByteStringConstraint(regexp="needle") self.violates(c3, "no present") self.conforms(c3, "needle in a haystack") c4 = schema.ByteStringConstraint(regexp="[abc]+") self.violates(c4, "spelled entirely without those letters") self.conforms(c4, "add better cases") c5 = schema.ByteStringConstraint(regexp=re.compile("\d+\s\w+")) self.conforms(c5, ": 123 boo") self.violates(c5, "more than 1 spaces") self.violates(c5, "letters first 123") def testString(self): # this test will change once the definition of "StringConstraint" # changes. For now, we assert that StringConstraint is the same as # ByteStringConstraint. c = schema.StringConstraint(20) self.conforms(c, "I'm short") self.violates(c, u"I am unicode") def testUnicode(self): c = schema.UnicodeConstraint(10) self.violates(c, "I'm a bytestring") self.conforms(c, u"I'm short") self.violates(c, u"I am too long") self.conforms(c, u"a" * 10) self.violates(c, u"a" * 11) self.violates(c, 123) self.violates(c, Dummy()) self.violates(c, None) c2 = schema.UnicodeConstraint(15, 10) self.violates(c2, "I'm a bytestring") self.violates(c2, u"too short") self.conforms(c2, u"long enough") self.violates(c2, u"this is too long") c3 = schema.UnicodeConstraint(regexp="needle") self.violates(c3, "I'm a bytestring") self.violates(c3, u"no present") self.conforms(c3, u"needle in a haystack") c4 = schema.UnicodeConstraint(regexp="[abc]+") self.violates(c4, "I'm a bytestring") self.violates(c4, u"spelled entirely without those letters") self.conforms(c4, u"add better cases") c5 = schema.UnicodeConstraint(regexp=re.compile("\d+\s\w+")) self.violates(c5, "I'm a bytestring") self.conforms(c5, u": 123 boo") self.violates(c5, u"more than 1 spaces") self.violates(c5, u"letters first 123") def testBool(self): c = schema.BooleanConstraint() self.conforms(c, False) self.conforms(c, True) self.violates(c, 0) self.violates(c, 1) self.violates(c, "vrai") self.violates(c, Dummy()) self.violates(c, None) def testPoly(self): c = schema.PolyConstraint(schema.ByteStringConstraint(100), schema.IntegerConstraint()) self.conforms(c, "string") self.conforms(c, 123) self.violates(c, u"unicode") self.violates(c, 123.4) self.violates(c, ["not", "a", "list"]) def testTuple(self): c = schema.TupleConstraint(schema.ByteStringConstraint(10), schema.ByteStringConstraint(100), schema.IntegerConstraint() ) self.conforms(c, ("hi", "there buddy, you're number", 1)) self.violates(c, "nope") self.violates(c, ("string", "string", "NaN")) self.violates(c, ("string that is too long", "string", 1)) self.violates(c, ["Are tuples", "and lists the same?", 0]) def testNestedTuple(self): inner = schema.TupleConstraint(schema.ByteStringConstraint(10), schema.IntegerConstraint()) outer = schema.TupleConstraint(schema.ByteStringConstraint(100), inner) self.conforms(inner, ("hi", 2)) self.conforms(outer, ("long string here", ("short", 3))) self.violates(outer, (("long string here", ("short", 3, "extra")))) self.violates(outer, (("long string here", ("too long string", 3)))) outer2 = schema.TupleConstraint(inner, inner) self.conforms(outer2, (("hi", 1), ("there", 2)) ) self.violates(outer2, ("hi", 1, "flat", 2) ) def testRecursion(self): # we have to fiddle with PolyConstraint's innards value = schema.ChoiceOf(schema.ByteStringConstraint(), schema.IntegerConstraint(), # will add 'value' here ) self.conforms(value, "key") self.conforms(value, 123) self.violates(value, []) mapping = schema.TupleConstraint(schema.ByteStringConstraint(10), value) self.conforms(mapping, ("name", "key")) self.conforms(mapping, ("name", 123)) value.alternatives = value.alternatives + (mapping,) # but note that the constraint can still be applied self.conforms(mapping, ("name", 123)) self.conforms(mapping, ("name", "key")) self.conforms(mapping, ("name", ("key", "value"))) self.conforms(mapping, ("name", ("key", 123))) self.violates(mapping, ("name", ("key", []))) l = [] l.append(l) self.violates(mapping, ("name", l)) def testList(self): l = schema.ListOf(schema.ByteStringConstraint(10)) self.conforms(l, ["one", "two", "three"]) self.violates(l, ("can't", "fool", "me")) self.violates(l, ["but", "perspicacity", "is too long"]) self.violates(l, [0, "numbers", "allowed"]) self.conforms(l, ["short", "sweet"]) l2 = schema.ListOf(schema.ByteStringConstraint(10), 3) self.conforms(l2, ["the number", "shall be", "three"]) self.violates(l2, ["five", "is", "...", "right", "out"]) l3 = schema.ListOf(schema.ByteStringConstraint(10), None) self.conforms(l3, ["long"] * 35) self.violates(l3, ["number", 1, "rule", "is", 0, "numbers"]) l4 = schema.ListOf(schema.ByteStringConstraint(10), 3, 3) self.conforms(l4, ["three", "is", "good"]) self.violates(l4, ["but", "four", "is", "bad"]) self.violates(l4, ["two", "too"]) def testSet(self): l = schema.SetOf(schema.IntegerConstraint(), 3) self.conforms(l, set([])) self.conforms(l, set([1])) self.conforms(l, set([1,2,3])) self.violates(l, set([1,2,3,4])) self.violates(l, set(["not a number"])) self.conforms(l, frozenset([])) self.conforms(l, frozenset([1])) self.conforms(l, frozenset([1,2,3])) self.violates(l, frozenset([1,2,3,4])) self.violates(l, frozenset(["not a number"])) l = schema.SetOf(schema.IntegerConstraint(), 3, True) self.conforms(l, set([])) self.conforms(l, set([1])) self.conforms(l, set([1,2,3])) self.violates(l, set([1,2,3,4])) self.violates(l, set(["not a number"])) self.violates(l, frozenset([])) self.violates(l, frozenset([1])) self.violates(l, frozenset([1,2,3])) self.violates(l, frozenset([1,2,3,4])) self.violates(l, frozenset(["not a number"])) l = schema.SetOf(schema.IntegerConstraint(), 3, False) self.violates(l, set([])) self.violates(l, set([1])) self.violates(l, set([1,2,3])) self.violates(l, set([1,2,3,4])) self.violates(l, set(["not a number"])) self.conforms(l, frozenset([])) self.conforms(l, frozenset([1])) self.conforms(l, frozenset([1,2,3])) self.violates(l, frozenset([1,2,3,4])) self.violates(l, frozenset(["not a number"])) def testDict(self): d = schema.DictOf(schema.ByteStringConstraint(10), schema.IntegerConstraint(), maxKeys=4) self.conforms(d, {"a": 1, "b": 2}) self.conforms(d, {"foo": 123, "bar": 345, "blah": 456, "yar": 789}) self.violates(d, None) self.violates(d, 12) self.violates(d, ["nope"]) self.violates(d, ("nice", "try")) self.violates(d, {1:2, 3:4}) self.violates(d, {"a": "b"}) self.violates(d, {"a": 1, "b": 2, "c": 3, "d": 4, "toomuch": 5}) def testAttrDict(self): d = copyable.AttributeDictConstraint(('a', int), ('b', str)) self.conforms(d, {"a": 1, "b": "string"}) self.violates(d, {"a": 1, "b": 2}) self.violates(d, {"a": 1, "b": "string", "c": "is a crowd"}) d = copyable.AttributeDictConstraint(('a', int), ('b', str), ignoreUnknown=True) self.conforms(d, {"a": 1, "b": "string"}) self.violates(d, {"a": 1, "b": 2}) self.conforms(d, {"a": 1, "b": "string", "c": "is a crowd"}) d = copyable.AttributeDictConstraint(attributes={"a": int, "b": str}) self.conforms(d, {"a": 1, "b": "string"}) self.violates(d, {"a": 1, "b": 2}) self.violates(d, {"a": 1, "b": "string", "c": "is a crowd"}) class CreateTest(unittest.TestCase): def check(self, obj, expected): self.failUnless(isinstance(obj, expected)) def testMakeConstraint(self): make = IConstraint c = make(int) self.check(c, schema.IntegerConstraint) self.failUnlessEqual(c.maxBytes, -1) c = make(str) self.check(c, schema.ByteStringConstraint) self.failUnlessEqual(c.maxLength, None) c = make(schema.ByteStringConstraint(2000)) self.check(c, schema.ByteStringConstraint) self.failUnlessEqual(c.maxLength, 2000) c = make(unicode) self.check(c, schema.UnicodeConstraint) self.failUnlessEqual(c.maxLength, None) self.check(make(bool), schema.BooleanConstraint) self.check(make(float), schema.NumberConstraint) self.check(make(schema.NumberConstraint()), schema.NumberConstraint) c = make((int, str)) self.check(c, schema.TupleConstraint) self.check(c.constraints[0], schema.IntegerConstraint) self.check(c.constraints[1], schema.ByteStringConstraint) c = make(common.RIHelper) self.check(c, RemoteInterfaceConstraint) self.failUnlessEqual(c.interface, common.RIHelper) c = make(common.IFoo) self.check(c, LocalInterfaceConstraint) self.failUnlessEqual(c.interface, common.IFoo) c = make(Referenceable) self.check(c, RemoteInterfaceConstraint) self.failUnlessEqual(c.interface, None) class Arguments(unittest.TestCase): def test_arguments(self): def foo(a=int, b=bool, c=int): return str r = RemoteMethodSchema(method=foo) getpos = r.getPositionalArgConstraint getkw = r.getKeywordArgConstraint self.failUnless(isinstance(getpos(0)[1], schema.IntegerConstraint)) self.failUnless(isinstance(getpos(1)[1], schema.BooleanConstraint)) self.failUnless(isinstance(getpos(2)[1], schema.IntegerConstraint)) self.failUnless(isinstance(getkw("a")[1], schema.IntegerConstraint)) self.failUnless(isinstance(getkw("b")[1], schema.BooleanConstraint)) self.failUnless(isinstance(getkw("c")[1], schema.IntegerConstraint)) self.failUnless(isinstance(r.getResponseConstraint(), schema.ByteStringConstraint)) self.failUnless(isinstance(getkw("c", 1, [])[1], schema.IntegerConstraint)) self.failUnlessRaises(schema.Violation, getkw, "a", 1, []) self.failUnlessRaises(schema.Violation, getkw, "b", 1, ["b"]) self.failUnlessRaises(schema.Violation, getkw, "a", 2, []) self.failUnless(isinstance(getkw("c", 2, [])[1], schema.IntegerConstraint)) self.failUnless(isinstance(getkw("c", 0, ["a", "b"])[1], schema.IntegerConstraint)) try: r.checkAllArgs((1,True,2), {}, False) r.checkAllArgs((), {"a":1, "b":False, "c":2}, False) r.checkAllArgs((1,), {"b":False, "c":2}, False) r.checkAllArgs((1,True), {"c":3}, False) r.checkResults("good", False) except schema.Violation: self.fail("that shouldn't have raised a Violation") self.failUnlessRaises(schema.Violation, # 2 is not bool r.checkAllArgs, (1,2,3), {}, False) self.failUnlessRaises(schema.Violation, # too many r.checkAllArgs, (1,True,3,4), {}, False) self.failUnlessRaises(schema.Violation, # double "a" r.checkAllArgs, (1,), {"a":1, "b":True, "c": 3}, False) self.failUnlessRaises(schema.Violation, # missing required "b" r.checkAllArgs, (1,), {"c": 3}, False) self.failUnlessRaises(schema.Violation, # missing required "a" r.checkAllArgs, (), {"b":True, "c": 3}, False) self.failUnlessRaises(schema.Violation, r.checkResults, 12, False) def test_bad_arguments(self): def foo(nodefault): return str self.failUnlessRaises(InvalidRemoteInterface, RemoteMethodSchema, method=foo) def bar(nodefault, a=int): return str self.failUnlessRaises(InvalidRemoteInterface, RemoteMethodSchema, method=bar) class Interfaces(unittest.TestCase): def check_inbound(self, obj, constraint): try: constraint.checkObject(obj, True) except Violation, f: self.fail("constraint was violated: %s" % f) def check_outbound(self, obj, constraint): try: constraint.checkObject(obj, False) except Violation, f: self.fail("constraint was violated: %s" % f) def violates_inbound(self, obj, constraint): try: constraint.checkObject(obj, True) except Violation: return self.fail("constraint wasn't violated") def violates_outbound(self, obj, constraint): try: constraint.checkObject(obj, False) except Violation: return self.fail("constraint wasn't violated") def test_referenceable(self): h = common.HelperTarget() c1 = RemoteInterfaceConstraint(common.RIHelper) c2 = RemoteInterfaceConstraint(common.RIMyTarget) self.violates_inbound("bogus", c1) self.violates_outbound("bogus", c1) self.check_outbound(h, c1) self.violates_inbound(h, c1) self.violates_inbound(h, c2) self.violates_outbound(h, c2) def test_remotereference(self): # we need to create a fake RemoteReference here tracker = RemoteReferenceTracker(broker.Broker(TubRef("fake-tubid")), 0, None, common.RIHelper.__remote_name__) rr = RemoteReference(tracker) c1 = RemoteInterfaceConstraint(common.RIHelper) self.check_inbound(rr, c1) self.check_outbound(rr, c1) # gift c2 = RemoteInterfaceConstraint(common.RIMyTarget) self.violates_inbound(rr, c2) self.violates_outbound(rr, c2) foolscap-0.13.1/src/foolscap/test/test_serialize.py0000644000076500000240000001174312766553111023014 0ustar warnerstaff00000000000000# -*- test-case-name: foolscap.test.test_serialize -*- from twisted.trial import unittest from twisted.application import service from cStringIO import StringIO import gc from foolscap.api import Referenceable, Copyable, RemoteCopy, \ flushEventualQueue, serialize, unserialize, Tub from foolscap.referenceable import RemoteReference from foolscap.tokens import Violation from foolscap.util import allocate_tcp_port from foolscap.test.common import ShouldFailMixin class Foo: # instances of non-Copyable classes are not serializable pass class Bar(Copyable, RemoteCopy): # but if they're Copyable, they're ok typeToCopy = "bar" copytype = "bar" pass class Serialize(unittest.TestCase, ShouldFailMixin): def setUp(self): self.s = service.MultiService() self.s.startService() def tearDown(self): d = self.s.stopService() d.addCallback(flushEventualQueue) return d def NOT_test_data_synchronous(self): obj = ["look at the pretty graph", 3, True] obj.append(obj) # and look at the pretty cycle data = serialize(obj) obj2 = unserialize(data) self.failUnlessEqual(obj2[1], 3) self.failUnlessIdentical(obj2[3], obj2) def test_data(self): obj = ["simple graph", 3, True] d = serialize(obj) d.addCallback(lambda data: unserialize(data)) def _check(obj2): self.failUnlessEqual(obj2[1], 3) d.addCallback(_check) return d def test_cycle(self): obj = ["look at the pretty graph", 3, True] obj.append(obj) # and look at the pretty cycle d = serialize(obj) d.addCallback(lambda data: unserialize(data)) def _check(obj2): self.failUnlessEqual(obj2[1], 3) self.failUnlessIdentical(obj2[3], obj2) d.addCallback(_check) return d def test_copyable(self): obj = ["fire pretty", Bar()] d = serialize(obj) d.addCallback(lambda data: unserialize(data)) def _check(obj2): self.failUnless(isinstance(obj2[1], Bar)) self.failIfIdentical(obj[1], obj2[1]) d.addCallback(_check) return d def test_data_outstream(self): obj = ["look at the pretty graph", 3, True] obj.append(obj) # and look at the pretty cycle b = StringIO() d = serialize(obj, outstream=b) def _out(res): self.failUnlessIdentical(res, b) return b.getvalue() d.addCallback(_out) d.addCallback(lambda data: unserialize(data)) def _check(obj2): self.failUnlessEqual(obj2[1], 3) self.failUnlessIdentical(obj2[3], obj2) d.addCallback(_check) return d def test_unhandled_objects(self): obj1 = [1, Referenceable()] d = self.shouldFail(Violation, "1", "This object can only be serialized by a broker", serialize, obj1) obj2 = [1, Foo()] d.addCallback(lambda ign: self.shouldFail(Violation, "2", "cannot serialize = 1 and len(barry_connections) >= 1: return True return False d = self.poll(_check) def _validate(res): self.failUnless(isinstance(bill_connections[0], RemoteReference)) self.failUnless(isinstance(barry_connections[0], RemoteReference)) self.failIf(bill_connections[0] == barry_connections[0]) d.addCallback(_validate) self.services.append(t1) eventually(t1.startService) return d class NameLookup(TargetMixin, MakeTubsMixin, unittest.TestCase): # test registerNameLookupHandler def setUp(self): TargetMixin.setUp(self) self.tubA, self.tubB = self.makeTubs(2) self.url_on_b = self.tubB.registerReference(Referenceable()) self.lookups = [] self.lookups2 = [] self.names = {} self.names2 = {} def tearDown(self): d = TargetMixin.tearDown(self) def _more(res): return defer.DeferredList([s.stopService() for s in self.services]) d.addCallback(_more) d.addCallback(flushEventualQueue) return d def lookup(self, name): self.lookups.append(name) return self.names.get(name, None) def lookup2(self, name): self.lookups2.append(name) return self.names2.get(name, None) def testNameLookup(self): t1 = HelperTarget() t2 = HelperTarget() self.names["foo"] = t1 self.names2["bar"] = t2 self.names2["baz"] = t2 self.tubB.registerNameLookupHandler(self.lookup) self.tubB.registerNameLookupHandler(self.lookup2) # hack up a new furl pointing at the same tub but with a name that # hasn't been registered. s = SturdyRef(self.url_on_b) s.name = "foo" d = self.tubA.getReference(s) def _check(res): self.failUnless(isinstance(res, RemoteReference)) self.failUnlessEqual(self.lookups, ["foo"]) # the first lookup should short-circuit the process self.failUnlessEqual(self.lookups2, []) self.lookups = []; self.lookups2 = [] s.name = "bar" return self.tubA.getReference(s) d.addCallback(_check) def _check2(res): self.failUnless(isinstance(res, RemoteReference)) # if the first lookup fails, the second handler should be asked self.failUnlessEqual(self.lookups, ["bar"]) self.failUnlessEqual(self.lookups2, ["bar"]) self.lookups = []; self.lookups2 = [] # make sure that loopbacks use this too return self.tubB.getReference(s) d.addCallback(_check2) def _check3(res): self.failUnless(isinstance(res, RemoteReference)) self.failUnlessEqual(self.lookups, ["bar"]) self.failUnlessEqual(self.lookups2, ["bar"]) self.lookups = []; self.lookups2 = [] # and make sure we can de-register handlers self.tubB.unregisterNameLookupHandler(self.lookup) s.name = "baz" return self.tubA.getReference(s) d.addCallback(_check3) def _check4(res): self.failUnless(isinstance(res, RemoteReference)) self.failUnlessEqual(self.lookups, []) self.failUnlessEqual(self.lookups2, ["baz"]) self.lookups = []; self.lookups2 = [] d.addCallback(_check4) return d class Shutdown(unittest.TestCase, ShouldFailMixin): def test_doublestop(self): tub = Tub() tub.startService() d = tub.stopService() d.addCallback(lambda res: self.shouldFail(RuntimeError, "test_doublestop_startService", "Sorry, but Tubs cannot be restarted", tub.startService)) d.addCallback(lambda res: self.shouldFail(RuntimeError, "test_doublestop_getReference", "Sorry, but this Tub has been shut down", tub.getReference, "furl")) d.addCallback(lambda res: self.shouldFail(RuntimeError, "test_doublestop_connectTo", "Sorry, but this Tub has been shut down", tub.connectTo, "furl", None)) return d def test_wait_for_brokers(self): """ The L{Deferred} returned by L{Tub.stopService} fires only after the L{Broker} connections belonging to the L{Tub} have disconnected. """ tub = Tub() tub.startService() another_tub = Tub() another_tub.startService() brokers = list(tub.brokerClass(None) for i in range(3)) for n, b in enumerate(brokers): b.makeConnection(StringTransport()) ref = SturdyRef(encode_furl(another_tub.tubID, [], str(n))) tub.brokerAttached(ref, b, isClient=(n % 2)==1) stopping = tub.stopService() d = flushEventualQueue() def event(ignored): self.assertNoResult(stopping) for b in brokers: b.connectionLost(failure.Failure(Exception("Connection lost"))) return flushEventualQueue() d.addCallback(event) def connectionsLost(ignored): self.successResultOf(stopping) d.addCallback(connectionsLost) return d class Receiver(Referenceable): def __init__(self, tub): self.tub = tub self.done_d = defer.Deferred() def remote_one(self): d = self.tub.stopService() d.addBoth(lambda r: fireEventually(r)) d.addBoth(self.done_d.callback) def remote_two(self): msg = "Receiver.remote_two: I shouldn't be called" print msg f = failure.Failure(ValueError(msg)) log.err(f) class CancelPendingDeliveries(StallMixin, MakeTubsMixin, unittest.TestCase): def setUp(self): self.tubA, self.tubB = self.makeTubs(2) def tearDown(self): dl = [defer.succeed(None)] if self.tubA.running: dl.append(defer.maybeDeferred(self.tubA.stopService)) if self.tubB.running: dl.append(defer.maybeDeferred(self.tubB.stopService)) d = defer.DeferredList(dl) d.addCallback(flushEventualQueue) return d def test_cancel_pending_deliveries(self): # when a Tub is stopped, any deliveries that were pending should be # discarded. TubA sends remote_one+remote_two (and we hope they # arrive in the same chunk). TubB responds to remote_one by shutting # down. remote_two should be discarded. The bug was that remote_two # would cause an unhandled error on the TubB side. r = Receiver(self.tubB) furl = self.tubB.registerReference(r) d = self.tubA.getReference(furl) def _go(rref): # we want these two to get sent and received in the same hunk rref.callRemoteOnly("one") rref.callRemoteOnly("two") return r.done_d d.addCallback(_go) # let remote_two do its log.err before we move on to the next test d.addCallback(self.stall, 1.0) return d class BadLocationFURL(unittest.TestCase): def setUp(self): self.s = service.MultiService() self.s.startService() def tearDown(self): d = self.s.stopService() d.addCallback(flushEventualQueue) return d def test_empty_location(self): # bug #129: a FURL with no location hints causes a synchronous # exception in Tub.getReference(), instead of an errback'ed Deferred. tubA = Tub() tubA.setServiceParent(self.s) tubB = Tub() tubB.setServiceParent(self.s) # This is a hack to get a FURL with empty location hints. The correct # way to make a Tub unreachable is to not call .setLocation() at all. tubB.setLocation("") r = Receiver(tubB) furl = tubB.registerReference(r) # the buggy behavior is that the following call raises an exception d = tubA.getReference(furl) # whereas it ought to return a Deferred self.failUnless(isinstance(d, defer.Deferred)) def _check(f): self.failUnless(isinstance(f, failure.Failure), f) self.failUnless(f.check(NoLocationHintsError), f) d.addBoth(_check) return d def test_future(self): tubA = Tub() tubA.setServiceParent(self.s) tubB = Tub() tubB.setServiceParent(self.s) # "future:stuff" is interpreted as a "location hint format from the # future", which we're supposed to ignore, and are thus left with no # hints tubB.setLocation("future:stuff") r = Receiver(tubB) furl = tubB.registerReference(r) # the buggy behavior is that the following call raises an exception d = tubA.getReference(furl) # whereas it ought to return a Deferred self.failUnless(isinstance(d, defer.Deferred)) def _check(f): self.failUnless(isinstance(f, failure.Failure), f) self.failUnless(f.check(NoLocationHintsError), f) d.addBoth(_check) return d foolscap-0.13.1/src/foolscap/test/test_unreachable.py0000644000076500000240000001032412766553111023270 0ustar warnerstaff00000000000000import os from twisted.trial import unittest from twisted.application import service from foolscap.api import Tub, Referenceable from foolscap.tokens import NoLocationError, NoLocationHintsError from foolscap.util import allocate_tcp_port from foolscap.eventual import flushEventualQueue from foolscap.test.common import ShouldFailMixin class Receiver(Referenceable): def __init__(self): self.obj = None def remote_call(self, obj): self.obj = obj return 1 def remote_gift_me(self): return self.obj class References(ShouldFailMixin, unittest.TestCase): def setUp(self): self.s = service.MultiService() self.s.startService() def tearDown(self): d = self.s.stopService() d.addCallback(flushEventualQueue) return d def test_unreachable_client(self): # A "client-only" Tub has no location set. It should still be # possible to connect to objects in other (location-bearing) server # Tubs, and objects in the client Tub can still be sent to (and used # by) the server Tub. client_tub = Tub() client_tub.setServiceParent(self.s) server_tub = Tub() server_tub.setServiceParent(self.s) portnum = allocate_tcp_port() server_tub.listenOn("tcp:%d:interface=127.0.0.1" % portnum) server_tub.setLocation("tcp:127.0.0.1:%d" % portnum) s = Receiver() # no FURL, not directly reachable r = Receiver() furl = server_tub.registerReference(r) d = client_tub.getReference(furl) d.addCallback(lambda rref: rref.callRemote("call", s)) d.addCallback(lambda res: self.failUnlessEqual(res, 1)) d.addCallback(lambda _: self.failIfEqual(r.obj, None)) def _inspect_obj(_): self.failUnlessEqual(r.obj.getSturdyRef().getURL(), None) d.addCallback(_inspect_obj) d.addCallback(lambda _: r.obj.callRemote("call", 2)) d.addCallback(lambda _: self.failUnlessEqual(s.obj, 2)) return d def test_unreachable_gift(self): client_tub = Tub() client_tub.setServiceParent(self.s) server_tub = Tub() server_tub.setServiceParent(self.s) recipient_tub = Tub() recipient_tub.setServiceParent(self.s) portnum = allocate_tcp_port() server_tub.listenOn("tcp:%d:interface=127.0.0.1" % portnum) server_tub.setLocation("tcp:127.0.0.1:%d" % portnum) s = Receiver() # no FURL, not directly reachable r = Receiver() furl = server_tub.registerReference(r) d = client_tub.getReference(furl) d.addCallback(lambda rref: rref.callRemote("call", s)) d.addCallback(lambda res: self.failUnlessEqual(res, 1)) d.addCallback(lambda _: recipient_tub.getReference(furl)) # when server_tub tries to send the lame 's' rref to recipient_tub, # the RemoteReferenceTracker won't have a FURL, so it will be # serialized as a (their-reference furl="") sequence. Then # recipient_tub will try to resolve it, and will throw a # NoLocationHintsError. It might be more natural to send # (their-reference furl=None), but the constraint schema on # their-references forbids non-strings. It might also seem # appropriate to raise a Violation (i.e. server_tub is bad for trying # to send it, rather than foisting the problem off to recipient_tub), # but that causes the connection explode and fall out of sync. d.addCallback(lambda rref: self.shouldFail(NoLocationHintsError, "gift_me", None, rref.callRemote, "gift_me")) return d def test_logport_furlfile1(self): basedir = "unreachable/References/logport_furlfile1" os.makedirs(basedir) furlfile = os.path.join(basedir, "logport.furl") t = Tub() # setOption before setServiceParent t.setOption("logport-furlfile", furlfile) t.setServiceParent(self.s) self.failUnlessRaises(NoLocationError, t.getLogPort) self.failUnlessRaises(NoLocationError, t.getLogPortFURL) self.failIf(os.path.exists(furlfile)) # without .setLocation, the furlfile will never be created foolscap-0.13.1/src/foolscap/test/test_util.py0000644000076500000240000000753212766553111022003 0ustar warnerstaff00000000000000 from twisted.trial import unittest from twisted.internet import reactor, defer, protocol, endpoints from twisted.python import failure from foolscap import util, eventual, base32 class AsyncAND(unittest.TestCase): def setUp(self): self.fired = False self.failed = False def callback(self, res): self.fired = True def errback(self, res): self.failed = True def attach(self, d): d.addCallbacks(self.callback, self.errback) return d def shouldNotFire(self, ignored=None): self.failIf(self.fired) self.failIf(self.failed) def shouldFire(self, ignored=None): self.failUnless(self.fired) self.failIf(self.failed) def shouldFail(self, ignored=None): self.failUnless(self.failed) self.failIf(self.fired) def tearDown(self): return eventual.flushEventualQueue() def test_empty(self): self.attach(util.AsyncAND([])) self.shouldFire() def test_simple(self): d1 = eventual.fireEventually(None) a = util.AsyncAND([d1]) self.attach(a) a.addBoth(self.shouldFire) return a def test_two(self): d1 = defer.Deferred() d2 = defer.Deferred() self.attach(util.AsyncAND([d1, d2])) self.shouldNotFire() d1.callback(1) self.shouldNotFire() d2.callback(2) self.shouldFire() def test_one_failure_1(self): d1 = defer.Deferred() d2 = defer.Deferred() self.attach(util.AsyncAND([d1, d2])) self.shouldNotFire() d1.callback(1) self.shouldNotFire() d2.errback(RuntimeError()) self.shouldFail() def test_one_failure_2(self): d1 = defer.Deferred() d2 = defer.Deferred() self.attach(util.AsyncAND([d1, d2])) self.shouldNotFire() d1.errback(RuntimeError()) self.shouldFail() d2.callback(1) self.shouldFail() def test_two_failure(self): d1 = defer.Deferred() d2 = defer.Deferred() self.attach(util.AsyncAND([d1, d2])) def _should_fire(res): self.failIf(isinstance(res, failure.Failure)) def _should_fail(f): self.failUnless(isinstance(f, failure.Failure)) d1.addBoth(_should_fire) d2.addBoth(_should_fail) self.shouldNotFire() d1.errback(RuntimeError()) self.shouldFail() d2.errback(RuntimeError()) self.shouldFail() class Base32(unittest.TestCase): def test_is_base32(self): self.failUnless(base32.is_base32("abc456")) self.failUnless(base32.is_base32("456")) self.failUnless(base32.is_base32("")) self.failIf(base32.is_base32("123")) # 1 is not in rfc4648 base32 self.failIf(base32.is_base32(".123")) self.failIf(base32.is_base32("_")) self.failIf(base32.is_base32("a b c")) class Time(unittest.TestCase): def test_format(self): when = 1339286175.7071271 self.failUnlessEqual(util.format_time(when, "utc"), "2012-06-09_23:56:15.707127Z") self.failUnlessEqual(util.format_time(when, "epoch"), "1339286175.707") self.failUnless(":" in util.format_time(when, "short-local")) self.failUnless(":" in util.format_time(when, "long-local")) class AllocatePort(unittest.TestCase): def test_allocate(self): p = util.allocate_tcp_port() self.failUnless(isinstance(p, int)) self.failUnless(1 <= p <= 65535, p) # the allocation function should release the port before it # returns, so it should be possible to listen on it immediately ep = endpoints.TCP4ServerEndpoint(reactor, p, interface="127.0.0.1") d = ep.listen(protocol.Factory()) d.addCallback(lambda port: port.stopListening()) return d foolscap-0.13.1/src/foolscap/tokens.py0000644000076500000240000004105112766553111020305 0ustar warnerstaff00000000000000 from twisted.python.failure import Failure from zope.interface import Attribute, Interface # delimiter characters. LIST = chr(0x80) # old INT = chr(0x81) STRING = chr(0x82) NEG = chr(0x83) FLOAT = chr(0x84) # "optional" -- these might be refused by a low-level implementation. LONGINT = chr(0x85) # old LONGNEG = chr(0x86) # old # really optional; this is is part of the 'pb' vocabulary VOCAB = chr(0x87) # newbanana tokens OPEN = chr(0x88) CLOSE = chr(0x89) ABORT = chr(0x8A) ERROR = chr(0x8D) PING = chr(0x8E) PONG = chr(0x8F) tokenNames = { LIST: "LIST", INT: "INT", STRING: "STRING", NEG: "NEG", FLOAT: "FLOAT", LONGINT: "LONGINT", LONGNEG: "LONGNEG", VOCAB: "VOCAB", OPEN: "OPEN", CLOSE: "CLOSE", ABORT: "ABORT", ERROR: "ERROR", PING: "PING", PONG: "PONG", } SIZE_LIMIT = 1000 # default limit on the body length of long tokens (STRING, # LONGINT, LONGNEG, ERROR) class InvalidRemoteInterface(Exception): pass class UnknownSchemaType(Exception): pass class Violation(Exception): """This exception is raised in response to a schema violation. It indicates that the incoming token stream has violated a constraint imposed by the recipient. The current Unslicer is abandoned and the error is propagated upwards to the enclosing Unslicer parent by providing an BananaFailure object to the parent's .receiveChild method. All remaining tokens for the current Unslicer are to be dropped. """ """.where: this string describes which node of the object graph was being handled when the exception took place.""" where = "" def setLocation(self, where): self.where = where def getLocation(self): return self.where def prependLocation(self, prefix): if self.where: self.where = prefix + " " + self.where else: self.where = prefix def appendLocation(self, suffix): if self.where: self.where = self.where + " " + suffix else: self.where = suffix def __str__(self): if self.where: return "Violation (%s): %s" % (self.where, self.args) else: return "Violation: %s" % (self.args,) class RemoteException(Exception): """When the Tub is in expose-remote-exception-types=False mode, this exception is raised in response to any remote exception. It wraps a CopiedFailure, which can be examined by callers who want to know more than the fact that something failed on the remote end.""" def __init__(self, failure): self.failure = failure def __str__(self): return "" % str(self.failure) class BananaError(Exception): """This exception is raised in response to a fundamental protocol violation. The connection should be dropped immediately. .where is an optional string that describes the node of the object graph where the failure was noticed. """ where = None def __str__(self): if self.where: return "BananaError(in %s): %s" % (self.where, self.args) else: return "BananaError: %s" % (self.args,) class NegotiationError(Exception): pass class DuplicateConnection(NegotiationError): pass class RemoteNegotiationError(Exception): """The other end hung up on us because they had a NegotiationError on their side.""" pass class PBError(Exception): pass class BananaFailure(Failure): """This is a marker subclass of Failure, to let Unslicer.receiveChild distinguish between an unserialized Failure instance and a a failure in a child Unslicer""" pass class WrongTubIdError(Exception): """getReference(furlFile=) used a FURL with a different TubID""" class WrongNameError(Exception): """getReference(furlFule=) used a FURL with a different name""" class NoLocationError(Exception): """This Tub has no location set, so we cannot make references to it.""" class NoLocationHintsError(Exception): """We cannot make a connection without some location hints""" class ISlicer(Interface): """I know how to slice objects into tokens.""" sendOpen = Attribute(\ """True if an OPEN/CLOSE token pair should be sent around the Slicer's body tokens. Only special-purpose Slicers (like the RootSlicer) should use False. """) trackReferences = Attribute(\ """True if the object we slice is referenceable: i.e. it is useful or necessary to send multiple copies as a single instance and a bunch of References, rather than as separate copies. Instances are referenceable, as are mutable containers like lists.""") streamable = Attribute(\ """True if children of this object are allowed to use Deferreds to stall production of new tokens. This must be set in slice() before yielding each child object, and affects that child and all descendants. Streaming is only allowed if the parent also allows streaming: if slice() is called with streamable=False, then self.streamable must be False too. It can be changed from within the slice() generator at any time as long as this restriction is obeyed. This attribute is read when each child Slicer is started.""") def slice(streamable, banana): """Return an iterator which provides Index Tokens and the Body Tokens of the object's serialized form. This is frequently implemented with a generator (i.e. 'yield' appears in the body of this function). Do not yield the OPEN or the CLOSE token, those will be handled elsewhere. If a Violation exception is raised, slicing will cease. An ABORT token followed by a CLOSE token will be emitted. If 'streamable' is True, the iterator may yield a Deferred to indicate that slicing should wait until the Deferred is fired. If the Deferred is errbacked, the connection will be dropped. TODO: it should be possible to errback with a Violation.""" def registerRefID(refid, obj): """Register the relationship between 'refid' (a number taken from the cumulative count of OPEN tokens sent over our connection: 0 is the object described by the very first OPEN sent over the wire) and the object. If the object is sent a second time, a Reference may be used in its place. Slicers usually delgate this function upwards to the RootSlicer, but it can be handled at any level to allow local scoping of references (they might only be valid within a single RPC invocation, for example). This method is *not* allowed to raise a Violation, as that will mess up the transmit logic. If it raises any other exception, the connection will be dropped.""" def childAborted(f): """Notify the Slicer that one of its child slicers (as produced by its .slice iterator) has caused an error. If the slicer got started, it has now emitted an ABORT token and terminated its token stream. If it did not get started (usually because the child object was unserializable), there has not yet been any trace of the object in the token stream. The corresponding Unslicer (receiving this token stream) will get an BananaFailure and is likely to ignore any remaining tokens from us, so it may be reasonable for the parent Slicer to give up as well. If the Slicer wishes to abandon their own sequence, it should simply return the failure object passed in. If it wants to absorb the error, it should return None.""" def slicerForObject(obj): """Get a new Slicer for some child object. Slicers usually delegate this method up to the RootSlicer. References are handled by producing a ReferenceSlicer here. These references can have various scopes. If something on the stack does not want the object to be sent, it can raise a Violation exception. This is the 'taster' function.""" def describe(): """Return a short string describing where in the object tree this slicer is sitting, relative to its parent. These strings are obtained from every slicer in the stack, and joined to describe where any problems occurred.""" class IRootSlicer(Interface): def allowStreaming(streamable): """Specify whether or not child Slicers will be allowed to stream.""" def connectionLost(why): """Called when the transport is closed. The RootSlicer may choose to abandon objects being sent here.""" class IUnslicer(Interface): # .parent # start/receiveChild/receiveClose/finish are # the main "here are some tokens, make an object out of them" entry # points used by Unbanana. # start/receiveChild can call self.protocol.abandonUnslicer(failure, # self) to tell the protocol that the unslicer has given up on life and # all its remaining tokens should be discarded. The failure will be # given to the late unslicer's parent in lieu of the object normally # returned by receiveClose. # start/receiveChild/receiveClose/finish may raise a Violation # exception, which tells the protocol that this object is contaminated # and should be abandoned. An BananaFailure will be passed to its # parent. # Note, however, that it is not valid to both call abandonUnslicer *and* # raise a Violation. That would discard too much. def setConstraint(constraint): """Add a constraint for this unslicer. The unslicer will enforce this constraint upon all incoming data. The constraint must be of an appropriate type (a ListUnslicer will only accept a ListConstraint, etc.). It must not be None. To leave us unconstrained, do not call this method. If this method is not called, the Unslicer will accept any valid banana as input, which probably means there is no limit on the number of bytes it will accept (and therefore on the memory it could be made to consume) before it finally accepts or rejects the input. """ def start(count): """Called to initialize the new slice. The 'count' argument is the reference id: if this object might be shared (and therefore the target of a 'reference' token), it should call self.protocol.setObject(count, obj) with the object being created. If this object is not available yet (tuples), it should save a Deferred there instead. """ def checkToken(typebyte, size): """Check to see if the given token is acceptable (does it conform to the constraint?). It will not be asked about ABORT or CLOSE tokens, but it *will* be asked about OPEN. It should enfore a length limit for long tokens (STRING and LONGINT/LONGNEG types). If STRING is acceptable, then VOCAB should be too. It should return None if the token and the size are acceptable. Should raise Violation if the schema indiates the token is not acceptable. Should raise BananaError if the type byte violates the basic Banana protocol. (if no schema is in effect, this should never raise Violation, but might still raise BananaError). """ def openerCheckToken(typebyte, size, opentype): """'typebyte' is the type of an incoming index token. 'size' is the value of header associated with this typebyte. 'opentype' is a list of open tokens that we've received so far, not including the one that this token hopes to create. This method should ask the current opener if this index token is acceptable, and is used in lieu of checkToken() when the receiver is in the index phase. Usually implemented by calling self.opener.openerCheckToken, thus delegating the question to the RootUnslicer. """ def doOpen(opentype): """opentype is a tuple. Return None if more index tokens are required. Check to see if this kind of child object conforms to the constraint, raise Violation if not. Create a new Unslicer (usually by delegating to self.parent.doOpen, up to the RootUnslicer). Set a constraint on the child unslicer, if any. """ def receiveChild(childobject, ready_deferred): """'childobject' is being handed to this unslicer. It may be a primitive type (number or string), or a composite type produced by another Unslicer. It might also be a Deferred, which indicates that the actual object is not ready (perhaps a tuple with an element that is not yet referenceable), in which case you should add a callback to it that will fill in the appropriate object later. This callback is required to return the object when it is done, so multiple such callbacks can be chained. The childobject/ready_deferred argument pair is taken directly from the output of receiveClose(). If ready_deferred is non-None, you should return a dependent Deferred from your own receiveClose method.""" def reportViolation(bf): """You have received an error instead of a child object. If you wish to give up and propagate the error upwards, return the BananaFailure object you were just given. To absorb the error and keep going with your sequence, return None.""" def receiveClose(): """Called when the Close token is received. Returns a tuple of (object/referenceable-deferred, complete-deferred), or an BananaFailure if something went wrong. There are four potential cases:: (obj, None): the object is complete and ready to go (d1, None): the object cannot be referenced yet, probably because it is an immutable container, and one of its children cannot be referenced yet. The deferred will fire by the time the cycle has been fully deserialized, with the object as its argument. (obj, d2): the object can be referenced, but it is not yet complete, probably because some component of it is 'slow' (see below). The Deferred will fire (with an argument of None) when the object is ready to be used. It is not guaranteed to fire by the time the enclosing top-level object has finished deserializing. (d1, d2): the object cannot yet be referenced, and even if it could be, it would not yet be ready for use. Any potential users should wait until both deferreds fire before using it. The first deferred (d1) is guaranteed to fire before the top-most enclosing object (a CallUnslicer, for PB methods) is closed. (if it does not fire, that indicates a broken cycle). It is present to handle cycles that include immutable containers, like tuples. Mutable containers *must* return a reference to an object (even if it is not yet ready to be used, because it contains placeholders to tuples that have not yet been created), otherwise those cycles cannot be broken and the object graph will not reconstructable. The second (d2) has no such guarantees about when it will fire. It indicates a dependence upon 'slow' external events. The first use case for such 'slow' objects is a globally-referenceable object which requires a new Broker connection before it can be used, so the Deferred will not fire until a TCP connection has been established and the first stages of PB negotiation have been completed. If necessary, unbanana.setObject should be called, then the Deferred created in start() should be fired with the new object.""" def finish(): """Called when the unslicer is popped off the stack. This is called even if the pop is because of an exception. The unslicer should perform cleanup, including firing the Deferred with an BananaFailure if the object it is creating could not be created. TODO: can receiveClose and finish be merged? Or should the child object be returned from finish() instead of receiveClose? """ def describe(): """Return a short string describing where in the object tree this unslicer is sitting, relative to its parent. These strings are obtained from every unslicer in the stack, and joined to describe where any problems occurred.""" def where(): """This returns a string that describes the location of this unslicer, starting at the root of the object tree.""" foolscap-0.13.1/src/foolscap/util.py0000644000076500000240000002014313204160675017753 0ustar warnerstaff00000000000000import os, sys import socket import time from twisted.internet import defer, reactor, protocol from twisted.python.runtime import platformType class AsyncAND(defer.Deferred): """Like DeferredList, but results are discarded and failures handled in a more convenient fashion. Create me with a list of Deferreds. I will fire my callback (with None) if and when all of my component Deferreds fire successfully. I will fire my errback when and if any of my component Deferreds errbacks, in which case I will absorb the failure. If a second Deferred errbacks, I will not absorb that failure. This means that you can put a bunch of Deferreds together into an AsyncAND and then forget about them. If all succeed, the AsyncAND will fire. If one fails, that Failure will be propagated to the AsyncAND. If multiple ones fail, the first Failure will go to the AsyncAND and the rest will be left unhandled (and therefore logged). """ def __init__(self, deferredList): defer.Deferred.__init__(self) if not deferredList: self.callback(None) return self.remaining = len(deferredList) self._fired = False for d in deferredList: d.addCallbacks(self._cbDeferred, self._cbDeferred, callbackArgs=(True,), errbackArgs=(False,)) def _cbDeferred(self, result, succeeded): self.remaining -= 1 if succeeded: if not self._fired and self.remaining == 0: # the last input has fired. We fire. self._fired = True self.callback(None) return else: if not self._fired: # the first Failure is carried into our output self._fired = True self.errback(result) return None else: # second and later Failures are not absorbed return result # adapted from Tahoe: finds a single publically-visible address, or None. # Tahoe also uses code to run /bin/ifconfig (or equivalent) to find other # addresses, but that's a bit heavy for this. Note that this runs # synchronously. Also note that this doesn't require the reactor to be # running. def get_local_ip_for(target='A.ROOT-SERVERS.NET'): """Find out what our IP address is for use by a given target. @return: the IP address as a dotted-quad string which could be used by to connect to us. It might work for them, it might not. If there is no suitable address (perhaps we don't currently have an externally-visible interface), this will return None. """ try: target_ipaddr = socket.gethostbyname(target) except socket.gaierror: # DNS isn't running return None udpprot = protocol.DatagramProtocol() port = reactor.listenUDP(0, udpprot) try: udpprot.transport.connect(target_ipaddr, 7) localip = udpprot.transport.getHost().host except socket.error: # no route to that host localip = None port.stopListening() # note, this returns a Deferred return localip FORMAT_TIME_MODES = ["short-local", "long-local", "utc", "epoch"] def format_time(when, mode): if mode == "short-local": time_s = time.strftime("%H:%M:%S", time.localtime(when)) time_s = time_s + ".%03d" % int(1000*(when - int(when))) elif mode == "long-local": lt = time.localtime(when) time_s = time.strftime("%Y-%m-%d_%H:%M:%S", lt) time_s = time_s + ".%06d" % int(1000000*(when - int(when))) time_s += time.strftime("%z", lt) elif mode == "utc": time_s = time.strftime("%Y-%m-%d_%H:%M:%S", time.gmtime(when)) time_s = time_s + ".%06d" % int(1000000*(when - int(when))) time_s += "Z" elif mode == "epoch": time_s = "%.03f" % when return time_s def move_into_place(source, dest): """Atomically replace a file, or as near to it as the platform allows. The dest file may or may not exist.""" # from Tahoe if "win32" in sys.platform.lower(): try: os.remove(dest) except: pass os.rename(source, dest) def isSubstring(small, big): assert type(small) is str and type(big) is str return small in big def allocate_tcp_port(): """Return an (integer) available TCP port on localhost. This briefly listens on the port in question, then closes it right away.""" # Making this work correctly on multiple OSes is non-trivial: # * on OS-X: # * Binding the test socket to 127.0.0.1 lets the kernel give us a # LISTEN port that some other process is using, if they bound it to # ANY (0.0.0.0). These will fail when we attempt to # listen(bind=0.0.0.0) ourselves # * Binding the test socket to 0.0.0.0 lets the kernel give us LISTEN # ports bound to 127.0.0.1, although then our subsequent listen() # call usually succeeds. # * In both cases, the kernel can give us a port that's in use by the # near side of an ESTABLISHED socket. If the process which owns that # socket is not owned by the same user as us, listen() will fail. # * Doing a listen() right away (on the kernel-allocated socket) # succeeds, but a subsequent listen() on a new socket (bound to # the same port) will fail. # * on Linux: # * The kernel never gives us a port in use by a LISTEN socket, whether # we bind the test socket to 127.0.0.1 or 0.0.0.0 # * Binding it to 127.0.0.1 does let the kernel give us ports used in # an ESTABLISHED connection. Our listen() will fail regardless of who # owns that socket. (note that we are using SO_REUSEADDR but not # SO_REUSEPORT, which would probably affect things). # # # So to make this work properly everywhere, allocate_tcp_port() needs two # phases: first we allocate a port (with 0.0.0.0), then we close that # socket, then we open a second socket, bind the second socket to the # same port, then try to listen. If the listen() fails, we loop back and # try again. # In addition, on at least OS-X, the kernel will give us a port that's in # use by some other process, when that process has bound it to 127.0.0.1, # and our bind/listen (to 0.0.0.0) will succeed, but a subsequent caller # who tries to bind it to 127.0.0.1 will get an error in listen(). So we # must actually test the proposed socket twice: once bound to 0.0.0.0, # and again bound to 127.0.0.1. This probably isn't complete for # applications which bind to a specific outward-facing interface, but I'm # ok with that; anything other than 0.0.0.0 or 127.0.0.1 is likely to use # manually-selected ports, assigned by the user or sysadmin. # Ideally we'd refrain from doing listen(), to minimize impact on the # system, and we'd bind the port to 127.0.0.1, to avoid making it look # like we're accepting data from the outside world (in situations where # we're going to end up binding the port to 127.0.0.1 anyways). But for # the above reasons, neither would work. We *do* add SO_REUSEADDR, to # make sure our lingering socket won't prevent our caller from opening it # themselves in a few moments (note that Twisted's # tcp.Port.createInternetSocket sets SO_REUSEADDR, among other flags). count = 0 while True: s = _make_socket() s.bind(("0.0.0.0", 0)) port = s.getsockname()[1] s.close() s = _make_socket() try: s.bind(("0.0.0.0", port)) s.listen(5) # this is what sometimes fails s.close() s = _make_socket() s.bind(("127.0.0.1", port)) s.listen(5) s.close() return port except socket.error: s.close() count += 1 if count > 100: raise # try again def _make_socket(): s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) if platformType == "posix" and sys.platform != "cygwin": s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) return s foolscap-0.13.1/src/foolscap/vocab.py0000644000076500000240000000233412766553111020075 0ustar warnerstaff00000000000000 from hashlib import sha1 # here is the list of initial vocab tables. If the two ends negotiate to use # initial-vocab-table-index N, then both sides will start with the words from # INITIAL_VOCAB_TABLES[n] for their VOCABized tokens. vocab_v0 = [] vocab_v1 = [ # all opentypes used in 0.0.6 "none", "boolean", "reference", "dict", "list", "tuple", "set", "immutable-set", "unicode", "set-vocab", "add-vocab", "call", "arguments", "answer", "error", "my-reference", "your-reference", "their-reference", "copyable", # these are only used by storage.py "instance", "module", "class", "method", "function", # I'm not sure this one is actually used anywhere, but the first 127 of # these are basically free. "attrdict", ] INITIAL_VOCAB_TABLES = { 0: vocab_v0, 1: vocab_v1 } # to insure both sides agree on the actual words, we can hash the vocab table # into a short string. This is included in the negotiation decision and # compared by the receiving side. def hashVocabTable(table_index): data = "\x00".join(INITIAL_VOCAB_TABLES[table_index]) digest = sha1(data).hexdigest() return digest[:4] def getVocabRange(): keys = INITIAL_VOCAB_TABLES.keys() return min(keys), max(keys) foolscap-0.13.1/src/foolscap.egg-info/0000755000076500000240000000000013204747603020120 5ustar warnerstaff00000000000000foolscap-0.13.1/src/foolscap.egg-info/dependency_links.txt0000644000076500000240000000000113204747603024166 0ustar warnerstaff00000000000000 foolscap-0.13.1/src/foolscap.egg-info/entry_points.txt0000644000076500000240000000025513204747603023420 0ustar warnerstaff00000000000000[console_scripts] flappclient = foolscap.appserver.client:run_flappclient flappserver = foolscap.appserver.cli:run_flappserver flogtool = foolscap.logging.cli:run_flogtool foolscap-0.13.1/src/foolscap.egg-info/PKG-INFO0000644000076500000240000000204213204747603021213 0ustar warnerstaff00000000000000Metadata-Version: 1.1 Name: foolscap Version: 0.13.1 Summary: Foolscap contains an RPC protocol for Twisted. Home-page: http://foolscap.lothar.com/trac Author: Brian Warner Author-email: warner-foolscap@lothar.com License: MIT Description-Content-Type: UNKNOWN Description: Foolscap (aka newpb) is a new version of Twisted's native RPC protocol, known as 'Perspective Broker'. This allows an object in one process to be used by code in a distant process. This module provides data marshaling, a remote object reference system, and a capability-based security model. Platform: any Classifier: Development Status :: 3 - Alpha Classifier: Operating System :: OS Independent Classifier: License :: OSI Approved :: MIT License Classifier: Programming Language :: Python Classifier: Topic :: Internet Classifier: Topic :: Software Development :: Libraries :: Python Modules Classifier: Topic :: System :: Distributed Computing Classifier: Topic :: System :: Networking Classifier: Topic :: Software Development :: Object Brokering foolscap-0.13.1/src/foolscap.egg-info/requires.txt0000644000076500000240000000022013204747603022512 0ustar warnerstaff00000000000000twisted[tls]>=16.0.0 pyOpenSSL [dev] mock txsocksx txtorcon>=0.16.1 txi2p>=0.3.2 [i2p] txi2p>=0.3.2 [socks] txsocksx [tor] txtorcon>=0.16.1 foolscap-0.13.1/src/foolscap.egg-info/SOURCES.txt0000644000076500000240000000752213204747603022012 0ustar warnerstaff00000000000000.coveragerc ChangeLog.0.6.4 LICENSE MANIFEST.in Makefile NEWS README README.packagers setup.cfg setup.py tox.ini versioneer.py doc/connection-handlers.rst doc/copyable.rst doc/failures.rst doc/flappserver.rst doc/jobs.txt doc/logging.rst doc/schema.rst doc/serializing.rst doc/todo.txt doc/use-cases.txt doc/using-foolscap.rst doc/listings/copyable-receive.py doc/listings/copyable-send.py doc/listings/pb2client.py doc/listings/pb2server.py doc/listings/pb3calculator.py doc/listings/pb3user.py doc/specifications/banana.rst doc/specifications/logfiles.rst doc/specifications/pb.rst misc/classify_foolscap.py src/foolscap/__init__.py src/foolscap/_version.py src/foolscap/api.py src/foolscap/banana.py src/foolscap/base32.py src/foolscap/broker.py src/foolscap/call.py src/foolscap/connection.py src/foolscap/constraint.py src/foolscap/copyable.py src/foolscap/crypto.py src/foolscap/eventual.py src/foolscap/furl.py src/foolscap/info.py src/foolscap/ipb.py src/foolscap/negotiate.py src/foolscap/observer.py src/foolscap/pb.py src/foolscap/promise.py src/foolscap/reconnector.py src/foolscap/referenceable.py src/foolscap/remoteinterface.py src/foolscap/schema.py src/foolscap/slicer.py src/foolscap/storage.py src/foolscap/stringchain.py src/foolscap/tokens.py src/foolscap/util.py src/foolscap/vocab.py src/foolscap.egg-info/PKG-INFO src/foolscap.egg-info/SOURCES.txt src/foolscap.egg-info/dependency_links.txt src/foolscap.egg-info/entry_points.txt src/foolscap.egg-info/requires.txt src/foolscap.egg-info/top_level.txt src/foolscap/appserver/__init__.py src/foolscap/appserver/cli.py src/foolscap/appserver/client.py src/foolscap/appserver/server.py src/foolscap/appserver/services.py src/foolscap/connections/__init__.py src/foolscap/connections/i2p.py src/foolscap/connections/socks.py src/foolscap/connections/tcp.py src/foolscap/connections/tor.py src/foolscap/logging/__init__.py src/foolscap/logging/app_versions.py src/foolscap/logging/cli.py src/foolscap/logging/dumper.py src/foolscap/logging/filter.py src/foolscap/logging/flogfile.py src/foolscap/logging/gatherer.py src/foolscap/logging/incident.py src/foolscap/logging/interfaces.py src/foolscap/logging/levels.py src/foolscap/logging/log.py src/foolscap/logging/publish.py src/foolscap/logging/tail.py src/foolscap/logging/web.py src/foolscap/slicers/__init__.py src/foolscap/slicers/allslicers.py src/foolscap/slicers/bool.py src/foolscap/slicers/decimal_slicer.py src/foolscap/slicers/dict.py src/foolscap/slicers/list.py src/foolscap/slicers/none.py src/foolscap/slicers/root.py src/foolscap/slicers/set.py src/foolscap/slicers/tuple.py src/foolscap/slicers/unicode.py src/foolscap/slicers/vocab.py src/foolscap/test/__init__.py src/foolscap/test/apphelper.py src/foolscap/test/bench_banana.py src/foolscap/test/check-connections-client.py src/foolscap/test/check-connections-server.py src/foolscap/test/common.py src/foolscap/test/run_trial.py src/foolscap/test/test__versions.py src/foolscap/test/test_appserver.py src/foolscap/test/test_banana.py src/foolscap/test/test_call.py src/foolscap/test/test_connection.py src/foolscap/test/test_copyable.py src/foolscap/test/test_crypto.py src/foolscap/test/test_eventual.py src/foolscap/test/test_gifts.py src/foolscap/test/test_info.py src/foolscap/test/test_interfaces.py src/foolscap/test/test_keepalive.py src/foolscap/test/test_listener.py src/foolscap/test/test_logging.py src/foolscap/test/test_loopback.py src/foolscap/test/test_negotiate.py src/foolscap/test/test_observer.py src/foolscap/test/test_pb.py src/foolscap/test/test_promise.py src/foolscap/test/test_reconnector.py src/foolscap/test/test_reference.py src/foolscap/test/test_registration.py src/foolscap/test/test_schema.py src/foolscap/test/test_serialize.py src/foolscap/test/test_stringchain.py src/foolscap/test/test_sturdyref.py src/foolscap/test/test_tub.py src/foolscap/test/test_unreachable.py src/foolscap/test/test_util.pyfoolscap-0.13.1/src/foolscap.egg-info/top_level.txt0000644000076500000240000000001113204747603022642 0ustar warnerstaff00000000000000foolscap foolscap-0.13.1/tox.ini0000644000076500000240000000211513204511065015332 0ustar warnerstaff00000000000000[tox] envlist = py27 minversion = 2.4.0 [testenv] basepython=python2.7 passenv = USERPROFILE HOMEDRIVE HOMEPATH PYTHONWARNINGS usedevelop = True extras = dev deps = pyflakes {env:TWISTED:} {env:PYOPENSSL:} commands = pyflakes setup.py src trial {posargs:foolscap} # on my home machine, 'coverage --branch' increases runtime by 10% (over # tests without any coverage tracking) [testenv:coverage] deps = pyflakes coverage {env:TWISTED:} {env:PYOPENSSL:} commands = pyflakes setup.py src coverage run --branch -m foolscap.test.run_trial {posargs:foolscap} coverage xml [testenv:upcoming-deprecations] deps = # we want twisted[tls] (to get service-identity), but to do that with a # URL, you need the extra "#egg=twisted" bit git+https://github.com/twisted/twisted#egg=twisted[tls] setenv = PYTHONWARNINGS=default::DeprecationWarning commands = python misc/run-deprecations.py --warnings=_trial_temp/deprecation-warnings.log trial --rterrors {posargs:foolscap} foolscap-0.13.1/versioneer.py0000644000076500000240000020600313204160675016563 0ustar warnerstaff00000000000000 # Version: 0.18 """The Versioneer - like a rocketeer, but for versions. The Versioneer ============== * like a rocketeer, but for versions! * https://github.com/warner/python-versioneer * Brian Warner * License: Public Domain * Compatible With: python2.6, 2.7, 3.2, 3.3, 3.4, 3.5, 3.6, and pypy * [![Latest Version] (https://pypip.in/version/versioneer/badge.svg?style=flat) ](https://pypi.python.org/pypi/versioneer/) * [![Build Status] (https://travis-ci.org/warner/python-versioneer.png?branch=master) ](https://travis-ci.org/warner/python-versioneer) This is a tool for managing a recorded version number in distutils-based python projects. The goal is to remove the tedious and error-prone "update the embedded version string" step from your release process. Making a new release should be as easy as recording a new tag in your version-control system, and maybe making new tarballs. ## Quick Install * `pip install versioneer` to somewhere to your $PATH * add a `[versioneer]` section to your setup.cfg (see below) * run `versioneer install` in your source tree, commit the results ## Version Identifiers Source trees come from a variety of places: * a version-control system checkout (mostly used by developers) * a nightly tarball, produced by build automation * a snapshot tarball, produced by a web-based VCS browser, like github's "tarball from tag" feature * a release tarball, produced by "setup.py sdist", distributed through PyPI Within each source tree, the version identifier (either a string or a number, this tool is format-agnostic) can come from a variety of places: * ask the VCS tool itself, e.g. "git describe" (for checkouts), which knows about recent "tags" and an absolute revision-id * the name of the directory into which the tarball was unpacked * an expanded VCS keyword ($Id$, etc) * a `_version.py` created by some earlier build step For released software, the version identifier is closely related to a VCS tag. Some projects use tag names that include more than just the version string (e.g. "myproject-1.2" instead of just "1.2"), in which case the tool needs to strip the tag prefix to extract the version identifier. For unreleased software (between tags), the version identifier should provide enough information to help developers recreate the same tree, while also giving them an idea of roughly how old the tree is (after version 1.2, before version 1.3). Many VCS systems can report a description that captures this, for example `git describe --tags --dirty --always` reports things like "0.7-1-g574ab98-dirty" to indicate that the checkout is one revision past the 0.7 tag, has a unique revision id of "574ab98", and is "dirty" (it has uncommitted changes. The version identifier is used for multiple purposes: * to allow the module to self-identify its version: `myproject.__version__` * to choose a name and prefix for a 'setup.py sdist' tarball ## Theory of Operation Versioneer works by adding a special `_version.py` file into your source tree, where your `__init__.py` can import it. This `_version.py` knows how to dynamically ask the VCS tool for version information at import time. `_version.py` also contains `$Revision$` markers, and the installation process marks `_version.py` to have this marker rewritten with a tag name during the `git archive` command. As a result, generated tarballs will contain enough information to get the proper version. To allow `setup.py` to compute a version too, a `versioneer.py` is added to the top level of your source tree, next to `setup.py` and the `setup.cfg` that configures it. This overrides several distutils/setuptools commands to compute the version when invoked, and changes `setup.py build` and `setup.py sdist` to replace `_version.py` with a small static file that contains just the generated version data. ## Installation See [INSTALL.md](./INSTALL.md) for detailed installation instructions. ## Version-String Flavors Code which uses Versioneer can learn about its version string at runtime by importing `_version` from your main `__init__.py` file and running the `get_versions()` function. From the "outside" (e.g. in `setup.py`), you can import the top-level `versioneer.py` and run `get_versions()`. Both functions return a dictionary with different flavors of version information: * `['version']`: A condensed version string, rendered using the selected style. This is the most commonly used value for the project's version string. The default "pep440" style yields strings like `0.11`, `0.11+2.g1076c97`, or `0.11+2.g1076c97.dirty`. See the "Styles" section below for alternative styles. * `['full-revisionid']`: detailed revision identifier. For Git, this is the full SHA1 commit id, e.g. "1076c978a8d3cfc70f408fe5974aa6c092c949ac". * `['date']`: Date and time of the latest `HEAD` commit. For Git, it is the commit date in ISO 8601 format. This will be None if the date is not available. * `['dirty']`: a boolean, True if the tree has uncommitted changes. Note that this is only accurate if run in a VCS checkout, otherwise it is likely to be False or None * `['error']`: if the version string could not be computed, this will be set to a string describing the problem, otherwise it will be None. It may be useful to throw an exception in setup.py if this is set, to avoid e.g. creating tarballs with a version string of "unknown". Some variants are more useful than others. Including `full-revisionid` in a bug report should allow developers to reconstruct the exact code being tested (or indicate the presence of local changes that should be shared with the developers). `version` is suitable for display in an "about" box or a CLI `--version` output: it can be easily compared against release notes and lists of bugs fixed in various releases. The installer adds the following text to your `__init__.py` to place a basic version in `YOURPROJECT.__version__`: from ._version import get_versions __version__ = get_versions()['version'] del get_versions ## Styles The setup.cfg `style=` configuration controls how the VCS information is rendered into a version string. The default style, "pep440", produces a PEP440-compliant string, equal to the un-prefixed tag name for actual releases, and containing an additional "local version" section with more detail for in-between builds. For Git, this is TAG[+DISTANCE.gHEX[.dirty]] , using information from `git describe --tags --dirty --always`. For example "0.11+2.g1076c97.dirty" indicates that the tree is like the "1076c97" commit but has uncommitted changes (".dirty"), and that this commit is two revisions ("+2") beyond the "0.11" tag. For released software (exactly equal to a known tag), the identifier will only contain the stripped tag, e.g. "0.11". Other styles are available. See [details.md](details.md) in the Versioneer source tree for descriptions. ## Debugging Versioneer tries to avoid fatal errors: if something goes wrong, it will tend to return a version of "0+unknown". To investigate the problem, run `setup.py version`, which will run the version-lookup code in a verbose mode, and will display the full contents of `get_versions()` (including the `error` string, which may help identify what went wrong). ## Known Limitations Some situations are known to cause problems for Versioneer. This details the most significant ones. More can be found on Github [issues page](https://github.com/warner/python-versioneer/issues). ### Subprojects Versioneer has limited support for source trees in which `setup.py` is not in the root directory (e.g. `setup.py` and `.git/` are *not* siblings). The are two common reasons why `setup.py` might not be in the root: * Source trees which contain multiple subprojects, such as [Buildbot](https://github.com/buildbot/buildbot), which contains both "master" and "slave" subprojects, each with their own `setup.py`, `setup.cfg`, and `tox.ini`. Projects like these produce multiple PyPI distributions (and upload multiple independently-installable tarballs). * Source trees whose main purpose is to contain a C library, but which also provide bindings to Python (and perhaps other langauges) in subdirectories. Versioneer will look for `.git` in parent directories, and most operations should get the right version string. However `pip` and `setuptools` have bugs and implementation details which frequently cause `pip install .` from a subproject directory to fail to find a correct version string (so it usually defaults to `0+unknown`). `pip install --editable .` should work correctly. `setup.py install` might work too. Pip-8.1.1 is known to have this problem, but hopefully it will get fixed in some later version. [Bug #38](https://github.com/warner/python-versioneer/issues/38) is tracking this issue. The discussion in [PR #61](https://github.com/warner/python-versioneer/pull/61) describes the issue from the Versioneer side in more detail. [pip PR#3176](https://github.com/pypa/pip/pull/3176) and [pip PR#3615](https://github.com/pypa/pip/pull/3615) contain work to improve pip to let Versioneer work correctly. Versioneer-0.16 and earlier only looked for a `.git` directory next to the `setup.cfg`, so subprojects were completely unsupported with those releases. ### Editable installs with setuptools <= 18.5 `setup.py develop` and `pip install --editable .` allow you to install a project into a virtualenv once, then continue editing the source code (and test) without re-installing after every change. "Entry-point scripts" (`setup(entry_points={"console_scripts": ..})`) are a convenient way to specify executable scripts that should be installed along with the python package. These both work as expected when using modern setuptools. When using setuptools-18.5 or earlier, however, certain operations will cause `pkg_resources.DistributionNotFound` errors when running the entrypoint script, which must be resolved by re-installing the package. This happens when the install happens with one version, then the egg_info data is regenerated while a different version is checked out. Many setup.py commands cause egg_info to be rebuilt (including `sdist`, `wheel`, and installing into a different virtualenv), so this can be surprising. [Bug #83](https://github.com/warner/python-versioneer/issues/83) describes this one, but upgrading to a newer version of setuptools should probably resolve it. ### Unicode version strings While Versioneer works (and is continually tested) with both Python 2 and Python 3, it is not entirely consistent with bytes-vs-unicode distinctions. Newer releases probably generate unicode version strings on py2. It's not clear that this is wrong, but it may be surprising for applications when then write these strings to a network connection or include them in bytes-oriented APIs like cryptographic checksums. [Bug #71](https://github.com/warner/python-versioneer/issues/71) investigates this question. ## Updating Versioneer To upgrade your project to a new release of Versioneer, do the following: * install the new Versioneer (`pip install -U versioneer` or equivalent) * edit `setup.cfg`, if necessary, to include any new configuration settings indicated by the release notes. See [UPGRADING](./UPGRADING.md) for details. * re-run `versioneer install` in your source tree, to replace `SRC/_version.py` * commit any changed files ## Future Directions This tool is designed to make it easily extended to other version-control systems: all VCS-specific components are in separate directories like src/git/ . The top-level `versioneer.py` script is assembled from these components by running make-versioneer.py . In the future, make-versioneer.py will take a VCS name as an argument, and will construct a version of `versioneer.py` that is specific to the given VCS. It might also take the configuration arguments that are currently provided manually during installation by editing setup.py . Alternatively, it might go the other direction and include code from all supported VCS systems, reducing the number of intermediate scripts. ## License To make Versioneer easier to embed, all its code is dedicated to the public domain. The `_version.py` that it creates is also in the public domain. Specifically, both are released under the Creative Commons "Public Domain Dedication" license (CC0-1.0), as described in https://creativecommons.org/publicdomain/zero/1.0/ . """ from __future__ import print_function try: import configparser except ImportError: import ConfigParser as configparser import errno import json import os import re import subprocess import sys class VersioneerConfig: """Container for Versioneer configuration parameters.""" def get_root(): """Get the project root directory. We require that all commands are run from the project root, i.e. the directory that contains setup.py, setup.cfg, and versioneer.py . """ root = os.path.realpath(os.path.abspath(os.getcwd())) setup_py = os.path.join(root, "setup.py") versioneer_py = os.path.join(root, "versioneer.py") if not (os.path.exists(setup_py) or os.path.exists(versioneer_py)): # allow 'python path/to/setup.py COMMAND' root = os.path.dirname(os.path.realpath(os.path.abspath(sys.argv[0]))) setup_py = os.path.join(root, "setup.py") versioneer_py = os.path.join(root, "versioneer.py") if not (os.path.exists(setup_py) or os.path.exists(versioneer_py)): err = ("Versioneer was unable to run the project root directory. " "Versioneer requires setup.py to be executed from " "its immediate directory (like 'python setup.py COMMAND'), " "or in a way that lets it use sys.argv[0] to find the root " "(like 'python path/to/setup.py COMMAND').") raise VersioneerBadRootError(err) try: # Certain runtime workflows (setup.py install/develop in a setuptools # tree) execute all dependencies in a single python process, so # "versioneer" may be imported multiple times, and python's shared # module-import table will cache the first one. So we can't use # os.path.dirname(__file__), as that will find whichever # versioneer.py was first imported, even in later projects. me = os.path.realpath(os.path.abspath(__file__)) me_dir = os.path.normcase(os.path.splitext(me)[0]) vsr_dir = os.path.normcase(os.path.splitext(versioneer_py)[0]) if me_dir != vsr_dir: print("Warning: build in %s is using versioneer.py from %s" % (os.path.dirname(me), versioneer_py)) except NameError: pass return root def get_config_from_root(root): """Read the project setup.cfg file to determine Versioneer config.""" # This might raise EnvironmentError (if setup.cfg is missing), or # configparser.NoSectionError (if it lacks a [versioneer] section), or # configparser.NoOptionError (if it lacks "VCS="). See the docstring at # the top of versioneer.py for instructions on writing your setup.cfg . setup_cfg = os.path.join(root, "setup.cfg") parser = configparser.SafeConfigParser() with open(setup_cfg, "r") as f: parser.readfp(f) VCS = parser.get("versioneer", "VCS") # mandatory def get(parser, name): if parser.has_option("versioneer", name): return parser.get("versioneer", name) return None cfg = VersioneerConfig() cfg.VCS = VCS cfg.style = get(parser, "style") or "" cfg.versionfile_source = get(parser, "versionfile_source") cfg.versionfile_build = get(parser, "versionfile_build") cfg.tag_prefix = get(parser, "tag_prefix") if cfg.tag_prefix in ("''", '""'): cfg.tag_prefix = "" cfg.parentdir_prefix = get(parser, "parentdir_prefix") cfg.verbose = get(parser, "verbose") return cfg class NotThisMethod(Exception): """Exception raised if a method is not valid for the current scenario.""" # these dictionaries contain VCS-specific tools LONG_VERSION_PY = {} HANDLERS = {} def register_vcs_handler(vcs, method): # decorator """Decorator to mark a method as the handler for a particular VCS.""" def decorate(f): """Store f in HANDLERS[vcs][method].""" if vcs not in HANDLERS: HANDLERS[vcs] = {} HANDLERS[vcs][method] = f return f return decorate def run_command(commands, args, cwd=None, verbose=False, hide_stderr=False, env=None): """Call the given command(s).""" assert isinstance(commands, list) p = None for c in commands: try: dispcmd = str([c] + args) # remember shell=False, so use git.cmd on windows, not just git p = subprocess.Popen([c] + args, cwd=cwd, env=env, stdout=subprocess.PIPE, stderr=(subprocess.PIPE if hide_stderr else None)) break except EnvironmentError: e = sys.exc_info()[1] if e.errno == errno.ENOENT: continue if verbose: print("unable to run %s" % dispcmd) print(e) return None, None else: if verbose: print("unable to find command, tried %s" % (commands,)) return None, None stdout = p.communicate()[0].strip() if sys.version_info[0] >= 3: stdout = stdout.decode() if p.returncode != 0: if verbose: print("unable to run %s (error)" % dispcmd) print("stdout was %s" % stdout) return None, p.returncode return stdout, p.returncode LONG_VERSION_PY['git'] = ''' # This file helps to compute a version number in source trees obtained from # git-archive tarball (such as those provided by githubs download-from-tag # feature). Distribution tarballs (built by setup.py sdist) and build # directories (produced by setup.py build) will contain a much shorter file # that just contains the computed version number. # This file is released into the public domain. Generated by # versioneer-0.18 (https://github.com/warner/python-versioneer) """Git implementation of _version.py.""" import errno import os import re import subprocess import sys def get_keywords(): """Get the keywords needed to look up the version information.""" # these strings will be replaced by git during git-archive. # setup.py/versioneer.py will grep for the variable names, so they must # each be defined on a line of their own. _version.py will just call # get_keywords(). git_refnames = "%(DOLLAR)sFormat:%%d%(DOLLAR)s" git_full = "%(DOLLAR)sFormat:%%H%(DOLLAR)s" git_date = "%(DOLLAR)sFormat:%%ci%(DOLLAR)s" keywords = {"refnames": git_refnames, "full": git_full, "date": git_date} return keywords class VersioneerConfig: """Container for Versioneer configuration parameters.""" def get_config(): """Create, populate and return the VersioneerConfig() object.""" # these strings are filled in when 'setup.py versioneer' creates # _version.py cfg = VersioneerConfig() cfg.VCS = "git" cfg.style = "%(STYLE)s" cfg.tag_prefix = "%(TAG_PREFIX)s" cfg.parentdir_prefix = "%(PARENTDIR_PREFIX)s" cfg.versionfile_source = "%(VERSIONFILE_SOURCE)s" cfg.verbose = False return cfg class NotThisMethod(Exception): """Exception raised if a method is not valid for the current scenario.""" LONG_VERSION_PY = {} HANDLERS = {} def register_vcs_handler(vcs, method): # decorator """Decorator to mark a method as the handler for a particular VCS.""" def decorate(f): """Store f in HANDLERS[vcs][method].""" if vcs not in HANDLERS: HANDLERS[vcs] = {} HANDLERS[vcs][method] = f return f return decorate def run_command(commands, args, cwd=None, verbose=False, hide_stderr=False, env=None): """Call the given command(s).""" assert isinstance(commands, list) p = None for c in commands: try: dispcmd = str([c] + args) # remember shell=False, so use git.cmd on windows, not just git p = subprocess.Popen([c] + args, cwd=cwd, env=env, stdout=subprocess.PIPE, stderr=(subprocess.PIPE if hide_stderr else None)) break except EnvironmentError: e = sys.exc_info()[1] if e.errno == errno.ENOENT: continue if verbose: print("unable to run %%s" %% dispcmd) print(e) return None, None else: if verbose: print("unable to find command, tried %%s" %% (commands,)) return None, None stdout = p.communicate()[0].strip() if sys.version_info[0] >= 3: stdout = stdout.decode() if p.returncode != 0: if verbose: print("unable to run %%s (error)" %% dispcmd) print("stdout was %%s" %% stdout) return None, p.returncode return stdout, p.returncode def versions_from_parentdir(parentdir_prefix, root, verbose): """Try to determine the version from the parent directory name. Source tarballs conventionally unpack into a directory that includes both the project name and a version string. We will also support searching up two directory levels for an appropriately named parent directory """ rootdirs = [] for i in range(3): dirname = os.path.basename(root) if dirname.startswith(parentdir_prefix): return {"version": dirname[len(parentdir_prefix):], "full-revisionid": None, "dirty": False, "error": None, "date": None} else: rootdirs.append(root) root = os.path.dirname(root) # up a level if verbose: print("Tried directories %%s but none started with prefix %%s" %% (str(rootdirs), parentdir_prefix)) raise NotThisMethod("rootdir doesn't start with parentdir_prefix") @register_vcs_handler("git", "get_keywords") def git_get_keywords(versionfile_abs): """Extract version information from the given file.""" # the code embedded in _version.py can just fetch the value of these # keywords. When used from setup.py, we don't want to import _version.py, # so we do it with a regexp instead. This function is not used from # _version.py. keywords = {} try: f = open(versionfile_abs, "r") for line in f.readlines(): if line.strip().startswith("git_refnames ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["refnames"] = mo.group(1) if line.strip().startswith("git_full ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["full"] = mo.group(1) if line.strip().startswith("git_date ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["date"] = mo.group(1) f.close() except EnvironmentError: pass return keywords @register_vcs_handler("git", "keywords") def git_versions_from_keywords(keywords, tag_prefix, verbose): """Get version information from git keywords.""" if not keywords: raise NotThisMethod("no keywords at all, weird") date = keywords.get("date") if date is not None: # git-2.2.0 added "%%cI", which expands to an ISO-8601 -compliant # datestamp. However we prefer "%%ci" (which expands to an "ISO-8601 # -like" string, which we must then edit to make compliant), because # it's been around since git-1.5.3, and it's too difficult to # discover which version we're using, or to work around using an # older one. date = date.strip().replace(" ", "T", 1).replace(" ", "", 1) refnames = keywords["refnames"].strip() if refnames.startswith("$Format"): if verbose: print("keywords are unexpanded, not using") raise NotThisMethod("unexpanded keywords, not a git-archive tarball") refs = set([r.strip() for r in refnames.strip("()").split(",")]) # starting in git-1.8.3, tags are listed as "tag: foo-1.0" instead of # just "foo-1.0". If we see a "tag: " prefix, prefer those. TAG = "tag: " tags = set([r[len(TAG):] for r in refs if r.startswith(TAG)]) if not tags: # Either we're using git < 1.8.3, or there really are no tags. We use # a heuristic: assume all version tags have a digit. The old git %%d # expansion behaves like git log --decorate=short and strips out the # refs/heads/ and refs/tags/ prefixes that would let us distinguish # between branches and tags. By ignoring refnames without digits, we # filter out many common branch names like "release" and # "stabilization", as well as "HEAD" and "master". tags = set([r for r in refs if re.search(r'\d', r)]) if verbose: print("discarding '%%s', no digits" %% ",".join(refs - tags)) if verbose: print("likely tags: %%s" %% ",".join(sorted(tags))) for ref in sorted(tags): # sorting will prefer e.g. "2.0" over "2.0rc1" if ref.startswith(tag_prefix): r = ref[len(tag_prefix):] if verbose: print("picking %%s" %% r) return {"version": r, "full-revisionid": keywords["full"].strip(), "dirty": False, "error": None, "date": date} # no suitable tags, so version is "0+unknown", but full hex is still there if verbose: print("no suitable tags, using unknown + full revision id") return {"version": "0+unknown", "full-revisionid": keywords["full"].strip(), "dirty": False, "error": "no suitable tags", "date": None} @register_vcs_handler("git", "pieces_from_vcs") def git_pieces_from_vcs(tag_prefix, root, verbose, run_command=run_command): """Get version from 'git describe' in the root of the source tree. This only gets called if the git-archive 'subst' keywords were *not* expanded, and _version.py hasn't already been rewritten with a short version string, meaning we're inside a checked out source tree. """ GITS = ["git"] if sys.platform == "win32": GITS = ["git.cmd", "git.exe"] out, rc = run_command(GITS, ["rev-parse", "--git-dir"], cwd=root, hide_stderr=True) if rc != 0: if verbose: print("Directory %%s not under git control" %% root) raise NotThisMethod("'git rev-parse --git-dir' returned error") # if there is a tag matching tag_prefix, this yields TAG-NUM-gHEX[-dirty] # if there isn't one, this yields HEX[-dirty] (no NUM) describe_out, rc = run_command(GITS, ["describe", "--tags", "--dirty", "--always", "--long", "--match", "%%s*" %% tag_prefix], cwd=root) # --long was added in git-1.5.5 if describe_out is None: raise NotThisMethod("'git describe' failed") describe_out = describe_out.strip() full_out, rc = run_command(GITS, ["rev-parse", "HEAD"], cwd=root) if full_out is None: raise NotThisMethod("'git rev-parse' failed") full_out = full_out.strip() pieces = {} pieces["long"] = full_out pieces["short"] = full_out[:7] # maybe improved later pieces["error"] = None # parse describe_out. It will be like TAG-NUM-gHEX[-dirty] or HEX[-dirty] # TAG might have hyphens. git_describe = describe_out # look for -dirty suffix dirty = git_describe.endswith("-dirty") pieces["dirty"] = dirty if dirty: git_describe = git_describe[:git_describe.rindex("-dirty")] # now we have TAG-NUM-gHEX or HEX if "-" in git_describe: # TAG-NUM-gHEX mo = re.search(r'^(.+)-(\d+)-g([0-9a-f]+)$', git_describe) if not mo: # unparseable. Maybe git-describe is misbehaving? pieces["error"] = ("unable to parse git-describe output: '%%s'" %% describe_out) return pieces # tag full_tag = mo.group(1) if not full_tag.startswith(tag_prefix): if verbose: fmt = "tag '%%s' doesn't start with prefix '%%s'" print(fmt %% (full_tag, tag_prefix)) pieces["error"] = ("tag '%%s' doesn't start with prefix '%%s'" %% (full_tag, tag_prefix)) return pieces pieces["closest-tag"] = full_tag[len(tag_prefix):] # distance: number of commits since tag pieces["distance"] = int(mo.group(2)) # commit: short hex revision ID pieces["short"] = mo.group(3) else: # HEX: no tags pieces["closest-tag"] = None count_out, rc = run_command(GITS, ["rev-list", "HEAD", "--count"], cwd=root) pieces["distance"] = int(count_out) # total number of commits # commit date: see ISO-8601 comment in git_versions_from_keywords() date = run_command(GITS, ["show", "-s", "--format=%%ci", "HEAD"], cwd=root)[0].strip() pieces["date"] = date.strip().replace(" ", "T", 1).replace(" ", "", 1) return pieces def plus_or_dot(pieces): """Return a + if we don't already have one, else return a .""" if "+" in pieces.get("closest-tag", ""): return "." return "+" def render_pep440(pieces): """Build up version string, with post-release "local version identifier". Our goal: TAG[+DISTANCE.gHEX[.dirty]] . Note that if you get a tagged build and then dirty it, you'll get TAG+0.gHEX.dirty Exceptions: 1: no tags. git_describe was just HEX. 0+untagged.DISTANCE.gHEX[.dirty] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += plus_or_dot(pieces) rendered += "%%d.g%%s" %% (pieces["distance"], pieces["short"]) if pieces["dirty"]: rendered += ".dirty" else: # exception #1 rendered = "0+untagged.%%d.g%%s" %% (pieces["distance"], pieces["short"]) if pieces["dirty"]: rendered += ".dirty" return rendered def render_pep440_pre(pieces): """TAG[.post.devDISTANCE] -- No -dirty. Exceptions: 1: no tags. 0.post.devDISTANCE """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"]: rendered += ".post.dev%%d" %% pieces["distance"] else: # exception #1 rendered = "0.post.dev%%d" %% pieces["distance"] return rendered def render_pep440_post(pieces): """TAG[.postDISTANCE[.dev0]+gHEX] . The ".dev0" means dirty. Note that .dev0 sorts backwards (a dirty tree will appear "older" than the corresponding clean one), but you shouldn't be releasing software with -dirty anyways. Exceptions: 1: no tags. 0.postDISTANCE[.dev0] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += ".post%%d" %% pieces["distance"] if pieces["dirty"]: rendered += ".dev0" rendered += plus_or_dot(pieces) rendered += "g%%s" %% pieces["short"] else: # exception #1 rendered = "0.post%%d" %% pieces["distance"] if pieces["dirty"]: rendered += ".dev0" rendered += "+g%%s" %% pieces["short"] return rendered def render_pep440_old(pieces): """TAG[.postDISTANCE[.dev0]] . The ".dev0" means dirty. Eexceptions: 1: no tags. 0.postDISTANCE[.dev0] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += ".post%%d" %% pieces["distance"] if pieces["dirty"]: rendered += ".dev0" else: # exception #1 rendered = "0.post%%d" %% pieces["distance"] if pieces["dirty"]: rendered += ".dev0" return rendered def render_git_describe(pieces): """TAG[-DISTANCE-gHEX][-dirty]. Like 'git describe --tags --dirty --always'. Exceptions: 1: no tags. HEX[-dirty] (note: no 'g' prefix) """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"]: rendered += "-%%d-g%%s" %% (pieces["distance"], pieces["short"]) else: # exception #1 rendered = pieces["short"] if pieces["dirty"]: rendered += "-dirty" return rendered def render_git_describe_long(pieces): """TAG-DISTANCE-gHEX[-dirty]. Like 'git describe --tags --dirty --always -long'. The distance/hash is unconditional. Exceptions: 1: no tags. HEX[-dirty] (note: no 'g' prefix) """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] rendered += "-%%d-g%%s" %% (pieces["distance"], pieces["short"]) else: # exception #1 rendered = pieces["short"] if pieces["dirty"]: rendered += "-dirty" return rendered def render(pieces, style): """Render the given version pieces into the requested style.""" if pieces["error"]: return {"version": "unknown", "full-revisionid": pieces.get("long"), "dirty": None, "error": pieces["error"], "date": None} if not style or style == "default": style = "pep440" # the default if style == "pep440": rendered = render_pep440(pieces) elif style == "pep440-pre": rendered = render_pep440_pre(pieces) elif style == "pep440-post": rendered = render_pep440_post(pieces) elif style == "pep440-old": rendered = render_pep440_old(pieces) elif style == "git-describe": rendered = render_git_describe(pieces) elif style == "git-describe-long": rendered = render_git_describe_long(pieces) else: raise ValueError("unknown style '%%s'" %% style) return {"version": rendered, "full-revisionid": pieces["long"], "dirty": pieces["dirty"], "error": None, "date": pieces.get("date")} def get_versions(): """Get version information or return default if unable to do so.""" # I am in _version.py, which lives at ROOT/VERSIONFILE_SOURCE. If we have # __file__, we can work backwards from there to the root. Some # py2exe/bbfreeze/non-CPython implementations don't do __file__, in which # case we can only use expanded keywords. cfg = get_config() verbose = cfg.verbose try: return git_versions_from_keywords(get_keywords(), cfg.tag_prefix, verbose) except NotThisMethod: pass try: root = os.path.realpath(__file__) # versionfile_source is the relative path from the top of the source # tree (where the .git directory might live) to this file. Invert # this to find the root from __file__. for i in cfg.versionfile_source.split('/'): root = os.path.dirname(root) except NameError: return {"version": "0+unknown", "full-revisionid": None, "dirty": None, "error": "unable to find root of source tree", "date": None} try: pieces = git_pieces_from_vcs(cfg.tag_prefix, root, verbose) return render(pieces, cfg.style) except NotThisMethod: pass try: if cfg.parentdir_prefix: return versions_from_parentdir(cfg.parentdir_prefix, root, verbose) except NotThisMethod: pass return {"version": "0+unknown", "full-revisionid": None, "dirty": None, "error": "unable to compute version", "date": None} ''' @register_vcs_handler("git", "get_keywords") def git_get_keywords(versionfile_abs): """Extract version information from the given file.""" # the code embedded in _version.py can just fetch the value of these # keywords. When used from setup.py, we don't want to import _version.py, # so we do it with a regexp instead. This function is not used from # _version.py. keywords = {} try: f = open(versionfile_abs, "r") for line in f.readlines(): if line.strip().startswith("git_refnames ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["refnames"] = mo.group(1) if line.strip().startswith("git_full ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["full"] = mo.group(1) if line.strip().startswith("git_date ="): mo = re.search(r'=\s*"(.*)"', line) if mo: keywords["date"] = mo.group(1) f.close() except EnvironmentError: pass return keywords @register_vcs_handler("git", "keywords") def git_versions_from_keywords(keywords, tag_prefix, verbose): """Get version information from git keywords.""" if not keywords: raise NotThisMethod("no keywords at all, weird") date = keywords.get("date") if date is not None: # git-2.2.0 added "%cI", which expands to an ISO-8601 -compliant # datestamp. However we prefer "%ci" (which expands to an "ISO-8601 # -like" string, which we must then edit to make compliant), because # it's been around since git-1.5.3, and it's too difficult to # discover which version we're using, or to work around using an # older one. date = date.strip().replace(" ", "T", 1).replace(" ", "", 1) refnames = keywords["refnames"].strip() if refnames.startswith("$Format"): if verbose: print("keywords are unexpanded, not using") raise NotThisMethod("unexpanded keywords, not a git-archive tarball") refs = set([r.strip() for r in refnames.strip("()").split(",")]) # starting in git-1.8.3, tags are listed as "tag: foo-1.0" instead of # just "foo-1.0". If we see a "tag: " prefix, prefer those. TAG = "tag: " tags = set([r[len(TAG):] for r in refs if r.startswith(TAG)]) if not tags: # Either we're using git < 1.8.3, or there really are no tags. We use # a heuristic: assume all version tags have a digit. The old git %d # expansion behaves like git log --decorate=short and strips out the # refs/heads/ and refs/tags/ prefixes that would let us distinguish # between branches and tags. By ignoring refnames without digits, we # filter out many common branch names like "release" and # "stabilization", as well as "HEAD" and "master". tags = set([r for r in refs if re.search(r'\d', r)]) if verbose: print("discarding '%s', no digits" % ",".join(refs - tags)) if verbose: print("likely tags: %s" % ",".join(sorted(tags))) for ref in sorted(tags): # sorting will prefer e.g. "2.0" over "2.0rc1" if ref.startswith(tag_prefix): r = ref[len(tag_prefix):] if verbose: print("picking %s" % r) return {"version": r, "full-revisionid": keywords["full"].strip(), "dirty": False, "error": None, "date": date} # no suitable tags, so version is "0+unknown", but full hex is still there if verbose: print("no suitable tags, using unknown + full revision id") return {"version": "0+unknown", "full-revisionid": keywords["full"].strip(), "dirty": False, "error": "no suitable tags", "date": None} @register_vcs_handler("git", "pieces_from_vcs") def git_pieces_from_vcs(tag_prefix, root, verbose, run_command=run_command): """Get version from 'git describe' in the root of the source tree. This only gets called if the git-archive 'subst' keywords were *not* expanded, and _version.py hasn't already been rewritten with a short version string, meaning we're inside a checked out source tree. """ GITS = ["git"] if sys.platform == "win32": GITS = ["git.cmd", "git.exe"] out, rc = run_command(GITS, ["rev-parse", "--git-dir"], cwd=root, hide_stderr=True) if rc != 0: if verbose: print("Directory %s not under git control" % root) raise NotThisMethod("'git rev-parse --git-dir' returned error") # if there is a tag matching tag_prefix, this yields TAG-NUM-gHEX[-dirty] # if there isn't one, this yields HEX[-dirty] (no NUM) describe_out, rc = run_command(GITS, ["describe", "--tags", "--dirty", "--always", "--long", "--match", "%s*" % tag_prefix], cwd=root) # --long was added in git-1.5.5 if describe_out is None: raise NotThisMethod("'git describe' failed") describe_out = describe_out.strip() full_out, rc = run_command(GITS, ["rev-parse", "HEAD"], cwd=root) if full_out is None: raise NotThisMethod("'git rev-parse' failed") full_out = full_out.strip() pieces = {} pieces["long"] = full_out pieces["short"] = full_out[:7] # maybe improved later pieces["error"] = None # parse describe_out. It will be like TAG-NUM-gHEX[-dirty] or HEX[-dirty] # TAG might have hyphens. git_describe = describe_out # look for -dirty suffix dirty = git_describe.endswith("-dirty") pieces["dirty"] = dirty if dirty: git_describe = git_describe[:git_describe.rindex("-dirty")] # now we have TAG-NUM-gHEX or HEX if "-" in git_describe: # TAG-NUM-gHEX mo = re.search(r'^(.+)-(\d+)-g([0-9a-f]+)$', git_describe) if not mo: # unparseable. Maybe git-describe is misbehaving? pieces["error"] = ("unable to parse git-describe output: '%s'" % describe_out) return pieces # tag full_tag = mo.group(1) if not full_tag.startswith(tag_prefix): if verbose: fmt = "tag '%s' doesn't start with prefix '%s'" print(fmt % (full_tag, tag_prefix)) pieces["error"] = ("tag '%s' doesn't start with prefix '%s'" % (full_tag, tag_prefix)) return pieces pieces["closest-tag"] = full_tag[len(tag_prefix):] # distance: number of commits since tag pieces["distance"] = int(mo.group(2)) # commit: short hex revision ID pieces["short"] = mo.group(3) else: # HEX: no tags pieces["closest-tag"] = None count_out, rc = run_command(GITS, ["rev-list", "HEAD", "--count"], cwd=root) pieces["distance"] = int(count_out) # total number of commits # commit date: see ISO-8601 comment in git_versions_from_keywords() date = run_command(GITS, ["show", "-s", "--format=%ci", "HEAD"], cwd=root)[0].strip() pieces["date"] = date.strip().replace(" ", "T", 1).replace(" ", "", 1) return pieces def do_vcs_install(manifest_in, versionfile_source, ipy): """Git-specific installation logic for Versioneer. For Git, this means creating/changing .gitattributes to mark _version.py for export-subst keyword substitution. """ GITS = ["git"] if sys.platform == "win32": GITS = ["git.cmd", "git.exe"] files = [manifest_in, versionfile_source] if ipy: files.append(ipy) try: me = __file__ if me.endswith(".pyc") or me.endswith(".pyo"): me = os.path.splitext(me)[0] + ".py" versioneer_file = os.path.relpath(me) except NameError: versioneer_file = "versioneer.py" files.append(versioneer_file) present = False try: f = open(".gitattributes", "r") for line in f.readlines(): if line.strip().startswith(versionfile_source): if "export-subst" in line.strip().split()[1:]: present = True f.close() except EnvironmentError: pass if not present: f = open(".gitattributes", "a+") f.write("%s export-subst\n" % versionfile_source) f.close() files.append(".gitattributes") run_command(GITS, ["add", "--"] + files) def versions_from_parentdir(parentdir_prefix, root, verbose): """Try to determine the version from the parent directory name. Source tarballs conventionally unpack into a directory that includes both the project name and a version string. We will also support searching up two directory levels for an appropriately named parent directory """ rootdirs = [] for i in range(3): dirname = os.path.basename(root) if dirname.startswith(parentdir_prefix): return {"version": dirname[len(parentdir_prefix):], "full-revisionid": None, "dirty": False, "error": None, "date": None} else: rootdirs.append(root) root = os.path.dirname(root) # up a level if verbose: print("Tried directories %s but none started with prefix %s" % (str(rootdirs), parentdir_prefix)) raise NotThisMethod("rootdir doesn't start with parentdir_prefix") SHORT_VERSION_PY = """ # This file was generated by 'versioneer.py' (0.18) from # revision-control system data, or from the parent directory name of an # unpacked source archive. Distribution tarballs contain a pre-generated copy # of this file. import json version_json = ''' %s ''' # END VERSION_JSON def get_versions(): return json.loads(version_json) """ def versions_from_file(filename): """Try to determine the version from _version.py if present.""" try: with open(filename) as f: contents = f.read() except EnvironmentError: raise NotThisMethod("unable to read _version.py") mo = re.search(r"version_json = '''\n(.*)''' # END VERSION_JSON", contents, re.M | re.S) if not mo: mo = re.search(r"version_json = '''\r\n(.*)''' # END VERSION_JSON", contents, re.M | re.S) if not mo: raise NotThisMethod("no version_json in _version.py") return json.loads(mo.group(1)) def write_to_version_file(filename, versions): """Write the given version number to the given _version.py file.""" os.unlink(filename) contents = json.dumps(versions, sort_keys=True, indent=1, separators=(",", ": ")) with open(filename, "w") as f: f.write(SHORT_VERSION_PY % contents) print("set %s to '%s'" % (filename, versions["version"])) def plus_or_dot(pieces): """Return a + if we don't already have one, else return a .""" if "+" in pieces.get("closest-tag", ""): return "." return "+" def render_pep440(pieces): """Build up version string, with post-release "local version identifier". Our goal: TAG[+DISTANCE.gHEX[.dirty]] . Note that if you get a tagged build and then dirty it, you'll get TAG+0.gHEX.dirty Exceptions: 1: no tags. git_describe was just HEX. 0+untagged.DISTANCE.gHEX[.dirty] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += plus_or_dot(pieces) rendered += "%d.g%s" % (pieces["distance"], pieces["short"]) if pieces["dirty"]: rendered += ".dirty" else: # exception #1 rendered = "0+untagged.%d.g%s" % (pieces["distance"], pieces["short"]) if pieces["dirty"]: rendered += ".dirty" return rendered def render_pep440_pre(pieces): """TAG[.post.devDISTANCE] -- No -dirty. Exceptions: 1: no tags. 0.post.devDISTANCE """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"]: rendered += ".post.dev%d" % pieces["distance"] else: # exception #1 rendered = "0.post.dev%d" % pieces["distance"] return rendered def render_pep440_post(pieces): """TAG[.postDISTANCE[.dev0]+gHEX] . The ".dev0" means dirty. Note that .dev0 sorts backwards (a dirty tree will appear "older" than the corresponding clean one), but you shouldn't be releasing software with -dirty anyways. Exceptions: 1: no tags. 0.postDISTANCE[.dev0] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += ".post%d" % pieces["distance"] if pieces["dirty"]: rendered += ".dev0" rendered += plus_or_dot(pieces) rendered += "g%s" % pieces["short"] else: # exception #1 rendered = "0.post%d" % pieces["distance"] if pieces["dirty"]: rendered += ".dev0" rendered += "+g%s" % pieces["short"] return rendered def render_pep440_old(pieces): """TAG[.postDISTANCE[.dev0]] . The ".dev0" means dirty. Eexceptions: 1: no tags. 0.postDISTANCE[.dev0] """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"] or pieces["dirty"]: rendered += ".post%d" % pieces["distance"] if pieces["dirty"]: rendered += ".dev0" else: # exception #1 rendered = "0.post%d" % pieces["distance"] if pieces["dirty"]: rendered += ".dev0" return rendered def render_git_describe(pieces): """TAG[-DISTANCE-gHEX][-dirty]. Like 'git describe --tags --dirty --always'. Exceptions: 1: no tags. HEX[-dirty] (note: no 'g' prefix) """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] if pieces["distance"]: rendered += "-%d-g%s" % (pieces["distance"], pieces["short"]) else: # exception #1 rendered = pieces["short"] if pieces["dirty"]: rendered += "-dirty" return rendered def render_git_describe_long(pieces): """TAG-DISTANCE-gHEX[-dirty]. Like 'git describe --tags --dirty --always -long'. The distance/hash is unconditional. Exceptions: 1: no tags. HEX[-dirty] (note: no 'g' prefix) """ if pieces["closest-tag"]: rendered = pieces["closest-tag"] rendered += "-%d-g%s" % (pieces["distance"], pieces["short"]) else: # exception #1 rendered = pieces["short"] if pieces["dirty"]: rendered += "-dirty" return rendered def render(pieces, style): """Render the given version pieces into the requested style.""" if pieces["error"]: return {"version": "unknown", "full-revisionid": pieces.get("long"), "dirty": None, "error": pieces["error"], "date": None} if not style or style == "default": style = "pep440" # the default if style == "pep440": rendered = render_pep440(pieces) elif style == "pep440-pre": rendered = render_pep440_pre(pieces) elif style == "pep440-post": rendered = render_pep440_post(pieces) elif style == "pep440-old": rendered = render_pep440_old(pieces) elif style == "git-describe": rendered = render_git_describe(pieces) elif style == "git-describe-long": rendered = render_git_describe_long(pieces) else: raise ValueError("unknown style '%s'" % style) return {"version": rendered, "full-revisionid": pieces["long"], "dirty": pieces["dirty"], "error": None, "date": pieces.get("date")} class VersioneerBadRootError(Exception): """The project root directory is unknown or missing key files.""" def get_versions(verbose=False): """Get the project version from whatever source is available. Returns dict with two keys: 'version' and 'full'. """ if "versioneer" in sys.modules: # see the discussion in cmdclass.py:get_cmdclass() del sys.modules["versioneer"] root = get_root() cfg = get_config_from_root(root) assert cfg.VCS is not None, "please set [versioneer]VCS= in setup.cfg" handlers = HANDLERS.get(cfg.VCS) assert handlers, "unrecognized VCS '%s'" % cfg.VCS verbose = verbose or cfg.verbose assert cfg.versionfile_source is not None, \ "please set versioneer.versionfile_source" assert cfg.tag_prefix is not None, "please set versioneer.tag_prefix" versionfile_abs = os.path.join(root, cfg.versionfile_source) # extract version from first of: _version.py, VCS command (e.g. 'git # describe'), parentdir. This is meant to work for developers using a # source checkout, for users of a tarball created by 'setup.py sdist', # and for users of a tarball/zipball created by 'git archive' or github's # download-from-tag feature or the equivalent in other VCSes. get_keywords_f = handlers.get("get_keywords") from_keywords_f = handlers.get("keywords") if get_keywords_f and from_keywords_f: try: keywords = get_keywords_f(versionfile_abs) ver = from_keywords_f(keywords, cfg.tag_prefix, verbose) if verbose: print("got version from expanded keyword %s" % ver) return ver except NotThisMethod: pass try: ver = versions_from_file(versionfile_abs) if verbose: print("got version from file %s %s" % (versionfile_abs, ver)) return ver except NotThisMethod: pass from_vcs_f = handlers.get("pieces_from_vcs") if from_vcs_f: try: pieces = from_vcs_f(cfg.tag_prefix, root, verbose) ver = render(pieces, cfg.style) if verbose: print("got version from VCS %s" % ver) return ver except NotThisMethod: pass try: if cfg.parentdir_prefix: ver = versions_from_parentdir(cfg.parentdir_prefix, root, verbose) if verbose: print("got version from parentdir %s" % ver) return ver except NotThisMethod: pass if verbose: print("unable to compute version") return {"version": "0+unknown", "full-revisionid": None, "dirty": None, "error": "unable to compute version", "date": None} def get_version(): """Get the short version string for this project.""" return get_versions()["version"] def get_cmdclass(): """Get the custom setuptools/distutils subclasses used by Versioneer.""" if "versioneer" in sys.modules: del sys.modules["versioneer"] # this fixes the "python setup.py develop" case (also 'install' and # 'easy_install .'), in which subdependencies of the main project are # built (using setup.py bdist_egg) in the same python process. Assume # a main project A and a dependency B, which use different versions # of Versioneer. A's setup.py imports A's Versioneer, leaving it in # sys.modules by the time B's setup.py is executed, causing B to run # with the wrong versioneer. Setuptools wraps the sub-dep builds in a # sandbox that restores sys.modules to it's pre-build state, so the # parent is protected against the child's "import versioneer". By # removing ourselves from sys.modules here, before the child build # happens, we protect the child from the parent's versioneer too. # Also see https://github.com/warner/python-versioneer/issues/52 cmds = {} # we add "version" to both distutils and setuptools from distutils.core import Command class cmd_version(Command): description = "report generated version string" user_options = [] boolean_options = [] def initialize_options(self): pass def finalize_options(self): pass def run(self): vers = get_versions(verbose=True) print("Version: %s" % vers["version"]) print(" full-revisionid: %s" % vers.get("full-revisionid")) print(" dirty: %s" % vers.get("dirty")) print(" date: %s" % vers.get("date")) if vers["error"]: print(" error: %s" % vers["error"]) cmds["version"] = cmd_version # we override "build_py" in both distutils and setuptools # # most invocation pathways end up running build_py: # distutils/build -> build_py # distutils/install -> distutils/build ->.. # setuptools/bdist_wheel -> distutils/install ->.. # setuptools/bdist_egg -> distutils/install_lib -> build_py # setuptools/install -> bdist_egg ->.. # setuptools/develop -> ? # pip install: # copies source tree to a tempdir before running egg_info/etc # if .git isn't copied too, 'git describe' will fail # then does setup.py bdist_wheel, or sometimes setup.py install # setup.py egg_info -> ? # we override different "build_py" commands for both environments if "setuptools" in sys.modules: from setuptools.command.build_py import build_py as _build_py else: from distutils.command.build_py import build_py as _build_py class cmd_build_py(_build_py): def run(self): root = get_root() cfg = get_config_from_root(root) versions = get_versions() _build_py.run(self) # now locate _version.py in the new build/ directory and replace # it with an updated value if cfg.versionfile_build: target_versionfile = os.path.join(self.build_lib, cfg.versionfile_build) print("UPDATING %s" % target_versionfile) write_to_version_file(target_versionfile, versions) cmds["build_py"] = cmd_build_py if "cx_Freeze" in sys.modules: # cx_freeze enabled? from cx_Freeze.dist import build_exe as _build_exe # nczeczulin reports that py2exe won't like the pep440-style string # as FILEVERSION, but it can be used for PRODUCTVERSION, e.g. # setup(console=[{ # "version": versioneer.get_version().split("+", 1)[0], # FILEVERSION # "product_version": versioneer.get_version(), # ... class cmd_build_exe(_build_exe): def run(self): root = get_root() cfg = get_config_from_root(root) versions = get_versions() target_versionfile = cfg.versionfile_source print("UPDATING %s" % target_versionfile) write_to_version_file(target_versionfile, versions) _build_exe.run(self) os.unlink(target_versionfile) with open(cfg.versionfile_source, "w") as f: LONG = LONG_VERSION_PY[cfg.VCS] f.write(LONG % {"DOLLAR": "$", "STYLE": cfg.style, "TAG_PREFIX": cfg.tag_prefix, "PARENTDIR_PREFIX": cfg.parentdir_prefix, "VERSIONFILE_SOURCE": cfg.versionfile_source, }) cmds["build_exe"] = cmd_build_exe del cmds["build_py"] if 'py2exe' in sys.modules: # py2exe enabled? try: from py2exe.distutils_buildexe import py2exe as _py2exe # py3 except ImportError: from py2exe.build_exe import py2exe as _py2exe # py2 class cmd_py2exe(_py2exe): def run(self): root = get_root() cfg = get_config_from_root(root) versions = get_versions() target_versionfile = cfg.versionfile_source print("UPDATING %s" % target_versionfile) write_to_version_file(target_versionfile, versions) _py2exe.run(self) os.unlink(target_versionfile) with open(cfg.versionfile_source, "w") as f: LONG = LONG_VERSION_PY[cfg.VCS] f.write(LONG % {"DOLLAR": "$", "STYLE": cfg.style, "TAG_PREFIX": cfg.tag_prefix, "PARENTDIR_PREFIX": cfg.parentdir_prefix, "VERSIONFILE_SOURCE": cfg.versionfile_source, }) cmds["py2exe"] = cmd_py2exe # we override different "sdist" commands for both environments if "setuptools" in sys.modules: from setuptools.command.sdist import sdist as _sdist else: from distutils.command.sdist import sdist as _sdist class cmd_sdist(_sdist): def run(self): versions = get_versions() self._versioneer_generated_versions = versions # unless we update this, the command will keep using the old # version self.distribution.metadata.version = versions["version"] return _sdist.run(self) def make_release_tree(self, base_dir, files): root = get_root() cfg = get_config_from_root(root) _sdist.make_release_tree(self, base_dir, files) # now locate _version.py in the new base_dir directory # (remembering that it may be a hardlink) and replace it with an # updated value target_versionfile = os.path.join(base_dir, cfg.versionfile_source) print("UPDATING %s" % target_versionfile) write_to_version_file(target_versionfile, self._versioneer_generated_versions) cmds["sdist"] = cmd_sdist return cmds CONFIG_ERROR = """ setup.cfg is missing the necessary Versioneer configuration. You need a section like: [versioneer] VCS = git style = pep440 versionfile_source = src/myproject/_version.py versionfile_build = myproject/_version.py tag_prefix = parentdir_prefix = myproject- You will also need to edit your setup.py to use the results: import versioneer setup(version=versioneer.get_version(), cmdclass=versioneer.get_cmdclass(), ...) Please read the docstring in ./versioneer.py for configuration instructions, edit setup.cfg, and re-run the installer or 'python versioneer.py setup'. """ SAMPLE_CONFIG = """ # See the docstring in versioneer.py for instructions. Note that you must # re-run 'versioneer.py setup' after changing this section, and commit the # resulting files. [versioneer] #VCS = git #style = pep440 #versionfile_source = #versionfile_build = #tag_prefix = #parentdir_prefix = """ INIT_PY_SNIPPET = """ from ._version import get_versions __version__ = get_versions()['version'] del get_versions """ def do_setup(): """Main VCS-independent setup function for installing Versioneer.""" root = get_root() try: cfg = get_config_from_root(root) except (EnvironmentError, configparser.NoSectionError, configparser.NoOptionError) as e: if isinstance(e, (EnvironmentError, configparser.NoSectionError)): print("Adding sample versioneer config to setup.cfg", file=sys.stderr) with open(os.path.join(root, "setup.cfg"), "a") as f: f.write(SAMPLE_CONFIG) print(CONFIG_ERROR, file=sys.stderr) return 1 print(" creating %s" % cfg.versionfile_source) with open(cfg.versionfile_source, "w") as f: LONG = LONG_VERSION_PY[cfg.VCS] f.write(LONG % {"DOLLAR": "$", "STYLE": cfg.style, "TAG_PREFIX": cfg.tag_prefix, "PARENTDIR_PREFIX": cfg.parentdir_prefix, "VERSIONFILE_SOURCE": cfg.versionfile_source, }) ipy = os.path.join(os.path.dirname(cfg.versionfile_source), "__init__.py") if os.path.exists(ipy): try: with open(ipy, "r") as f: old = f.read() except EnvironmentError: old = "" if INIT_PY_SNIPPET not in old: print(" appending to %s" % ipy) with open(ipy, "a") as f: f.write(INIT_PY_SNIPPET) else: print(" %s unmodified" % ipy) else: print(" %s doesn't exist, ok" % ipy) ipy = None # Make sure both the top-level "versioneer.py" and versionfile_source # (PKG/_version.py, used by runtime code) are in MANIFEST.in, so # they'll be copied into source distributions. Pip won't be able to # install the package without this. manifest_in = os.path.join(root, "MANIFEST.in") simple_includes = set() try: with open(manifest_in, "r") as f: for line in f: if line.startswith("include "): for include in line.split()[1:]: simple_includes.add(include) except EnvironmentError: pass # That doesn't cover everything MANIFEST.in can do # (http://docs.python.org/2/distutils/sourcedist.html#commands), so # it might give some false negatives. Appending redundant 'include' # lines is safe, though. if "versioneer.py" not in simple_includes: print(" appending 'versioneer.py' to MANIFEST.in") with open(manifest_in, "a") as f: f.write("include versioneer.py\n") else: print(" 'versioneer.py' already in MANIFEST.in") if cfg.versionfile_source not in simple_includes: print(" appending versionfile_source ('%s') to MANIFEST.in" % cfg.versionfile_source) with open(manifest_in, "a") as f: f.write("include %s\n" % cfg.versionfile_source) else: print(" versionfile_source already in MANIFEST.in") # Make VCS-specific changes. For git, this means creating/changing # .gitattributes to mark _version.py for export-subst keyword # substitution. do_vcs_install(manifest_in, cfg.versionfile_source, ipy) return 0 def scan_setup_py(): """Validate the contents of setup.py against Versioneer's expectations.""" found = set() setters = False errors = 0 with open("setup.py", "r") as f: for line in f.readlines(): if "import versioneer" in line: found.add("import") if "versioneer.get_cmdclass()" in line: found.add("cmdclass") if "versioneer.get_version()" in line: found.add("get_version") if "versioneer.VCS" in line: setters = True if "versioneer.versionfile_source" in line: setters = True if len(found) != 3: print("") print("Your setup.py appears to be missing some important items") print("(but I might be wrong). Please make sure it has something") print("roughly like the following:") print("") print(" import versioneer") print(" setup( version=versioneer.get_version(),") print(" cmdclass=versioneer.get_cmdclass(), ...)") print("") errors += 1 if setters: print("You should remove lines like 'versioneer.VCS = ' and") print("'versioneer.versionfile_source = ' . This configuration") print("now lives in setup.cfg, and should be removed from setup.py") print("") errors += 1 return errors if __name__ == "__main__": cmd = sys.argv[1] if cmd == "setup": errors = do_setup() errors += scan_setup_py() if errors: sys.exit(1)