carrot-0.10.7/000755 000765 000024 00000000000 11453345106 013014 5ustar00askstaff000000 000000 carrot-0.10.7/AUTHORS000644 000765 000024 00000001263 11444077126 014072 0ustar00askstaff000000 000000 ================================== AUTHORS (in chronological order) ================================== Ask Solem Travis Cline Rune Halvorsen Sean Creeley Jason Cater Ian Struble Patrick Schneider Travis Swicegood Stephen Day Andrew Watts Paul McLanahan Ralf Nyren Jeff Balogh Adam Wentz Vincent Driessen Matt Swanson Christian Legnitto carrot-0.10.7/carrot/000755 000765 000024 00000000000 11453345106 014306 5ustar00askstaff000000 000000 carrot-0.10.7/carrot.egg-info/000755 000765 000024 00000000000 11453345106 016000 5ustar00askstaff000000 000000 carrot-0.10.7/Changelog000644 000765 000024 00000043174 11453344754 014647 0ustar00askstaff000000 000000 ================ Change history ================ 0.10.7 [2010-10-07 03:20 P.M CEST] ---------------------------------- * ``ConsumerSet``: Now possible to add/cancel consumers at runtime * To add another consumer to an active set do:: >>> cset.add_consumer(C) >>> # or >>> cset.add_consumer_from_dict(**declaration) >>> # ... >>> # consume() will declare new consumers >>> cset.consume() * To cancel an active consumer by queue name do:: >>> cset.cancel_by_queue(queue_name) 0.10.6 [2010-09-03 10:16 A.M CEST] ---------------------------------- * ``Publisher.send``: Now supports an exchange argument used to override the exchange to send the message to (defaults to ``Publisher.exchange``). Note that this exchange must have been declared. * STOMP backend: Now supports username and password authentication. * pika backend called basic_get with incorrect arguments. 0.10.5 [2010-06-03 09:02 A.M CEST] ---------------------------------- * In-memory backend: discard_all() now works correctly. * Added msgpack serialization support See http://msgpack.sourceforge.net for more information. To enable set:: serializer="msgpack" * Added dependency specification for building RPMs. $ python setup.py bdist_rpm 0.10.4 [2010-05-14 10:26 A.M CEST] ---------------------------------- * Added ``BrokerConnection.drain_events()`` (only works for amqplib/pika) `drain_events` waits for events on all active channels. * amqplib: Added timeout support to ``drain_events``. Example usage: >>> c = Consumer() >>> it = c.iterconsume() >>> # wait for event on any channel >>> try: ... connection.drain_events(timeout=1) ... except socket.timeout: pass * Added Consumer.consume / ConsumerSet.consume We're slowly moving away from ``iterconsume`` as this flow doesn't make sense. You often want to consume from several channels at once, so ``iterconsume`` will probably be deprecated at some point. "new-style" consume example:: >>> connection = BrokerConnection(..) >>> consumer = Consumer(connection, ...) >>> consumer.register_callback(handle_message) >>> consumer.consume() # declares consumers >>> while True: ... connection.drain_events() >>> consumer.cancel() # Cancel consumer. More elaborate example consuming from two channels, where the first channel consumes from multiple queues:: >>> connection = BrokerConnection(..) # The first channel receives jobs from several # queues. >>> queues = {"video": {"exchange": "jobs", ... "queue": "video", ... "routing_key": "video"}, ... "image": {"exchange": "jobs", ... "queue": "image", ... "routing_key": "image"}} >>> job_consumer = ConsumerSet(connection, from_dict=queues) >>> job_consumer.register_callback(handle_job) >>> job_consumer.consume() # The second channel receives remote control commands. >>> remote_consumer = Consumer(connection, queue="control", ... exchange="control") >>> remote_consumer.register_callback(handle_remote_command) >>> remote_consumer.consume() # The eventloop. # Receives a single message a pass and calls the appropriate # callback. >>> try: ... while True: ... connection.drain_events() ... finally: ... job_consumer.close() ... remote_consumer.close() ... connection.close() * amqplib: now raises ``KeyError`` if hostname isn't set. 0.10.3 [2010-03-08 05:01 P.M CEST] ---------------------------------- * Consumer/Publisher: A kwarg option set to ``None`` should always mean "use the class default value". This was not always the case, but has been fixed in this release. * DjangoBrokerConnection: Now accepts a custom ``settings`` object. E.g.: >>> conn = DjangoBrokerConnection(settings=mysettings) * Consumer.iterconsume: Now raises :exc:`StopIteration` if the channel is closed. (http://github.com/ask/carrot/issues/issue/24) * Fixed syntax error in the DjangoBrokerConnection which could be triggered if some conditions were met. * setup.cfg: Disable --enable-coverage from nosetests section * Consumer.iterconsume now works properly when using the Python Queue module based backend (http://github.com/ask/carrot/issues/issue/23). 0.10.2 [2010-02-03 11:43 A.M CEST] ---------------------------------- * Resolved a typo in the experimental Pika backend. 0.10.1 [2010-01-16 10:17 P.M CEST] ---------------------------------- * Fixed syntax error typo in the Pika backend. 0.10.0 [2010-01-15 12:08 A.M CEST] ---------------------------------- * Added experimental Pika backend for async support. See http://github.com/tonyg/pika * Python 2.4 compatibility. * Added intent revealing names for use with the delivery_mode attribute * The amqplib internal connection now supports waiting for events on any channel, so as to not block the event loop on a single channel. Example: >>> from carrot.connection import BrokerConnection >>> from carrot.messaging import Consumer, Publisher >>> from functools import partial >>> connection = BrokerConnection(...) >>> consumer1 = Consumer(queue="foo", exchange="foo") >>> consumer2 = Consumer(queue="bar", exchange="bar") >>> def callback(channel, message_data, message): ... print(%s: %s" % (channel, message_data)) >>> consumer1.register_callback(partial(callback, "channel1")) >>> consumer2.register_callback(partial(callback, "channel2")) >>> pub1 = Publisher(exchange="foo") >>> pub2 = Publisher(exchange="bar") >>> [(i % 2 and pub1 or pub2).send({"hello": "world"}) ... for i in range(100)] >>> while True: ... connection.connection.drain_events() But please be sure to note that this is an internal feature only, hopefully we will have a public interface for this for v1.0. 0.8.0 [2009-11-16 05:11 P.M CEST] --------------------------------- **BACKWARD INCOMPATIBLE CHANGES** * Django: ``AMQP_SERVER`` has been deprecated and renamed to ``BROKER_HOST``. ``AMQP_SERVER`` is still supported but will be removed in version 1.0. * Django: All ``AMQP_*`` settings has been renamed to ``BROKER_*``, the old settings still work but gives a deprecation warning. ``AMQP_*`` is scheduled for removal in v1.0. * You now have to include the class name in the CARROT_BACKEND string. E.g. where you before had "my.module.backends", you now need to include the class name: ``"my.module.backends.BackendClass"``. The aliases works like before. *BUGFIXES* * Cancel should delete the affected _consumer * qos()/flow() now working properly. * Fixed the bug in UUID4 which makes it return the same id over and over. *OTHER* * Sphinx docs: Remove django dependency when building docs. Thanks jetfar! * You can now build the documentatin using ``python setup.py build_sphinx`` (thanks jetfar!) 0.6.1 [2009-09-30 12:29 P.M CET] ------------------------------------------------------------------ * Forgot to implement qos/flow in the pyamqplib backend (doh). Big thanks to stevvooe! See issue `#18`_ * Renamed ConsumerSet._open_channels -> _open_consumers * Channel is now a weak reference so it's collected on channel exception. See issue `#17`_. Thanks to ltucker. * Publisher.auto_declare/Consumer.auto_declare: Can now disable the default behaviour of automatically declaring the queue, exchange and queue binding. * Need to close the connection to receive mandatory/immediate errors (related to issue `#16`_). Thanks to ltucker. * pyamqplib backend close didn't work properly, typo channel -> _channel * Adds carrot.backends.pystomp to the reference documentation. .. _`#18`: http://github.com/ask/carrot/issues#issue/18 .. _`#17`: http://github.com/ask/carrot/issues#issue/17 .. _`#16`: http://github.com/ask/carrot/issues#issue/16 0.6.0 [2009-09-17 16:41 P.M CET] ------------------------------------------------------------------ **BACKWARD INCOMPATIBLE CHANGES** * AMQPConnection renamed to BrokerConnection with AMQPConnection remaining an alias for backwards compatability. Similarly DjangoAMQPConnection is renamed to DjangoBrokerConnection. * AMQPConnection renamed to BrokerConnection DjangoAMQPConnection renamed to DjangoBrokerConnection (The previous names are still available but will be deprecated in 1.0) * The connection is now lazy, requested only when it's needed. To explicitly connect you have to evaluate the BrokerConnections ``connection`` attribute. >>> connection = BrokerConnection(...) # Not connected yet. >>> connection.connection; # Now it's connected * A channel is now lazy, requested only when it's needed. * pyamqplib.Message.amqp_message is now a private attribute **NEW FEATURES** * Basic support for STOMP using the stompy library. (Available at http://bitbucket.org/benjaminws/python-stomp/overview/) * Implements :meth:`Consumer.qos` and :meth:`Consumer.flow` for setting quality of service and flow control. **NEWS** * The current Message class is now available as an attribute on the Backend. * Default port is now handled by the backend and all AMQP_* settings to the DjangoBrokerConnection is now optional * Backend is now initialized in the connection instead of Consumer/Publisher, so backend_cls now has to be sent to AMQPConnection if you want to explicitly set it. * Specifying utf-8 as the content type when forcing unicode into a string. This removes the reference to the unbound content_type variable. 0.5.1 [2009-07-19 06:19 P.M CET] ------------------------------------------------------------------ * Handle messages without content_encoding attribute set. * Make delivery_info available on the Message instance. * Use anyjson to detect best installed JSON module on the system. 0.5.0 [2009-06-25 08:16 P.M CET] ------------------------------------------------------------------ **BACKWARD-INCOMPATIBLE CHANGES** * Custom encoder/decoder support has been moved to a centralized registry in ``carrot.serialization``. This means the ``encoder``/``decoder`` optional arguments to ``Publisher.send`` `(and the similar attributes of ``Publisher`` and ``Consumer`` classes) no longer exist. See ``README`` for details of the new system. * Any ``Consumer`` and ``Publisher`` instances should be upgraded at the same time since carrot now uses the AMQP ``content-type`` field to know how to automatically de-serialize your data. If you use an old-style ``Publisher`` with a new-style ``Consumer``, you will get a raw string back as ``message_data`` instead of your de-serialized data. An old-style ``Consumer`` will work with a new-style ``Publisher`` as long as you're using the default ``JSON`` serialization methods. * Acknowledging/Rejecting/Requeuing a message twice now raises an exception. *ENHANCEMENTS* * ``ConsumerSet``: Receive messages from several consumers. 0.4.5 [2009-06-15 01:58 P.M CET] ------------------------------------------------------------------ **BACKWARD-INCOMPATIBLE CHANGES** * the exchange is now also declared in the ``Publisher``. This means the following attributes (if used) must be set on *both* the ``Publisher`` and the ``Consumer``: ``exchange_type``, ``durable`` and ``auto_delete``. **IMPORTANT BUGS** * No_ack was always set to ``True`` when using ``Consumer.iterconsume``. 0.4.4 [2009-06-15 01:58 P.M CET] ------------------------------------------------------------------ * __init__.pyc was included in the distribution by error. 0.4.3 [2009-06-13 09:26 P.M CET] ------------------------------------------------------------------ * Fix typo with long_description in ``setup.py``. 0.4.2 [2009-06-13 08:30 P.M CET] ------------------------------------------------------------------ * Make sure README exists before reading it for ``long_description``. Thanks to jcater. * ``discard_all``: Use ``AMQP.queue_purge`` if ``filterfunc`` is not specified 0.4.1 [2009-06-08 04:21 P.M CET] ------------------------------------------------------------------ * Consumer/Publisher now correctly sets the encoder/decoder if they have been overriden by setting the class attribute. 0.4.0 [2009-06-06 01:39 P.M CET] ------------------------------------------------------------------ **IMPORTANT** Please don't use ``Consumer.wait`` in production. Use either of ``Consumer.iterconsume`` or ``Consumer.iterqueue``. **IMPORTANT** The ``ack`` argument to ``Consumer.process_next`` has been removed, use the instance-wide ``auto_ack`` argument/attribute instead. **IMPORTANT** ``Consumer.message_to_python`` has been removed, use ``message.decode()`` on the returned message object instead. **IMPORTANT** Consumer/Publisher/Messaging now no longer takes a backend instance, but a backend class, so the ``backend`` argument is renamed to ``backend_cls``. *WARNING* ``Consumer.process_next`` has been deprecated in favor of ``Consumer.fetch(enable_callbacks=True)`` and emits a ``DeprecationWarning`` if used. * ``Consumer.iterconsume``: New sane way to use basic_consume instead of ``Consumer.wait``: (Consmer.wait now uses this behind the scenes, just wrapped around a highly unsafe infinite loop.) * Consumer: New options: ``auto_ack`` and ``no_ack``. Auto ack means the consumer will automatically acknowledge new messages, and No-Ack means it will disable acknowledgement on the server alltogether (probably not what you want) * ``Consumer.iterqueue``: Now supports infinite argument, which means the iterator doesn't raise ``StopIteration`` if there's no messages, but instead yields ``None`` (thus never ends) * message argument to consumer callbacks is now a ``carrot.backends.pyamqplib.Message`` instance. See `[GH #4]`_. Thanks gregoirecachet! .. _`[GH #4]`: http://github.com/ask/carrot/issues/closed/#issue/4 * AMQPConnection, Consumer, Publisher and Messaging now supports the with statement. They automatically close when the with-statement block exists. * Consumer tags are now automatically generated for each class instance, so you should never get the "consumer tag already open" error anymore. * Loads of new unit tests. 0.4.0-pre7 [2009-06-03 05:08 P.M CET] ------------------------------------------------------------------ * Conform to pep8.py trying to raise our pypants score. * Documentation cleanup (thanks Rune Halvorsen) 0.4.0-pre6 [2009-06-03 04:55 P.M CET] ------------------------------------------------------------------ * exclusive implies auto_delete, not durable. Closes #2. Thanks gregoirecachet * Consumer tags are now automatically generated by the class module, name and a UUID. * New attribute ``Consumer.warn_if_exists:`` If True, emits a warning if the queue has already been declared. If a queue already exists, and you try to redeclare the queue with new settings, the new settings will be silently ignored, so this can be useful if you've recently changed the `routing_key` attribute or other settings. 0.4.0-pre3 [2009-05-29 02:27 P.M CET] ------------------------------------------------------------------ * Publisher.send: Now supports message priorities (a number between ``0`` and ``9``) * Publihser.send: Added ``routing_key`` keyword argument. Can override the routing key for a single message. * Publisher.send: Support for the ``immediate`` and ``mandatory`` flags. 0.4.0-pre2 [2009-05-29 02:27 P.M CET] ------------------------------------------------------------------ * AMQPConnection: Added ``connect_timeout`` timeout option, which is the timeout in seconds before we exhaust trying to establish a connection to the AMQP server. 0.4.0-pre1 [2009-05-27 04:27 P.M CET] ------------------------------------------------------------------ * This version introduces backends. The consumers and publishers all have an associated backend. Currently there are two backends available; ``pyamqlib`` and ``pyqueue``. The ``amqplib`` backend is for production use, while the ``Queue`` backend is for use while unit testing. * Consumer/Publisher operations no longer re-establishes the connection if the connection has been closed. * ``Consumer.next`` has been deprecated for a while, and has now been removed. * Message objects now have a ``decode`` method, to deserialize the message body. * You can now use the Consumer class standalone, without subclassing, by registering callbacks by using ``Consumer.register_callback``. * Ability to filter messages in ``Consumer.discard_all``. * carrot now includes a basic unit test suite, which hopefully will be more complete in later releases. * carrot now uses the Sphinx documentation system. 0.3.9 [2009-05-18 04:49 P.M CET] -------------------------------------------------------------- * Consumer.wait() now works properly again. Thanks Alexander Solovyov! 0.3.8 [2009-05-11 02:14 P.M CET] -------------------------------------------------------------- * Rearranged json module import order. New order is cjson > simplejson > json > django.util.somplejson * _Backwards incompatible change: Have to instantiate AMQPConnection object before passing it to consumers/publishers. e.g before when you did >>> consumer = Consumer(connection=DjangoAMQPConnection) you now have to do >>> consumer = Consumer(connection=DjangoAMQPConnection()) or sometimes you might even want to share the same connection with publisher/consumer. 0.2.1 [2009-03-24 05:48 P.M CET] -------------------------------------------------------------- * Fix typo "package" -> "packages" in setup.py 0.2.0 [2009-03-24 05:23 P.M ]` -------------------------------------------------------------- * hasty bugfix release, fixed syntax errors. 0.1.0 [2009-03-24 05:16 P.M ]` -------------------------------------------------------------- * Initial release carrot-0.10.7/docs/000755 000765 000024 00000000000 11453345106 013744 5ustar00askstaff000000 000000 carrot-0.10.7/FAQ000644 000765 000024 00000001066 11334033664 013352 0ustar00askstaff000000 000000 ============================ Frequently Asked Questions ============================ Questions ========= Q: Message.reject doesn't work? -------------------------------------- **Answer**: RabbitMQ (as of v1.5.5) has not implemented reject yet. There was a brief discussion about it on their mailing list, and the reason why it's not implemented yet is revealed: http://lists.rabbitmq.com/pipermail/rabbitmq-discuss/2009-January/003183.html Q: Message.requeue doesn't work? -------------------------------------- **Answer**: See _`Message.reject doesn't work?` carrot-0.10.7/INSTALL000644 000765 000024 00000000606 11334033664 014050 0ustar00askstaff000000 000000 Installation ============ You can install ``carrot`` either via the Python Package Index (PyPI) or from source. To install using ``pip``,:: $ pip install carrot To install using ``easy_install``,:: $ easy_install carrot If you have downloaded a source tarball you can install it by doing the following,:: $ python setup.py build # python setup.py install # as root carrot-0.10.7/LICENSE000644 000765 000024 00000002722 11334033664 014025 0ustar00askstaff000000 000000 Copyright (c) 2009, Ask Solem All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. Neither the name of Ask Solem nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. carrot-0.10.7/MANIFEST.in000644 000765 000024 00000000361 11334033664 014553 0ustar00askstaff000000 000000 include AUTHORS include Changelog include FAQ include INSTALL include LICENSE include MANIFEST.in include README.rst include README include THANKS include TODO recursive-include docs * recursive-include carrot *.py recursive-include tests * carrot-0.10.7/PKG-INFO000644 000765 000024 00000040555 11453345106 014122 0ustar00askstaff000000 000000 Metadata-Version: 1.0 Name: carrot Version: 0.10.7 Summary: AMQP Messaging Framework for Python Home-page: http://github.com/ask/carrot/ Author: Ask Solem Author-email: ask@celeryproject.org License: UNKNOWN Description: ############################################## carrot - AMQP Messaging Framework for Python ############################################## :Version: 0.10.7 **NOTE** This release contains backward-incompatible changes. Please read the `Changelog`_ for more information. .. _`Changelog`: http://ask.github.com/carrot/changelog.html Introduction ------------ `carrot` is an `AMQP`_ messaging queue framework. AMQP is the Advanced Message Queuing Protocol, an open standard protocol for message orientation, queuing, routing, reliability and security. The aim of `carrot` is to make messaging in Python as easy as possible by providing a high-level interface for producing and consuming messages. At the same time it is a goal to re-use what is already available as much as possible. `carrot` has pluggable messaging back-ends, so it is possible to support several messaging systems. Currently, there is support for `AMQP`_ (`py-amqplib`_, `pika`_), `STOMP`_ (`python-stomp`_). There's also an in-memory backend for testing purposes, using the `Python queue module`_. Several AMQP message broker implementations exists, including `RabbitMQ`_, `ZeroMQ`_ and `Apache ActiveMQ`_. You'll need to have one of these installed, personally we've been using `RabbitMQ`_. Before you start playing with ``carrot``, you should probably read up on AMQP, and you could start with the excellent article about using RabbitMQ under Python, `Rabbits and warrens`_. For more detailed information, you can refer to the `Wikipedia article about AMQP`_. .. _`RabbitMQ`: http://www.rabbitmq.com/ .. _`ZeroMQ`: http://www.zeromq.org/ .. _`AMQP`: http://amqp.org .. _`STOMP`: http://stomp.codehaus.org .. _`python-stomp`: http://bitbucket.org/asksol/python-stomp .. _`Python Queue module`: http://docs.python.org/library/queue.html .. _`Apache ActiveMQ`: http://activemq.apache.org/ .. _`Django`: http://www.djangoproject.com/ .. _`Rabbits and warrens`: http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/ .. _`py-amqplib`: http://barryp.org/software/py-amqplib/ .. _`pika`: http://github.com/tonyg/pika .. _`Wikipedia article about AMQP`: http://en.wikipedia.org/wiki/AMQP Documentation ------------- Carrot is using Sphinx, and the latest documentation is available at GitHub: http://github.com/ask/carrot/ Installation ============ You can install ``carrot`` either via the Python Package Index (PyPI) or from source. To install using ``pip``,:: $ pip install carrot To install using ``easy_install``,:: $ easy_install carrot If you have downloaded a source tarball you can install it by doing the following,:: $ python setup.py build # python setup.py install # as root Terminology =========== There are some concepts you should be familiar with before starting: * Publishers Publishers sends messages to an exchange. * Exchanges Messages are sent to exchanges. Exchanges are named and can be configured to use one of several routing algorithms. The exchange routes the messages to consumers by matching the routing key in the message with the routing key the consumer provides when binding to the exchange. * Consumers Consumers declares a queue, binds it to a exchange and receives messages from it. * Queues Queues receive messages sent to exchanges. The queues are declared by consumers. * Routing keys Every message has a routing key. The interpretation of the routing key depends on the exchange type. There are four default exchange types defined by the AMQP standard, and vendors can define custom types (so see your vendors manual for details). These are the default exchange types defined by AMQP/0.8: * Direct exchange Matches if the routing key property of the message and the ``routing_key`` attribute of the consumer are identical. * Fan-out exchange Always matches, even if the binding does not have a routing key. * Topic exchange Matches the routing key property of the message by a primitive pattern matching scheme. The message routing key then consists of words separated by dots (``"."``, like domain names), and two special characters are available; star (``"*"``) and hash (``"#"``). The star matches any word, and the hash matches zero or more words. For example ``"*.stock.#"`` matches the routing keys ``"usd.stock"`` and ``"eur.stock.db"`` but not ``"stock.nasdaq"``. Examples ======== Creating a connection --------------------- You can set up a connection by creating an instance of ``carrot.messaging.BrokerConnection``, with the appropriate options for your broker: >>> from carrot.connection import BrokerConnection >>> conn = BrokerConnection(hostname="localhost", port=5672, ... userid="test", password="test", ... virtual_host="test") If you're using Django you can use the ``carrot.connection.DjangoBrokerConnection`` class instead, which loads the connection settings from your ``settings.py``:: BROKER_HOST = "localhost" BROKER_PORT = 5672 BROKER_USER = "test" BROKER_PASSWORD = "secret" BROKER_VHOST = "/test" Then create a connection by doing: >>> from carrot.connection import DjangoBrokerConnection >>> conn = DjangoBrokerConnection() Receiving messages using a Consumer ----------------------------------- First we open up a Python shell and start a message consumer. This consumer declares a queue named ``"feed"``, receiving messages with the routing key ``"importer"`` from the ``"feed"`` exchange. The example then uses the consumers ``wait()`` method to go into consume mode, where it continuously polls the queue for new messages, and when a message is received it passes the message to all registered callbacks. >>> from carrot.messaging import Consumer >>> consumer = Consumer(connection=conn, queue="feed", ... exchange="feed", routing_key="importer") >>> def import_feed_callback(message_data, message): ... feed_url = message_data["import_feed"] ... print("Got feed import message for: %s" % feed_url) ... # something importing this feed url ... # import_feed(feed_url) ... message.ack() >>> consumer.register_callback(import_feed_callback) >>> consumer.wait() # Go into the consumer loop. Sending messages using a Publisher ---------------------------------- Then we open up another Python shell to send some messages to the consumer defined in the last section. >>> from carrot.messaging import Publisher >>> publisher = Publisher(connection=conn, ... exchange="feed", routing_key="importer") >>> publisher.send({"import_feed": "http://cnn.com/rss/edition.rss"}) >>> publisher.close() Look in the first Python shell again (where ``consumer.wait()`` is running), where the following text has been printed to the screen:: Got feed import message for: http://cnn.com/rss/edition.rss Serialization of Data ----------------------- By default every message is encoded using `JSON`_, so sending Python data structures like dictionaries and lists works. `YAML`_, `msgpack`_ and Python's built-in ``pickle`` module is also supported, and if needed you can register any custom serialization scheme you want to use. .. _`JSON`: http://www.json.org/ .. _`YAML`: http://yaml.org/ .. _`msgpack`: http://msgpack.sourceforge.net/ Each option has its advantages and disadvantages. ``json`` -- JSON is supported in many programming languages, is now a standard part of Python (since 2.6), and is fairly fast to decode using the modern Python libraries such as ``cjson or ``simplejson``. The primary disadvantage to ``JSON`` is that it limits you to the following data types: strings, unicode, floats, boolean, dictionaries, and lists. Decimals and dates are notably missing. Also, binary data will be transferred using base64 encoding, which will cause the transferred data to be around 34% larger than an encoding which supports native binary types. However, if your data fits inside the above constraints and you need cross-language support, the default setting of ``JSON`` is probably your best choice. ``pickle`` -- If you have no desire to support any language other than Python, then using the ``pickle`` encoding will gain you the support of all built-in Python data types (except class instances), smaller messages when sending binary files, and a slight speedup over ``JSON`` processing. ``yaml`` -- YAML has many of the same characteristics as ``json``, except that it natively supports more data types (including dates, recursive references, etc.) However, the Python libraries for YAML are a good bit slower than the libraries for JSON. If you need a more expressive set of data types and need to maintain cross-language compatibility, then ``YAML`` may be a better fit than the above. To instruct carrot to use an alternate serialization method, use one of the following options. 1. Set the serialization option on a per-Publisher basis: >>> from carrot.messaging import Publisher >>> publisher = Publisher(connection=conn, ... exchange="feed", routing_key="importer", ... serializer="yaml") 2. Set the serialization option on a per-call basis >>> from carrot.messaging import Publisher >>> publisher = Publisher(connection=conn, ... exchange="feed", routing_key="importer") >>> publisher.send({"import_feed": "http://cnn.com/rss/edition.rss"}, ... serializer="pickle") >>> publisher.close() Note that ``Consumer``s do not need the serialization method specified in their code. They can auto-detect the serialization method since we supply the ``Content-type`` header as part of the AMQP message. Sending raw data without Serialization --------------------------------------- In some cases, you don't need your message data to be serialized. If you pass in a plain string or unicode object as your message, then carrot will not waste cycles serializing/deserializing the data. You can optionally specify a ``content_type`` and ``content_encoding`` for the raw data: >>> from carrot.messaging import Publisher >>> publisher = Publisher(connection=conn, ... exchange="feed", routing_key="import_pictures") >>> publisher.send(open('~/my_picture.jpg','rb').read(), content_type="image/jpeg", content_encoding="binary") >>> publisher.close() The ``message`` object returned by the ``Consumer`` class will have a ``content_type`` and ``content_encoding`` attribute. Receiving messages without a callback -------------------------------------- You can also poll the queue manually, by using the ``fetch`` method. This method returns a ``Message`` object, from where you can get the message body, de-serialize the body to get the data, acknowledge, reject or re-queue the message. >>> consumer = Consumer(connection=conn, queue="feed", ... exchange="feed", routing_key="importer") >>> message = consumer.fetch() >>> if message: ... message_data = message.payload ... message.ack() ... else: ... # No messages waiting on the queue. >>> consumer.close() Sub-classing the messaging classes ---------------------------------- The ``Consumer``, and ``Publisher`` classes can also be sub classed. Thus you can define the above publisher and consumer like so: >>> from carrot.messaging import Publisher, Consumer >>> class FeedPublisher(Publisher): ... exchange = "feed" ... routing_key = "importer" ... ... def import_feed(self, feed_url): ... return self.send({"action": "import_feed", ... "feed_url": feed_url}) >>> class FeedConsumer(Consumer): ... queue = "feed" ... exchange = "feed" ... routing_key = "importer" ... ... def receive(self, message_data, message): ... action = message_data["action"] ... if action == "import_feed": ... # something importing this feed ... # import_feed(message_data["feed_url"]) message.ack() ... else: ... raise Exception("Unknown action: %s" % action) >>> publisher = FeedPublisher(connection=conn) >>> publisher.import_feed("http://cnn.com/rss/edition.rss") >>> publisher.close() >>> consumer = FeedConsumer(connection=conn) >>> consumer.wait() # Go into the consumer loop. Getting Help ============ Mailing list ------------ Join the `carrot-users`_ mailing list. .. _`carrot-users`: http://groups.google.com/group/carrot-users/ Bug tracker =========== If you have any suggestions, bug reports or annoyances please report them to our issue tracker at http://github.com/ask/carrot/issues/ Contributing ============ Development of ``carrot`` happens at Github: http://github.com/ask/carrot You are highly encouraged to participate in the development. If you don't like Github (for some reason) you're welcome to send regular patches. License ======= This software is licensed under the ``New BSD License``. See the ``LICENSE`` file in the top distribution directory for the full license text. Platform: any Classifier: Development Status :: 4 - Beta Classifier: Framework :: Django Classifier: Operating System :: OS Independent Classifier: Programming Language :: Python Classifier: License :: OSI Approved :: BSD License Classifier: Intended Audience :: Developers Classifier: Topic :: Communications Classifier: Topic :: System :: Distributed Computing Classifier: Topic :: Software Development :: Libraries :: Python Modules carrot-0.10.7/README000644 000765 000024 00000033307 11453345003 013676 0ustar00askstaff000000 000000 ############################################## carrot - AMQP Messaging Framework for Python ############################################## :Version: 0.10.7 **NOTE** This release contains backward-incompatible changes. Please read the `Changelog`_ for more information. .. _`Changelog`: http://ask.github.com/carrot/changelog.html Introduction ------------ `carrot` is an `AMQP`_ messaging queue framework. AMQP is the Advanced Message Queuing Protocol, an open standard protocol for message orientation, queuing, routing, reliability and security. The aim of `carrot` is to make messaging in Python as easy as possible by providing a high-level interface for producing and consuming messages. At the same time it is a goal to re-use what is already available as much as possible. `carrot` has pluggable messaging back-ends, so it is possible to support several messaging systems. Currently, there is support for `AMQP`_ (`py-amqplib`_, `pika`_), `STOMP`_ (`python-stomp`_). There's also an in-memory backend for testing purposes, using the `Python queue module`_. Several AMQP message broker implementations exists, including `RabbitMQ`_, `ZeroMQ`_ and `Apache ActiveMQ`_. You'll need to have one of these installed, personally we've been using `RabbitMQ`_. Before you start playing with ``carrot``, you should probably read up on AMQP, and you could start with the excellent article about using RabbitMQ under Python, `Rabbits and warrens`_. For more detailed information, you can refer to the `Wikipedia article about AMQP`_. .. _`RabbitMQ`: http://www.rabbitmq.com/ .. _`ZeroMQ`: http://www.zeromq.org/ .. _`AMQP`: http://amqp.org .. _`STOMP`: http://stomp.codehaus.org .. _`python-stomp`: http://bitbucket.org/asksol/python-stomp .. _`Python Queue module`: http://docs.python.org/library/queue.html .. _`Apache ActiveMQ`: http://activemq.apache.org/ .. _`Django`: http://www.djangoproject.com/ .. _`Rabbits and warrens`: http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/ .. _`py-amqplib`: http://barryp.org/software/py-amqplib/ .. _`pika`: http://github.com/tonyg/pika .. _`Wikipedia article about AMQP`: http://en.wikipedia.org/wiki/AMQP Documentation ------------- Carrot is using Sphinx, and the latest documentation is available at GitHub: http://github.com/ask/carrot/ Installation ============ You can install ``carrot`` either via the Python Package Index (PyPI) or from source. To install using ``pip``,:: $ pip install carrot To install using ``easy_install``,:: $ easy_install carrot If you have downloaded a source tarball you can install it by doing the following,:: $ python setup.py build # python setup.py install # as root Terminology =========== There are some concepts you should be familiar with before starting: * Publishers Publishers sends messages to an exchange. * Exchanges Messages are sent to exchanges. Exchanges are named and can be configured to use one of several routing algorithms. The exchange routes the messages to consumers by matching the routing key in the message with the routing key the consumer provides when binding to the exchange. * Consumers Consumers declares a queue, binds it to a exchange and receives messages from it. * Queues Queues receive messages sent to exchanges. The queues are declared by consumers. * Routing keys Every message has a routing key. The interpretation of the routing key depends on the exchange type. There are four default exchange types defined by the AMQP standard, and vendors can define custom types (so see your vendors manual for details). These are the default exchange types defined by AMQP/0.8: * Direct exchange Matches if the routing key property of the message and the ``routing_key`` attribute of the consumer are identical. * Fan-out exchange Always matches, even if the binding does not have a routing key. * Topic exchange Matches the routing key property of the message by a primitive pattern matching scheme. The message routing key then consists of words separated by dots (``"."``, like domain names), and two special characters are available; star (``"*"``) and hash (``"#"``). The star matches any word, and the hash matches zero or more words. For example ``"*.stock.#"`` matches the routing keys ``"usd.stock"`` and ``"eur.stock.db"`` but not ``"stock.nasdaq"``. Examples ======== Creating a connection --------------------- You can set up a connection by creating an instance of ``carrot.messaging.BrokerConnection``, with the appropriate options for your broker: >>> from carrot.connection import BrokerConnection >>> conn = BrokerConnection(hostname="localhost", port=5672, ... userid="test", password="test", ... virtual_host="test") If you're using Django you can use the ``carrot.connection.DjangoBrokerConnection`` class instead, which loads the connection settings from your ``settings.py``:: BROKER_HOST = "localhost" BROKER_PORT = 5672 BROKER_USER = "test" BROKER_PASSWORD = "secret" BROKER_VHOST = "/test" Then create a connection by doing: >>> from carrot.connection import DjangoBrokerConnection >>> conn = DjangoBrokerConnection() Receiving messages using a Consumer ----------------------------------- First we open up a Python shell and start a message consumer. This consumer declares a queue named ``"feed"``, receiving messages with the routing key ``"importer"`` from the ``"feed"`` exchange. The example then uses the consumers ``wait()`` method to go into consume mode, where it continuously polls the queue for new messages, and when a message is received it passes the message to all registered callbacks. >>> from carrot.messaging import Consumer >>> consumer = Consumer(connection=conn, queue="feed", ... exchange="feed", routing_key="importer") >>> def import_feed_callback(message_data, message): ... feed_url = message_data["import_feed"] ... print("Got feed import message for: %s" % feed_url) ... # something importing this feed url ... # import_feed(feed_url) ... message.ack() >>> consumer.register_callback(import_feed_callback) >>> consumer.wait() # Go into the consumer loop. Sending messages using a Publisher ---------------------------------- Then we open up another Python shell to send some messages to the consumer defined in the last section. >>> from carrot.messaging import Publisher >>> publisher = Publisher(connection=conn, ... exchange="feed", routing_key="importer") >>> publisher.send({"import_feed": "http://cnn.com/rss/edition.rss"}) >>> publisher.close() Look in the first Python shell again (where ``consumer.wait()`` is running), where the following text has been printed to the screen:: Got feed import message for: http://cnn.com/rss/edition.rss Serialization of Data ----------------------- By default every message is encoded using `JSON`_, so sending Python data structures like dictionaries and lists works. `YAML`_, `msgpack`_ and Python's built-in ``pickle`` module is also supported, and if needed you can register any custom serialization scheme you want to use. .. _`JSON`: http://www.json.org/ .. _`YAML`: http://yaml.org/ .. _`msgpack`: http://msgpack.sourceforge.net/ Each option has its advantages and disadvantages. ``json`` -- JSON is supported in many programming languages, is now a standard part of Python (since 2.6), and is fairly fast to decode using the modern Python libraries such as ``cjson or ``simplejson``. The primary disadvantage to ``JSON`` is that it limits you to the following data types: strings, unicode, floats, boolean, dictionaries, and lists. Decimals and dates are notably missing. Also, binary data will be transferred using base64 encoding, which will cause the transferred data to be around 34% larger than an encoding which supports native binary types. However, if your data fits inside the above constraints and you need cross-language support, the default setting of ``JSON`` is probably your best choice. ``pickle`` -- If you have no desire to support any language other than Python, then using the ``pickle`` encoding will gain you the support of all built-in Python data types (except class instances), smaller messages when sending binary files, and a slight speedup over ``JSON`` processing. ``yaml`` -- YAML has many of the same characteristics as ``json``, except that it natively supports more data types (including dates, recursive references, etc.) However, the Python libraries for YAML are a good bit slower than the libraries for JSON. If you need a more expressive set of data types and need to maintain cross-language compatibility, then ``YAML`` may be a better fit than the above. To instruct carrot to use an alternate serialization method, use one of the following options. 1. Set the serialization option on a per-Publisher basis: >>> from carrot.messaging import Publisher >>> publisher = Publisher(connection=conn, ... exchange="feed", routing_key="importer", ... serializer="yaml") 2. Set the serialization option on a per-call basis >>> from carrot.messaging import Publisher >>> publisher = Publisher(connection=conn, ... exchange="feed", routing_key="importer") >>> publisher.send({"import_feed": "http://cnn.com/rss/edition.rss"}, ... serializer="pickle") >>> publisher.close() Note that ``Consumer``s do not need the serialization method specified in their code. They can auto-detect the serialization method since we supply the ``Content-type`` header as part of the AMQP message. Sending raw data without Serialization --------------------------------------- In some cases, you don't need your message data to be serialized. If you pass in a plain string or unicode object as your message, then carrot will not waste cycles serializing/deserializing the data. You can optionally specify a ``content_type`` and ``content_encoding`` for the raw data: >>> from carrot.messaging import Publisher >>> publisher = Publisher(connection=conn, ... exchange="feed", routing_key="import_pictures") >>> publisher.send(open('~/my_picture.jpg','rb').read(), content_type="image/jpeg", content_encoding="binary") >>> publisher.close() The ``message`` object returned by the ``Consumer`` class will have a ``content_type`` and ``content_encoding`` attribute. Receiving messages without a callback -------------------------------------- You can also poll the queue manually, by using the ``fetch`` method. This method returns a ``Message`` object, from where you can get the message body, de-serialize the body to get the data, acknowledge, reject or re-queue the message. >>> consumer = Consumer(connection=conn, queue="feed", ... exchange="feed", routing_key="importer") >>> message = consumer.fetch() >>> if message: ... message_data = message.payload ... message.ack() ... else: ... # No messages waiting on the queue. >>> consumer.close() Sub-classing the messaging classes ---------------------------------- The ``Consumer``, and ``Publisher`` classes can also be sub classed. Thus you can define the above publisher and consumer like so: >>> from carrot.messaging import Publisher, Consumer >>> class FeedPublisher(Publisher): ... exchange = "feed" ... routing_key = "importer" ... ... def import_feed(self, feed_url): ... return self.send({"action": "import_feed", ... "feed_url": feed_url}) >>> class FeedConsumer(Consumer): ... queue = "feed" ... exchange = "feed" ... routing_key = "importer" ... ... def receive(self, message_data, message): ... action = message_data["action"] ... if action == "import_feed": ... # something importing this feed ... # import_feed(message_data["feed_url"]) message.ack() ... else: ... raise Exception("Unknown action: %s" % action) >>> publisher = FeedPublisher(connection=conn) >>> publisher.import_feed("http://cnn.com/rss/edition.rss") >>> publisher.close() >>> consumer = FeedConsumer(connection=conn) >>> consumer.wait() # Go into the consumer loop. Getting Help ============ Mailing list ------------ Join the `carrot-users`_ mailing list. .. _`carrot-users`: http://groups.google.com/group/carrot-users/ Bug tracker =========== If you have any suggestions, bug reports or annoyances please report them to our issue tracker at http://github.com/ask/carrot/issues/ Contributing ============ Development of ``carrot`` happens at Github: http://github.com/ask/carrot You are highly encouraged to participate in the development. If you don't like Github (for some reason) you're welcome to send regular patches. License ======= This software is licensed under the ``New BSD License``. See the ``LICENSE`` file in the top distribution directory for the full license text. carrot-0.10.7/README.rst000644 000765 000024 00000000000 11453345003 017357 1carrot-0.10.7/READMEustar00askstaff000000 000000 carrot-0.10.7/setup.cfg000644 000765 000024 00000000422 11453345106 014633 0ustar00askstaff000000 000000 [bdist_rpm] requires = anyjson amqplib >= 0.6 [build_sphinx] all_files = 1 build-dir = docs/.build source-dir = docs/ [egg_info] tag_build = tag_date = 0 tag_svn_revision = 0 [nosetests] verbosity = 1 detailed-errors = 1 [upload_sphinx] upload-dir = docs/.build/html carrot-0.10.7/setup.py000644 000765 000024 00000004745 11401660116 014532 0ustar00askstaff000000 000000 #!/usr/bin/env python # -*- coding: utf-8 -*- import os import codecs try: from setuptools import setup, find_packages except ImportError: from ez_setup import use_setuptools use_setuptools() from setuptools import setup, find_packages from distutils.command.install_data import install_data from distutils.command.install import INSTALL_SCHEMES import sys import carrot packages, data_files = [], [] root_dir = os.path.dirname(__file__) if root_dir != '': os.chdir(root_dir) src_dir = "carrot" def osx_install_data(install_data): def finalize_options(self): self.set_undefined_options("install", ("install_lib", "install_dir")) install_data.finalize_options(self) def fullsplit(path, result=None): if result is None: result = [] head, tail = os.path.split(path) if head == '': return [tail] + result if head == path: return result return fullsplit(head, [tail] + result) for scheme in INSTALL_SCHEMES.values(): scheme['data'] = scheme['purelib'] for dirpath, dirnames, filenames in os.walk(src_dir): # Ignore dirnames that start with '.' for i, dirname in enumerate(dirnames): if dirname.startswith("."): del dirnames[i] for filename in filenames: if filename.endswith(".py"): packages.append('.'.join(fullsplit(dirpath))) else: data_files.append([dirpath, [os.path.join(dirpath, f) for f in filenames]]) if os.path.exists("README.rst"): long_description = codecs.open('README.rst', "r", "utf-8").read() else: long_description = "See http://pypi.python.org/pypi/carrot" setup( name='carrot', version=carrot.__version__, description=carrot.__doc__, author=carrot.__author__, author_email=carrot.__contact__, url=carrot.__homepage__, platforms=["any"], packages=packages, data_files=data_files, zip_safe=False, test_suite="nose.collector", install_requires=[ 'anyjson', 'amqplib>=0.6', ], classifiers=[ "Development Status :: 4 - Beta", "Framework :: Django", "Operating System :: OS Independent", "Programming Language :: Python", "License :: OSI Approved :: BSD License", "Intended Audience :: Developers", "Topic :: Communications", "Topic :: System :: Distributed Computing", "Topic :: Software Development :: Libraries :: Python Modules", ], long_description=long_description, ) carrot-0.10.7/tests/000755 000765 000024 00000000000 11453345106 014156 5ustar00askstaff000000 000000 carrot-0.10.7/THANKS000644 000765 000024 00000000430 11334033664 013725 0ustar00askstaff000000 000000 Thanks to Barry Pederson for the py-amqplib library. Thanks to Grégoire Cachet for bug reports. Thanks to Martin Mahner for the Sphinx theme. Thanks to jcater for bug reports. Thanks to sebest for bug reports. Thanks to greut for bug reports carrot-0.10.7/TODO000644 000765 000024 00000000120 11334033664 013476 0ustar00askstaff000000 000000 Please see our Issue Tracker at GitHub: http://github.com/ask/carrot/issues carrot-0.10.7/tests/__init__.py000644 000765 000024 00000000000 11334033665 016257 0ustar00askstaff000000 000000 carrot-0.10.7/tests/backend.py000644 000765 000024 00000040145 11444370516 016126 0ustar00askstaff000000 000000 import os import sys import unittest import pickle import time from itertools import count sys.path.insert(0, os.pardir) sys.path.append(os.getcwd()) from carrot.messaging import Consumer, Publisher, ConsumerSet from carrot import serialization from tests.utils import establish_test_connection class AdvancedDataType(object): def __init__(self, something): self.data = something def fetch_next_message(consumer): while True: message = consumer.fetch() if message: return message class BackendMessagingCase(unittest.TestCase): nextq = count(1).next def setUp(self): self.conn = establish_test_connection() self.queue = TEST_QUEUE self.exchange = TEST_EXCHANGE self.routing_key = TEST_ROUTING_KEY def create_consumer(self, **options): queue = "%s%s" % (self.queue, self.nextq()) return Consumer(connection=self.conn, queue=queue, exchange=self.exchange, routing_key=self.routing_key, **options) def create_consumerset(self, queues={}, consumers=[], **options): return ConsumerSet(connection=self.conn, from_dict=queues, consumers=consumers, **options) def create_publisher(self, exchange=None, routing_key=None, **options): exchange = exchange or self.exchange routing_key = routing_key or self.routing_key return Publisher(connection=self.conn, exchange=exchange, routing_key=routing_key, **options) def test_regression_implied_auto_delete(self): consumer = self.create_consumer(exclusive=True, auto_declare=False) self.assertTrue(consumer.auto_delete, "exclusive implies auto_delete") consumer.close() consumer = self.create_consumer(durable=True, auto_delete=False, auto_declare=False) self.assertFalse(consumer.auto_delete, """durable does *not* imply auto_delete. regression: http://github.com/ask/carrot/issues/closed#issue/2""") consumer.close() def test_consumer_options(self): opposite_defaults = { "queue": "xyxyxyxy", "exchange": "xyxyxyxy", "routing_key": "xyxyxyxy", "durable": False, "exclusive": True, "auto_delete": True, "exchange_type": "topic", } consumer = Consumer(connection=self.conn, **opposite_defaults) for opt_name, opt_value in opposite_defaults.items(): self.assertEquals(getattr(consumer, opt_name), opt_value) consumer.close() def test_consumer_backend(self): consumer = self.create_consumer() self.assertTrue(consumer.backend.connection is self.conn) consumer.close() def test_consumer_queue_declared(self): consumer = self.create_consumer() self.assertTrue(consumer.backend.queue_exists(consumer.queue)) consumer.close() def test_consumer_callbacks(self): consumer = self.create_consumer() publisher = self.create_publisher() # raises on no callbacks self.assertRaises(NotImplementedError, consumer.receive, {}, {}) callback1_scratchpad = {} def callback1(message_data, message): callback1_scratchpad["message_data"] = message_data callback2_scratchpad = {} def callback2(message_data, message): callback2_scratchpad.update({"delivery_tag": message.delivery_tag, "message_body": message.body}) self.assertFalse(consumer.callbacks, "no default callbacks") consumer.register_callback(callback1) consumer.register_callback(callback2) self.assertEquals(len(consumer.callbacks), 2, "callbacks registered") self.assertTrue(consumer.callbacks[0] is callback1, "callbacks are ordered") self.assertTrue(consumer.callbacks[1] is callback2, "callbacks are ordered") body = {"foo": "bar"} message = self.create_raw_message(publisher, body, "Elaine was here") consumer._receive_callback(message) self.assertEquals(callback1_scratchpad.get("message_data"), body, "callback1 was called") self.assertEquals(callback2_scratchpad.get("delivery_tag"), "Elaine was here") consumer.close() publisher.close() def create_raw_message(self, publisher, body, delivery_tag): raw_message = publisher.create_message(body) raw_message.delivery_tag = delivery_tag return raw_message def test_empty_queue_returns_None(self): consumer = self.create_consumer() consumer.discard_all() self.assertFalse(consumer.fetch()) consumer.close() def test_custom_serialization_scheme(self): serialization.registry.register('custom_test', pickle.dumps, pickle.loads, content_type='application/x-custom-test', content_encoding='binary') consumer = self.create_consumer() publisher = self.create_publisher() consumer.discard_all() data = {"string": "The quick brown fox jumps over the lazy dog", "int": 10, "float": 3.14159265, "unicode": u"The quick brown fox jumps over the lazy dog", "advanced": AdvancedDataType("something"), "set": set(["george", "jerry", "elaine", "cosmo"]), "exception": Exception("There was an error"), } publisher.send(data, serializer='custom_test') message = fetch_next_message(consumer) backend = self.conn.create_backend() self.assertTrue(isinstance(message, backend.Message)) self.assertEquals(message.payload.get("int"), 10) self.assertEquals(message.content_type, 'application/x-custom-test') self.assertEquals(message.content_encoding, 'binary') decoded_data = message.decode() self.assertEquals(decoded_data.get("string"), "The quick brown fox jumps over the lazy dog") self.assertEquals(decoded_data.get("int"), 10) self.assertEquals(decoded_data.get("float"), 3.14159265) self.assertEquals(decoded_data.get("unicode"), u"The quick brown fox jumps over the lazy dog") self.assertEquals(decoded_data.get("set"), set(["george", "jerry", "elaine", "cosmo"])) self.assertTrue(isinstance(decoded_data.get("exception"), Exception)) self.assertEquals(decoded_data.get("exception").args[0], "There was an error") self.assertTrue(isinstance(decoded_data.get("advanced"), AdvancedDataType)) self.assertEquals(decoded_data["advanced"].data, "something") consumer.close() publisher.close() def test_consumer_fetch(self): consumer = self.create_consumer() publisher = self.create_publisher() consumer.discard_all() data = {"string": "The quick brown fox jumps over the lazy dog", "int": 10, "float": 3.14159265, "unicode": u"The quick brown fox jumps over the lazy dog", } publisher.send(data) message = fetch_next_message(consumer) backend = self.conn.create_backend() self.assertTrue(isinstance(message, backend.Message)) self.assertEquals(message.decode(), data) consumer.close() publisher.close() def test_consumer_process_next(self): consumer = self.create_consumer() publisher = self.create_publisher() consumer.discard_all() scratchpad = {} def callback(message_data, message): scratchpad["delivery_tag"] = message.delivery_tag consumer.register_callback(callback) publisher.send({"name_discovered": { "first_name": "Cosmo", "last_name": "Kramer"}}) while True: message = consumer.fetch(enable_callbacks=True) if message: break self.assertEquals(scratchpad.get("delivery_tag"), message.delivery_tag) consumer.close() publisher.close() def test_consumer_discard_all(self): consumer = self.create_consumer() publisher = self.create_publisher() consumer.discard_all() for i in xrange(100): publisher.send({"foo": "bar"}) time.sleep(0.5) self.assertEquals(consumer.discard_all(), 100) consumer.close() publisher.close() def test_iterqueue(self): consumer = self.create_consumer() publisher = self.create_publisher() num = consumer.discard_all() it = consumer.iterqueue(limit=100) consumer.register_callback(lambda *args: args) for i in xrange(100): publisher.send({"foo%d" % i: "bar%d" % i}) time.sleep(0.5) for i in xrange(100): try: message = it.next() data = message.decode() self.assertTrue("foo%d" % i in data, "foo%d not in data" % i) self.assertEquals(data.get("foo%d" % i), "bar%d" % i) except StopIteration: self.assertTrue(False, "iterqueue fails StopIteration") self.assertRaises(StopIteration, it.next) # no messages on queue raises StopIteration if infinite=False it = consumer.iterqueue() self.assertRaises(StopIteration, it.next) it = consumer.iterqueue(infinite=True) self.assertTrue(it.next() is None, "returns None if no messages and inifite=True") consumer.close() publisher.close() def test_publisher_message_priority(self): consumer = self.create_consumer() publisher = self.create_publisher() consumer.discard_all() m = publisher.create_message("foo", priority=9) publisher.send({"foo": "bar"}, routing_key="nowhere", priority=9, mandatory=False, immediate=False) consumer.discard_all() consumer.close() publisher.close() def test_backend_survives_channel_close_regr17(self): """ test that a backend instance is still functional after a method that results in a channel closure. """ backend = self.create_publisher().backend assert not backend.queue_exists('notaqueue') # after calling this once, the channel seems to close, but the # backend may be holding a reference to it... assert not backend.queue_exists('notaqueue') def disabled_publisher_mandatory_flag_regr16(self): """ Test that the publisher "mandatory" flag raises exceptions at appropriate times. """ routing_key = 'black_hole' assert self.conn.connection is not None message = {'foo': 'mandatory'} # sanity check cleanup from last test assert not self.create_consumer().backend.queue_exists(routing_key) publisher = self.create_publisher() # this should just get discarded silently, it's not mandatory publisher.send(message, routing_key=routing_key, mandatory=False) # This raises an unspecified exception because there is no queue to # deliver to self.assertRaises(Exception, publisher.send, message, routing_key=routing_key, mandatory=True) # now bind a queue to it consumer = Consumer(connection=self.conn, queue=routing_key, exchange=self.exchange, routing_key=routing_key, durable=False, exclusive=True) # check that it exists assert self.create_consumer().backend.queue_exists(routing_key) # this should now get routed to our consumer with no exception publisher.send(message, routing_key=routing_key, mandatory=True) def test_consumer_auto_ack(self): consumer = self.create_consumer(auto_ack=True) publisher = self.create_publisher() consumer.discard_all() publisher.send({"foo": "Baz"}) message = fetch_next_message(consumer) self.assertEquals(message._state, "ACK") consumer.close() publisher.close() publisher = self.create_publisher() consumer = self.create_consumer(auto_ack=False) publisher.send({"foo": "Baz"}) message = fetch_next_message(consumer) self.assertEquals(message._state, "RECEIVED") consumer.close() publisher.close() def test_consumer_consume(self): consumer = self.create_consumer(auto_ack=True) publisher = self.create_publisher() consumer.discard_all() data = {"foo": "Baz"} publisher.send(data) try: data2 = {"company": "Vandelay Industries"} publisher.send(data2) scratchpad = {} def callback(message_data, message): scratchpad["data"] = message_data consumer.register_callback(callback) it = consumer.iterconsume() it.next() self.assertEquals(scratchpad.get("data"), data) it.next() self.assertEquals(scratchpad.get("data"), data2) # Cancel consumer/close and restart. consumer.close() consumer = self.create_consumer(auto_ack=True) consumer.register_callback(callback) consumer.discard_all() scratchpad = {} # Test limits it = consumer.iterconsume(limit=4) publisher.send(data) publisher.send(data2) publisher.send(data) publisher.send(data2) publisher.send(data) it.next() self.assertEquals(scratchpad.get("data"), data) it.next() self.assertEquals(scratchpad.get("data"), data2) it.next() self.assertEquals(scratchpad.get("data"), data) it.next() self.assertEquals(scratchpad.get("data"), data2) self.assertRaises(StopIteration, it.next) finally: consumer.close() publisher.close() def test_consumerset_iterconsume(self): consumerset = self.create_consumerset(queues={ "bar": { "exchange": "foo", "exchange_type": "direct", "routing_key": "foo.bar", }, "baz": { "exchange": "foo", "exchange_type": "direct", "routing_key": "foo.baz", }, "bam": { "exchange": "foo", "exchange_type": "direct", "routing_key": "foo.bam", }, "xuzzy": { "exchange": "foo", "exchange_type": "direct", "routing_key": "foo.xuzzy", }}) publisher = self.create_publisher(exchange="foo") consumerset.discard_all() scratchpad = {} def callback(message_data, message): scratchpad["data"] = message_data def assertDataIs(what): self.assertEquals(scratchpad.get("data"), what) try: consumerset.register_callback(callback) it = consumerset.iterconsume() publisher.send({"rkey": "foo.xuzzy"}, routing_key="foo.xuzzy") it.next() assertDataIs({"rkey": "foo.xuzzy"}) publisher.send({"rkey": "foo.xuzzy"}, routing_key="foo.xuzzy") publisher.send({"rkey": "foo.bar"}, routing_key="foo.bar") publisher.send({"rkey": "foo.baz"}, routing_key="foo.baz") publisher.send({"rkey": "foo.bam"}, routing_key="foo.bam") it.next() assertDataIs({"rkey": "foo.xuzzy"}) it.next() assertDataIs({"rkey": "foo.bar"}) it.next() assertDataIs({"rkey": "foo.baz"}) it.next() assertDataIs({"rkey": "foo.bam"}) finally: consumerset.close() publisher.close() carrot-0.10.7/tests/test_django.py000644 000765 000024 00000004075 11334033665 017041 0ustar00askstaff000000 000000 import os import sys import unittest import pickle import time sys.path.insert(0, os.pardir) sys.path.append(os.getcwd()) from tests.utils import BROKER_HOST, BROKER_PORT, BROKER_VHOST, \ BROKER_USER, BROKER_PASSWORD from carrot.connection import DjangoBrokerConnection, BrokerConnection from UserDict import UserDict CARROT_BACKEND = "amqp" class DictWrapper(UserDict): def __init__(self, data): self.data = data def __getattr__(self, key): try: return self.data[key] except KeyError: raise AttributeError("'%s' object has no attribute '%s'" % ( self.__class__.__name__, key)) def configured_or_configure(settings, **conf): if settings.configured: for conf_name, conf_value in conf.items(): setattr(settings, conf_name, conf_value) else: settings.configure(default_settings=DictWrapper(conf)) class TestDjangoSpecific(unittest.TestCase): def test_DjangoBrokerConnection(self): try: from django.conf import settings except ImportError: sys.stderr.write( "Django is not installed. \ Not testing django specific features.\n") return configured_or_configure(settings, CARROT_BACKEND=CARROT_BACKEND, BROKER_HOST=BROKER_HOST, BROKER_PORT=BROKER_PORT, BROKER_VHOST=BROKER_VHOST, BROKER_USER=BROKER_USER, BROKER_PASSWORD=BROKER_PASSWORD) expected_values = { "backend_cls": CARROT_BACKEND, "hostname": BROKER_HOST, "port": BROKER_PORT, "virtual_host": BROKER_VHOST, "userid": BROKER_USER, "password": BROKER_PASSWORD} conn = DjangoBrokerConnection() self.assertTrue(isinstance(conn, BrokerConnection)) for val_name, val_value in expected_values.items(): self.assertEquals(getattr(conn, val_name, None), val_value) if __name__ == '__main__': unittest.main() carrot-0.10.7/tests/test_examples.py000644 000765 000024 00000011506 11334033665 017412 0ustar00askstaff000000 000000 import os import sys import unittest sys.path.insert(0, os.pardir) sys.path.append(os.getcwd()) from tests.utils import establish_test_connection from carrot.connection import BrokerConnection from carrot.backends.pyamqplib import Message README_QUEUE = "feed" README_EXCHANGE = "feed" README_ROUTING_KEY = "feed" class TimeoutError(Exception): """The operation timed out.""" def receive_a_message(consumer): while True: message = consumer.fetch() if message: return message def emulate_wait(consumer): message = receive_a_message(consumer) consumer._receive_callback(message) class CallbacksTestable(object): last_feed = None last_status = None last_body = None last_delivery_tag = None def import_feed(self, message_data, message): feed_url = message_data.get("import_feed") self.last_feed = feed_url if not feed_url: self.last_status = "REJECT" message.reject() else: self.last_status = "ACK" message.ack() def dump_message(self, message_data, message): self.last_body = message.body self.last_delivery_tag = message.delivery_tag def create_README_consumer(amqpconn): from carrot.messaging import Consumer consumer = Consumer(connection=amqpconn, queue=README_QUEUE, exchange=README_EXCHANGE, routing_key=README_ROUTING_KEY) tcallbacks = CallbacksTestable() consumer.register_callback(tcallbacks.import_feed) consumer.register_callback(tcallbacks.dump_message) return consumer, tcallbacks def create_README_publisher(amqpconn): from carrot.messaging import Publisher publisher = Publisher(connection=amqpconn, exchange=README_EXCHANGE, routing_key=README_ROUTING_KEY) return publisher class TestExamples(unittest.TestCase): def setUp(self): self.conn = establish_test_connection() self.consumer, self.tcallbacks = create_README_consumer(self.conn) self.consumer.discard_all() def test_connection(self): self.assertTrue(self.conn) self.assertTrue(self.conn.connection.channel()) def test_README_consumer(self): consumer = self.consumer tcallbacks = self.tcallbacks self.assertTrue(consumer.connection) self.assertTrue(isinstance(consumer.connection, BrokerConnection)) self.assertEquals(consumer.queue, README_QUEUE) self.assertEquals(consumer.exchange, README_EXCHANGE) self.assertEquals(consumer.routing_key, README_ROUTING_KEY) self.assertTrue(len(consumer.callbacks), 2) def test_README_publisher(self): publisher = create_README_publisher(self.conn) self.assertTrue(publisher.connection) self.assertTrue(isinstance(publisher.connection, BrokerConnection)) self.assertEquals(publisher.exchange, README_EXCHANGE) self.assertEquals(publisher.routing_key, README_ROUTING_KEY) def test_README_together(self): consumer = self.consumer tcallbacks = self.tcallbacks publisher = create_README_publisher(self.conn) feed_url = "http://cnn.com/rss/edition.rss" body = {"import_feed": feed_url} publisher.send(body) publisher.close() emulate_wait(consumer) self.assertEquals(tcallbacks.last_feed, feed_url) self.assertTrue(tcallbacks.last_delivery_tag) self.assertEquals(tcallbacks.last_status, "ACK") publisher = create_README_publisher(self.conn) body = {"foo": "FOO"} publisher.send(body) publisher.close() emulate_wait(consumer) self.assertFalse(tcallbacks.last_feed) self.assertTrue(tcallbacks.last_delivery_tag) self.assertEquals(tcallbacks.last_status, "REJECT") def test_subclassing(self): from carrot.messaging import Consumer, Publisher feed_url = "http://cnn.com/rss/edition.rss" testself = self class TConsumer(Consumer): queue = README_QUEUE exchange = README_EXCHANGE routing_key = README_ROUTING_KEY def receive(self, message_data, message): testself.assertTrue(isinstance(message, Message)) testself.assertTrue("import_feed" in message_data) testself.assertEquals(message_data.get("import_feed"), feed_url) class TPublisher(Publisher): exchange = README_EXCHANGE routing_key = README_ROUTING_KEY consumer = TConsumer(connection=self.conn) publisher = TPublisher(connection=self.conn) consumer.discard_all() publisher.send({"import_feed": feed_url}) publisher.close() emulate_wait(consumer) consumer.close() if __name__ == '__main__': unittest.main() carrot-0.10.7/tests/test_pyamqplib.py000644 000765 000024 00000002625 11334033665 017574 0ustar00askstaff000000 000000 import os import sys import unittest import pickle import time sys.path.insert(0, os.pardir) sys.path.append(os.getcwd()) from tests.utils import establish_test_connection from carrot.connection import BrokerConnection from carrot.messaging import Consumer, Publisher, ConsumerSet from carrot.backends.pyamqplib import Backend as AMQPLibBackend from carrot.backends.pyamqplib import Message as AMQPLibMessage from carrot import serialization from tests.backend import BackendMessagingCase TEST_QUEUE = "carrot.unittest" TEST_EXCHANGE = "carrot.unittest" TEST_ROUTING_KEY = "carrot.unittest" TEST_QUEUE_TWO = "carrot.unittest.two" TEST_EXCHANGE_TWO = "carrot.unittest.two" TEST_ROUTING_KEY_TWO = "carrot.unittest.two" TEST_CELERY_QUEUE = { TEST_QUEUE: { "exchange": TEST_EXCHANGE, "exchange_type": "direct", "routing_key": TEST_ROUTING_KEY, }, TEST_QUEUE_TWO: { "exchange": TEST_EXCHANGE_TWO, "exchange_type": "direct", "routing_key": TEST_ROUTING_KEY_TWO, }, } class TestAMQPlibMessaging(BackendMessagingCase): def setUp(self): self.conn = establish_test_connection() self.queue = TEST_QUEUE self.exchange = TEST_EXCHANGE self.routing_key = TEST_ROUTING_KEY BackendMessagingCase = None if __name__ == '__main__': unittest.main() carrot-0.10.7/tests/test_pyqueue.py000644 000765 000024 00000006016 11401660116 017261 0ustar00askstaff000000 000000 import os import sys import unittest import uuid sys.path.insert(0, os.pardir) sys.path.append(os.getcwd()) from carrot.backends.queue import Message as PyQueueMessage from carrot.backends.queue import Backend as PyQueueBackend from carrot.connection import BrokerConnection from carrot.messaging import Messaging, Consumer, Publisher def create_backend(): return PyQueueBackend(connection=BrokerConnection()) class TestPyQueueMessage(unittest.TestCase): def test_message(self): b = create_backend() self.assertTrue(b) message_body = "George Constanza" delivery_tag = str(uuid.uuid4()) m1 = PyQueueMessage(backend=b, body=message_body, delivery_tag=delivery_tag) m2 = PyQueueMessage(backend=b, body=message_body, delivery_tag=delivery_tag) m3 = PyQueueMessage(backend=b, body=message_body, delivery_tag=delivery_tag) self.assertEquals(m1.body, message_body) self.assertEquals(m1.delivery_tag, delivery_tag) m1.ack() m2.reject() m3.requeue() class TestPyQueueBackend(unittest.TestCase): def test_backend(self): b = create_backend() message_body = "Vandelay Industries" b.publish(b.prepare_message(message_body, "direct", content_type='text/plain', content_encoding="ascii"), exchange="test", routing_key="test") m_in_q = b.get() self.assertTrue(isinstance(m_in_q, PyQueueMessage)) self.assertEquals(m_in_q.body, message_body) def test_consumer_interface(self): to_send = ['No', 'soup', 'for', 'you!'] messages = [] def cb(message_data, message): messages.append(message_data) conn = BrokerConnection(backend_cls='memory') consumer = Consumer(connection=conn, queue="test", exchange="test", routing_key="test") consumer.register_callback(cb) publisher = Publisher(connection=conn, exchange="test", routing_key="test") for i in to_send: publisher.send(i) it = consumer.iterconsume() for i in range(len(to_send)): it.next() self.assertEqual(messages, to_send) class TMessaging(Messaging): exchange = "test" routing_key = "test" queue = "test" class TestMessaging(unittest.TestCase): def test_messaging(self): m = TMessaging(connection=BrokerConnection(backend_cls=PyQueueBackend)) self.assertTrue(m) self.assertEquals(m.fetch(), None) mdata = {"name": "Cosmo Kramer"} m.send(mdata) next_msg = m.fetch() next_msg_data = next_msg.decode() self.assertEquals(next_msg_data, mdata) self.assertEquals(m.fetch(), None) if __name__ == '__main__': unittest.main() carrot-0.10.7/tests/test_serialization.py000644 000765 000024 00000016463 11401660116 020450 0ustar00askstaff000000 000000 #!/usr/bin/python # -*- coding: utf-8 -*- import cPickle import sys import os import unittest import uuid sys.path.insert(0, os.pardir) sys.path.append(os.getcwd()) from carrot.serialization import registry # For content_encoding tests unicode_string = u'abcdé\u8463' unicode_string_as_utf8 = unicode_string.encode('utf-8') latin_string = u'abcdé' latin_string_as_latin1 = latin_string.encode('latin-1') latin_string_as_utf8 = latin_string.encode('utf-8') # For serialization tests py_data = {"string": "The quick brown fox jumps over the lazy dog", "int": 10, "float": 3.14159265, "unicode": u"Thé quick brown fox jumps over thé lazy dog", "list": ["george", "jerry", "elaine", "cosmo"], } # JSON serialization tests json_data = ('{"int": 10, "float": 3.1415926500000002, ' '"list": ["george", "jerry", "elaine", "cosmo"], ' '"string": "The quick brown fox jumps over the lazy ' 'dog", "unicode": "Th\\u00e9 quick brown fox jumps over ' 'th\\u00e9 lazy dog"}') # Pickle serialization tests pickle_data = cPickle.dumps(py_data) # YAML serialization tests yaml_data = ('float: 3.1415926500000002\nint: 10\n' 'list: [george, jerry, elaine, cosmo]\n' 'string: The quick brown fox jumps over the lazy dog\n' 'unicode: "Th\\xE9 quick brown fox ' 'jumps over th\\xE9 lazy dog"\n') msgpack_py_data = dict(py_data) # msgpack only supports tuples msgpack_py_data["list"] = tuple(msgpack_py_data["list"]) # Unicode chars are lost in transmit :( msgpack_py_data["unicode"] = 'Th quick brown fox jumps over th lazy dog' msgpack_data = ('\x85\xa3int\n\xa5float\xcb@\t!\xfbS\xc8\xd4\xf1\xa4list' '\x94\xa6george\xa5jerry\xa6elaine\xa5cosmo\xa6string\xda' '\x00+The quick brown fox jumps over the lazy dog\xa7unicode' '\xda\x00)Th quick brown fox jumps over th lazy dog') def say(m): sys.stderr.write("%s\n" % (m, )) class TestSerialization(unittest.TestCase): def test_content_type_decoding(self): content_type = 'plain/text' self.assertEquals(unicode_string, registry.decode( unicode_string_as_utf8, content_type='plain/text', content_encoding='utf-8')) self.assertEquals(latin_string, registry.decode( latin_string_as_latin1, content_type='application/data', content_encoding='latin-1')) def test_content_type_binary(self): content_type = 'plain/text' self.assertNotEquals(unicode_string, registry.decode( unicode_string_as_utf8, content_type='application/data', content_encoding='binary')) self.assertEquals(unicode_string_as_utf8, registry.decode( unicode_string_as_utf8, content_type='application/data', content_encoding='binary')) def test_content_type_encoding(self): # Using the "raw" serializer self.assertEquals(unicode_string_as_utf8, registry.encode( unicode_string, serializer="raw")[-1]) self.assertEquals(latin_string_as_utf8, registry.encode( latin_string, serializer="raw")[-1]) # And again w/o a specific serializer to check the # code where we force unicode objects into a string. self.assertEquals(unicode_string_as_utf8, registry.encode(unicode_string)[-1]) self.assertEquals(latin_string_as_utf8, registry.encode(latin_string)[-1]) def test_json_decode(self): self.assertEquals(py_data, registry.decode( json_data, content_type='application/json', content_encoding='utf-8')) def test_json_encode(self): self.assertEquals(registry.decode( registry.encode(py_data, serializer="json")[-1], content_type='application/json', content_encoding='utf-8'), registry.decode( json_data, content_type='application/json', content_encoding='utf-8')) def test_msgpack_decode(self): try: import msgpack except ImportError: return say("* msgpack-python not installed, will not execute " "related tests.") self.assertEquals(msgpack_py_data, registry.decode( msgpack_data, content_type='application/x-msgpack', content_encoding='binary')) def test_msgpack_encode(self): try: import msgpack except ImportError: return say("* msgpack-python not installed, will not execute " "related tests.") self.assertEquals(registry.decode( registry.encode(msgpack_py_data, serializer="msgpack")[-1], content_type='application/x-msgpack', content_encoding='binary'), registry.decode( msgpack_data, content_type='application/x-msgpack', content_encoding='binary')) def test_yaml_decode(self): try: import yaml except ImportError: return say("* PyYAML not installed, will not execute " "related tests.") self.assertEquals(py_data, registry.decode( yaml_data, content_type='application/x-yaml', content_encoding='utf-8')) def test_yaml_encode(self): try: import yaml except ImportError: return say("* PyYAML not installed, will not execute " "related tests.") self.assertEquals(registry.decode( registry.encode(py_data, serializer="yaml")[-1], content_type='application/x-yaml', content_encoding='utf-8'), registry.decode( yaml_data, content_type='application/x-yaml', content_encoding='utf-8')) def test_pickle_decode(self): self.assertEquals(py_data, registry.decode( pickle_data, content_type='application/x-python-serialize', content_encoding='binary')) def test_pickle_encode(self): self.assertEquals(pickle_data, registry.encode(py_data, serializer="pickle")[-1]) if __name__ == '__main__': unittest.main() carrot-0.10.7/tests/test_utils.py000644 000765 000024 00000000542 11401660116 016722 0ustar00askstaff000000 000000 import unittest from carrot import utils class TestUtils(unittest.TestCase): def test_partition_unicode(self): s = u'hi mom' self.assertEqual(utils.partition(s, ' '), (u'hi', u' ', u'mom')) def test_rpartition_unicode(self): s = u'hi mom !' self.assertEqual(utils.rpartition(s, ' '), (u'hi mom', u' ', u'!')) carrot-0.10.7/tests/test_with_statement.py000644 000765 000024 00000002156 11334033665 020634 0ustar00askstaff000000 000000 from __future__ import with_statement import os import sys import unittest sys.path.insert(0, os.pardir) sys.path.append(os.getcwd()) from tests.utils import test_connection_args from carrot.connection import BrokerConnection from carrot.messaging import Consumer, Publisher class TestTransactioned(unittest.TestCase): def test_with_statement(self): with BrokerConnection(**test_connection_args()) as conn: self.assertFalse(conn._closed) with Publisher(connection=conn, exchange="F", routing_key="G") \ as publisher: self.assertFalse(publisher._closed) self.assertTrue(conn._closed) self.assertTrue(publisher._closed) with BrokerConnection(**test_connection_args()) as conn: self.assertFalse(conn._closed) with Consumer(connection=conn, queue="E", exchange="F", routing_key="G") as consumer: self.assertFalse(consumer._closed) self.assertTrue(conn._closed) self.assertTrue(consumer._closed) if __name__ == '__main__': unittest.main() carrot-0.10.7/tests/utils.py000644 000765 000024 00000001555 11334033665 015700 0ustar00askstaff000000 000000 import os from carrot.connection import BrokerConnection BROKER_HOST = os.environ.get('BROKER_HOST', "localhost") BROKER_PORT = os.environ.get('BROKER_PORT', 5672) BROKER_VHOST = os.environ.get('BROKER_VHOST', "/") BROKER_USER = os.environ.get('BROKER_USER', "guest") BROKER_PASSWORD = os.environ.get('BROKER_PASSWORD', "guest") STOMP_HOST = os.environ.get('STOMP_HOST', 'localhost') STOMP_PORT = os.environ.get('STOMP_PORT', 61613) STOMP_QUEUE = os.environ.get('STOMP_QUEUE', '/queue/testcarrot') def test_connection_args(): return {"hostname": BROKER_HOST, "port": BROKER_PORT, "virtual_host": BROKER_VHOST, "userid": BROKER_USER, "password": BROKER_PASSWORD} def test_stomp_connection_args(): return {"hostname": STOMP_HOST, "port": STOMP_PORT} def establish_test_connection(): return BrokerConnection(**test_connection_args()) carrot-0.10.7/tests/xxxstmop.py000644 000765 000024 00000010354 11334033665 016447 0ustar00askstaff000000 000000 import os import sys import unittest import uuid sys.path.insert(0, os.pardir) sys.path.append(os.getcwd()) try: import stompy except ImportError: stompy = None Frame = StompMessage = StompBackend = object else: from carrot.backends.pystomp import Message as StompMessage from carrot.backends.pystomp import Backend as StompBackend from stompy.frame import Frame from carrot.connection import BrokerConnection from carrot.messaging import Publisher, Consumer from tests.utils import test_stomp_connection_args, STOMP_QUEUE from carrot.serialization import encode _no_stompy_msg = "* stompy (python-stomp) not installed. " \ + "Will not execute related tests." _no_stompy_msg_emitted = False def stompy_or_None(): def emit_no_stompy_msg(): global _no_stompy_msg_emitted if not _no_stompy_msg_emitted: sys.stderr.write("\n" + _no_stompy_msg + "\n") _no_stompy_msg_emitted = True if stompy is None: emit_no_stompy_msg() return None return stompy def create_connection(): return BrokerConnection(backend_cls=StompBackend, **test_stomp_connection_args()) def create_backend(): return create_connection().create_backend() class MockFrame(Frame): def mock(self, command=None, headers=None, body=None): self.command = command self.headers = headers self.body = body return self class TestStompMessage(unittest.TestCase): def test_message(self): if not stompy_or_None(): return b = create_backend() self.assertTrue(b) message_body = "George Constanza" delivery_tag = str(uuid.uuid4()) frame = MockFrame().mock(body=message_body, headers={ "message-id": delivery_tag, "content_type": "text/plain", "content_encoding": "utf-8", }) m1 = StompMessage(backend=b, frame=frame) m2 = StompMessage(backend=b, frame=frame) m3 = StompMessage(backend=b, frame=frame) self.assertEquals(m1.body, message_body) self.assertEquals(m1.delivery_tag, delivery_tag) #m1.ack() self.assertRaises(NotImplementedError, m2.reject) self.assertRaises(NotImplementedError, m3.requeue) class TestPyStompMessaging(unittest.TestCase): def setUp(self): if stompy_or_None(): self.conn = create_connection() self.queue = STOMP_QUEUE self.exchange = STOMP_QUEUE self.routing_key = STOMP_QUEUE def create_consumer(self, **options): return Consumer(connection=self.conn, queue=self.queue, exchange=self.exchange, routing_key=self.routing_key, **options) def create_publisher(self, **options): return Publisher(connection=self.conn, exchange=self.exchange, routing_key=self.routing_key, **options) def test_backend(self): if not stompy_or_None(): return publisher = self.create_publisher() consumer = self.create_consumer() for i in range(100): publisher.send({"foo%d" % i: "bar%d" % i}) publisher.close() discarded = consumer.discard_all() self.assertEquals(discarded, 100) publisher.close() consumer.close() publisher = self.create_publisher() for i in range(100): publisher.send({"foo%d" % i: "bar%d" % i}) consumer = self.create_consumer() for i in range(100): while True: message = consumer.fetch() if message: break self.assertTrue("foo%d" % i in message.payload) message.ack() publisher.close() consumer.close() consumer = self.create_consumer() discarded = consumer.discard_all() self.assertEquals(discarded, 0) def create_raw_message(self, publisher, body, delivery_tag): content_type, content_encoding, payload = encode(body) frame = MockFrame().mock(body=payload, headers={ "message-id": delivery_tag, "content-type": content_type, "content-encoding": content_encoding, }) return frame carrot-0.10.7/docs/.static/000755 000765 000024 00000000000 11453345106 015311 5ustar00askstaff000000 000000 carrot-0.10.7/docs/_ext/000755 000765 000024 00000000000 11453345106 014703 5ustar00askstaff000000 000000 carrot-0.10.7/docs/_theme/000755 000765 000024 00000000000 11453345106 015205 5ustar00askstaff000000 000000 carrot-0.10.7/docs/changelog.rst000644 000765 000024 00000000000 11453344754 022247 1carrot-0.10.7/Changelogustar00askstaff000000 000000 carrot-0.10.7/docs/conf.py000644 000765 000024 00000013706 11334033664 015253 0ustar00askstaff000000 000000 # -*- coding: utf-8 -*- # # Carrot documentation build configuration file, created by # sphinx-quickstart on Mon May 18 21:37:44 2009. # # This file is execfile()d with the current directory set to its #containing dir. # # The contents of this file are pickled, so don't put values in the namespace # that aren't pickleable (module imports are okay, they're removed #automatically). # # All configuration values have a default; values that are commented out # serve to show the default. import sys import os # If your extensions are in another directory, add it here. If the directory # is relative to the documentation root, use os.path.abspath to make it # absolute, like shown here. sys.path.insert(0, "../") import carrot # General configuration # --------------------- # Add any Sphinx extension module names here, as strings. # They can be extensions # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. extensions = ['sphinx.ext.autodoc'] # Add any paths that contain templates here, relative to this directory. templates_path = ['.templates'] # The suffix of source filenames. source_suffix = '.rst' # The encoding of source files. #source_encoding = 'utf-8' # The master toctree document. master_doc = 'index' # General information about the project. project = u'Carrot' copyright = u'2009, Ask Solem' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the # built documents. # # The short X.Y version. version = ".".join(map(str, carrot.VERSION[0:2])) # The full version, including alpha/beta/rc tags. release = carrot.__version__ # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. #language = None # There are two options for replacing |today|: either, you set today to some # non-false value, then it is used: #today = '' # Else, today_fmt is used as the format for a strftime call. #today_fmt = '%B %d, %Y' # List of documents that shouldn't be included in the build. #unused_docs = [] # List of directories, relative to source directory, that shouldn't be searched # for source files. exclude_trees = ['.build'] # The reST default role (used for this markup: `text`) to use for all # documents. #default_role = None # If true, '()' will be appended to :func: etc. cross-reference text. add_function_parentheses = True # If true, the current module name will be prepended to all description # unit titles (such as .. function::). #add_module_names = True # If true, sectionauthor and moduleauthor directives will be shown in the # output. They are ignored by default. #show_authors = False # The name of the Pygments (syntax highlighting) style to use. pygments_style = 'trac' #html_translator_class = "djangodocs.DjangoHTMLTranslator" # Options for HTML output # ----------------------- # The style sheet to use for HTML and HTML Help pages. A file of that name # must exist either in Sphinx' static/ path, or in one of the custom paths # given in html_static_path. #html_style = 'agogo.css' # The name for this set of Sphinx documents. If None, it defaults to # " v documentation". #html_title = None # A shorter title for the navigation bar. Default is the same as html_title. #html_short_title = None # The name of an image file (relative to this directory) to place at the top # of the sidebar. #html_logo = None # The name of an image file (within the static path) to use as favicon of the # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 # pixels large. #html_favicon = None # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ['.static'] # If not '', a 'Last updated on:' timestamp is inserted at every page bottom, # using the given strftime format. #html_last_updated_fmt = '%b %d, %Y' # If true, SmartyPants will be used to convert quotes and dashes to # typographically correct entities. html_use_smartypants = True # Custom sidebar templates, maps document names to template names. #html_sidebars = {} # Additional templates that should be rendered to pages, maps page names to # template names. #html_additional_pages = {} # If false, no module index is generated. html_use_modindex = True # If false, no index is generated. html_use_index = True # If true, the index is split into individual pages for each letter. #html_split_index = False # If true, the reST sources are included in the HTML build as _sources/. #html_copy_source = True # If true, an OpenSearch description file will be output, and all pages will # contain a tag referring to it. The value of this option must be the # base URL from which the finished HTML is served. #html_use_opensearch = '' # If nonempty, this is the file name suffix for HTML files (e.g. ".xhtml"). #html_file_suffix = '' # Output file base name for HTML help builder. htmlhelp_basename = 'Carrotdoc' # Options for LaTeX output # ------------------------ # The paper size ('letter' or 'a4'). #latex_paper_size = 'letter' # The font size ('10pt', '11pt' or '12pt'). #latex_font_size = '10pt' # Grouping the document tree into LaTeX files. List of tuples # (source start file, target name, title, author, document class # [howto/manual]). latex_documents = [ ('index', 'Carrot.tex', ur'Carrot Documentation', ur'Ask Solem', 'manual'), ] # The name of an image file (relative to this directory) to place at the top of # the title page. #latex_logo = None # For "manual" documents, if this is true, then toplevel headings are parts, # not chapters. #latex_use_parts = False # Additional stuff for the LaTeX preamble. #latex_preamble = '' # Documents to append as an appendix to all manuals. #latex_appendices = [] # If false, no module index is generated. #latex_use_modindex = True html_theme = "nature" html_theme_path = ["_theme"] carrot-0.10.7/docs/faq.rst000644 000765 000024 00000000000 11334033664 017600 1carrot-0.10.7/FAQustar00askstaff000000 000000 carrot-0.10.7/docs/index.rst000644 000765 000024 00000000722 11334033664 015607 0ustar00askstaff000000 000000 .. Carrot documentation master file, created by sphinx-quickstart on Mon May 18 21:37:44 2009. You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. Carrot Documentation ================================== Contents: .. toctree:: :maxdepth: 3 introduction faq reference/index changelog Indices and tables ================== * :ref:`genindex` * :ref:`modindex` * :ref:`search` carrot-0.10.7/docs/introduction.rst000644 000765 000024 00000000000 11453345003 022073 1carrot-0.10.7/READMEustar00askstaff000000 000000 carrot-0.10.7/docs/Makefile000644 000765 000024 00000004477 11334033664 015421 0ustar00askstaff000000 000000 # Makefile for Sphinx documentation # # You can set these variables from the command line. SPHINXOPTS = SPHINXBUILD = sphinx-build PAPER = # Internal variables. PAPEROPT_a4 = -D latex_paper_size=a4 PAPEROPT_letter = -D latex_paper_size=letter ALLSPHINXOPTS = -d .build/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . .PHONY: help clean html web pickle htmlhelp latex changes linkcheck help: @echo "Please use \`make ' where is one of" @echo " html to make standalone HTML files" @echo " pickle to make pickle files" @echo " json to make JSON files" @echo " htmlhelp to make HTML files and a HTML help project" @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" @echo " changes to make an overview over all changed/added/deprecated items" @echo " linkcheck to check all external links for integrity" clean: -rm -rf .build/* html: mkdir -p .build/html .build/doctrees $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) .build/html @echo @echo "Build finished. The HTML pages are in .build/html." pickle: mkdir -p .build/pickle .build/doctrees $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) .build/pickle @echo @echo "Build finished; now you can process the pickle files." web: pickle json: mkdir -p .build/json .build/doctrees $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) .build/json @echo @echo "Build finished; now you can process the JSON files." htmlhelp: mkdir -p .build/htmlhelp .build/doctrees $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) .build/htmlhelp @echo @echo "Build finished; now you can run HTML Help Workshop with the" \ ".hhp project file in .build/htmlhelp." latex: mkdir -p .build/latex .build/doctrees $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) .build/latex @echo @echo "Build finished; the LaTeX files are in .build/latex." @echo "Run \`make all-pdf' or \`make all-ps' in that directory to" \ "run these through (pdf)latex." changes: mkdir -p .build/changes .build/doctrees $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) .build/changes @echo @echo "The overview file is in .build/changes." linkcheck: mkdir -p .build/linkcheck .build/doctrees $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) .build/linkcheck @echo @echo "Link check complete; look for any errors in the above output " \ "or in .build/linkcheck/output.txt." carrot-0.10.7/docs/reference/000755 000765 000024 00000000000 11453345106 015702 5ustar00askstaff000000 000000 carrot-0.10.7/docs/reference/carrot.backends.base.rst000644 000765 000024 00000000300 11334033664 022402 0ustar00askstaff000000 000000 ===================================== carrot.backends.base ===================================== .. currentmodule:: carrot.backends.base .. automodule:: carrot.backends.base :members: carrot-0.10.7/docs/reference/carrot.backends.pikachu.rst000644 000765 000024 00000000311 11401660116 023107 0ustar00askstaff000000 000000 ===================================== carrot.backends.pikachu ===================================== .. currentmodule:: carrot.backends.pikachu .. automodule:: carrot.backends.pikachu :members: carrot-0.10.7/docs/reference/carrot.backends.pyamqplib.rst000644 000765 000024 00000000317 11334033665 023477 0ustar00askstaff000000 000000 ===================================== carrot.backends.pyamqplib ===================================== .. currentmodule:: carrot.backends.pyamqplib .. automodule:: carrot.backends.pyamqplib :members: carrot-0.10.7/docs/reference/carrot.backends.pystomp.rst000644 000765 000024 00000000311 11334033665 023206 0ustar00askstaff000000 000000 ===================================== carrot.backends.pystomp ===================================== .. currentmodule:: carrot.backends.pystomp .. automodule:: carrot.backends.pystomp :members: carrot-0.10.7/docs/reference/carrot.backends.queue.rst000644 000765 000024 00000000303 11334033665 022620 0ustar00askstaff000000 000000 ===================================== carrot.backends.queue ===================================== .. currentmodule:: carrot.backends.queue .. automodule:: carrot.backends.queue :members: carrot-0.10.7/docs/reference/carrot.backends.rst000644 000765 000024 00000000261 11334033665 021500 0ustar00askstaff000000 000000 ===================================== carrot.backends ===================================== .. currentmodule:: carrot.backends .. automodule:: carrot.backends :members: carrot-0.10.7/docs/reference/carrot.connection.rst000644 000765 000024 00000000267 11334033665 022073 0ustar00askstaff000000 000000 ===================================== carrot.connection ===================================== .. currentmodule:: carrot.connection .. automodule:: carrot.connection :members: carrot-0.10.7/docs/reference/carrot.messaging.rst000644 000765 000024 00000000542 11334033665 021705 0ustar00askstaff000000 000000 ===================================== carrot.messaging ===================================== The introduction to carrot that was here previously has been moved to the `introduction`_ page. .. _`introduction`: http://ask.github.com/carrot/introduction.html#examples .. currentmodule:: carrot.messaging .. automodule:: carrot.messaging :members: carrot-0.10.7/docs/reference/carrot.serialization.rst000644 000765 000024 00000000300 11334033665 022575 0ustar00askstaff000000 000000 ===================================== carrot.serialization ===================================== .. currentmodule:: carrot.serialization .. automodule:: carrot.serialization :members: carrot-0.10.7/docs/reference/carrot.utils.rst000644 000765 000024 00000000250 11401660116 021054 0ustar00askstaff000000 000000 ===================================== carrot.utils ===================================== .. currentmodule:: carrot.utils .. automodule:: carrot.utils :members: carrot-0.10.7/docs/reference/index.rst000644 000765 000024 00000000601 11401660116 017532 0ustar00askstaff000000 000000 =========================== API Reference =========================== :Release: |version| :Date: |today| .. toctree:: :maxdepth: 2 carrot.connection carrot.messaging carrot.backends carrot.backends.base carrot.backends.pyamqplib carrot.backends.pikachu carrot.backends.pystomp carrot.backends.queue carrot.serialization carrot.utils carrot-0.10.7/docs/_theme/agogo/000755 000765 000024 00000000000 11453345106 016301 5ustar00askstaff000000 000000 carrot-0.10.7/docs/_theme/nature/000755 000765 000024 00000000000 11453345106 016503 5ustar00askstaff000000 000000 carrot-0.10.7/docs/_theme/nature/static/000755 000765 000024 00000000000 11453345106 017772 5ustar00askstaff000000 000000 carrot-0.10.7/docs/_theme/nature/theme.conf000644 000765 000024 00000000122 11334033664 020450 0ustar00askstaff000000 000000 [theme] inherit = basic stylesheet = nature.css pygments_style = tango [options] carrot-0.10.7/docs/_theme/nature/static/nature.css_t000644 000765 000024 00000007317 11334033664 022336 0ustar00askstaff000000 000000 /** * Sphinx stylesheet -- default theme * ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ */ @import url("basic.css"); /* -- page layout ----------------------------------------------------------- */ body { font-family: Arial, sans-serif; font-size: 100%; background-color: #111; color: #555; margin: 0; padding: 0; } hr{ border: 1px solid #B1B4B6; } div.document { background-color: #eee; } div.body { background-color: #ffffff; color: #3E4349; padding: 0 30px 30px 30px; font-size: 0.8em; } div.footer { color: #555; width: 100%; padding: 13px 0; text-align: center; font-size: 75%; } div.footer a { color: #444; text-decoration: underline; } div.related { background-color: #6BA81E; line-height: 32px; color: #fff; text-shadow: 0px 1px 0 #444; font-size: 0.80em; } div.related a { color: #E2F3CC; } div.sphinxsidebar { font-size: 0.75em; line-height: 1.5em; } div.sphinxsidebarwrapper{ padding: 20px 0; } div.sphinxsidebar h3, div.sphinxsidebar h4 { font-family: Arial, sans-serif; color: #222; font-size: 1.2em; font-weight: normal; margin: 0; padding: 5px 10px; background-color: #ddd; text-shadow: 1px 1px 0 white } div.sphinxsidebar h4{ font-size: 1.1em; } div.sphinxsidebar h3 a { color: #444; } div.sphinxsidebar p { color: #888; padding: 5px 20px; } div.sphinxsidebar p.topless { } div.sphinxsidebar ul { margin: 10px 20px; padding: 0; color: #000; } div.sphinxsidebar a { color: #444; } div.sphinxsidebar input { border: 1px solid #ccc; font-family: sans-serif; font-size: 1em; } div.sphinxsidebar input[type=text]{ margin-left: 20px; } /* -- body styles ----------------------------------------------------------- */ a { color: #005B81; text-decoration: none; } a:hover { color: #E32E00; text-decoration: underline; } div.body h1, div.body h2, div.body h3, div.body h4, div.body h5, div.body h6 { font-family: Arial, sans-serif; background-color: #BED4EB; font-weight: normal; color: #212224; margin: 30px 0px 10px 0px; padding: 5px 0 5px 10px; text-shadow: 0px 1px 0 white } div.body h1 { border-top: 20px solid white; margin-top: 0; font-size: 200%; } div.body h2 { font-size: 150%; background-color: #C8D5E3; } div.body h3 { font-size: 120%; background-color: #D8DEE3; } div.body h4 { font-size: 110%; background-color: #D8DEE3; } div.body h5 { font-size: 100%; background-color: #D8DEE3; } div.body h6 { font-size: 100%; background-color: #D8DEE3; } a.headerlink { color: #c60f0f; font-size: 0.8em; padding: 0 4px 0 4px; text-decoration: none; } a.headerlink:hover { background-color: #c60f0f; color: white; } div.body p, div.body dd, div.body li { text-align: justify; line-height: 1.5em; } div.admonition p.admonition-title + p { display: inline; } div.highlight{ background-color: white; } div.note { background-color: #eee; border: 1px solid #ccc; } div.seealso { background-color: #ffc; border: 1px solid #ff6; } div.topic { background-color: #eee; } div.warning { background-color: #ffe4e4; border: 1px solid #f66; } p.admonition-title { display: inline; } p.admonition-title:after { content: ":"; } pre { padding: 10px; background-color: White; color: #222; line-height: 1.2em; border: 1px solid #C6C9CB; font-size: 1.2em; margin: 1.5em 0 1.5em 0; -webkit-box-shadow: 1px 1px 1px #d8d8d8; -moz-box-shadow: 1px 1px 1px #d8d8d8; } tt { background-color: #ecf0f3; color: #222; padding: 1px 2px; font-size: 1.2em; font-family: monospace; }carrot-0.10.7/docs/_theme/nature/static/pygments.css000644 000765 000024 00000005235 11334033664 022360 0ustar00askstaff000000 000000 .c { color: #999988; font-style: italic } /* Comment */ .k { font-weight: bold } /* Keyword */ .o { font-weight: bold } /* Operator */ .cm { color: #999988; font-style: italic } /* Comment.Multiline */ .cp { color: #999999; font-weight: bold } /* Comment.preproc */ .c1 { color: #999988; font-style: italic } /* Comment.Single */ .gd { color: #000000; background-color: #ffdddd } /* Generic.Deleted */ .ge { font-style: italic } /* Generic.Emph */ .gr { color: #aa0000 } /* Generic.Error */ .gh { color: #999999 } /* Generic.Heading */ .gi { color: #000000; background-color: #ddffdd } /* Generic.Inserted */ .go { color: #111 } /* Generic.Output */ .gp { color: #555555 } /* Generic.Prompt */ .gs { font-weight: bold } /* Generic.Strong */ .gu { color: #aaaaaa } /* Generic.Subheading */ .gt { color: #aa0000 } /* Generic.Traceback */ .kc { font-weight: bold } /* Keyword.Constant */ .kd { font-weight: bold } /* Keyword.Declaration */ .kp { font-weight: bold } /* Keyword.Pseudo */ .kr { font-weight: bold } /* Keyword.Reserved */ .kt { color: #445588; font-weight: bold } /* Keyword.Type */ .m { color: #009999 } /* Literal.Number */ .s { color: #bb8844 } /* Literal.String */ .na { color: #008080 } /* Name.Attribute */ .nb { color: #999999 } /* Name.Builtin */ .nc { color: #445588; font-weight: bold } /* Name.Class */ .no { color: #ff99ff } /* Name.Constant */ .ni { color: #800080 } /* Name.Entity */ .ne { color: #990000; font-weight: bold } /* Name.Exception */ .nf { color: #990000; font-weight: bold } /* Name.Function */ .nn { color: #555555 } /* Name.Namespace */ .nt { color: #000080 } /* Name.Tag */ .nv { color: purple } /* Name.Variable */ .ow { font-weight: bold } /* Operator.Word */ .mf { color: #009999 } /* Literal.Number.Float */ .mh { color: #009999 } /* Literal.Number.Hex */ .mi { color: #009999 } /* Literal.Number.Integer */ .mo { color: #009999 } /* Literal.Number.Oct */ .sb { color: #bb8844 } /* Literal.String.Backtick */ .sc { color: #bb8844 } /* Literal.String.Char */ .sd { color: #bb8844 } /* Literal.String.Doc */ .s2 { color: #bb8844 } /* Literal.String.Double */ .se { color: #bb8844 } /* Literal.String.Escape */ .sh { color: #bb8844 } /* Literal.String.Heredoc */ .si { color: #bb8844 } /* Literal.String.Interpol */ .sx { color: #bb8844 } /* Literal.String.Other */ .sr { color: #808000 } /* Literal.String.Regex */ .s1 { color: #bb8844 } /* Literal.String.Single */ .ss { color: #bb8844 } /* Literal.String.Symbol */ .bp { color: #999999 } /* Name.Builtin.Pseudo */ .vc { color: #ff99ff } /* Name.Variable.Class */ .vg { color: #ff99ff } /* Name.Variable.Global */ .vi { color: #ff99ff } /* Name.Variable.Instance */ .il { color: #009999 } /* Literal.Number.Integer.Long */carrot-0.10.7/docs/_theme/agogo/layout.html000644 000765 000024 00000021600 11334033664 020504 0ustar00askstaff000000 000000 {%- block doctype -%} {%- endblock %} {%- set reldelim1 = reldelim1 is not defined and ' »' or reldelim1 %} {%- set reldelim2 = reldelim2 is not defined and ' |' or reldelim2 %} {%- macro relbar() %}

{{ _('Navigation') }}

    {%- for rellink in rellinks %}
  • {{ rellink[3] }} {%- if not loop.first %}{{ reldelim2 }}{% endif %}
    • {%- endfor %} {%- block rootrellink %}
  • {{ shorttitle|e }}{{ reldelim1 }}
    • {%- endblock %} {%- for parent in parents %}
  • {{ parent.title }}{{ reldelim1 }}
    • {%- endfor %} {%- block relbaritems %} {% endblock %}
{%- endmacro %} {%- macro sidebar() %} {%- if not embedded %}{% if not theme_nosidebar|tobool %}
{%- block sidebarlogo %} {%- if logo %} {%- endif %} {%- endblock %} {%- block sidebartoc %} {%- if display_toc %}

{{ _('Table Of Contents') }}

{{ toc }} {%- endif %} {%- endblock %} {%- block sidebarrel %} {%- if prev %}

{{ _('Previous topic') }}

{{ prev.title }}

{%- endif %} {%- if next %}

{{ _('Next topic') }}

{{ next.title }}

{%- endif %} {%- endblock %} {%- block sidebarsourcelink %} {%- if show_source and has_source and sourcename %}

{{ _('This Page') }}

{%- endif %} {%- endblock %} {%- if customsidebar %} {% include customsidebar %} {%- endif %} {%- block sidebarsearch %} {%- if pagename != "search" %} {%- endif %} {%- endblock %}
{%- endif %}{% endif %} {%- endmacro %} {{ metatags }} {%- if not embedded %} {%- set titlesuffix = " — "|safe + docstitle|e %} {%- else %} {%- set titlesuffix = "" %} {%- endif %} {{ title|striptags }}{{ titlesuffix }} {%- if not embedded %} {%- for scriptfile in script_files %} {%- endfor %} {%- if use_opensearch %} {%- endif %} {%- if favicon %} {%- endif %} {%- endif %} {%- block linktags %} {%- if hasdoc('about') %} {%- endif %} {%- if hasdoc('genindex') %} {%- endif %} {%- if hasdoc('search') %} {%- endif %} {%- if hasdoc('copyright') %} {%- endif %} {%- if parents %} {%- endif %} {%- if next %} {%- endif %} {%- if prev %} {%- endif %} {%- endblock %} {%- block extrahead %} {% endblock %}

{{ shorttitle|e }}

{%- for rellink in rellinks %} {{ rellink[3] }} {%- if not loop.last %}{{ reldelim2 }}{% endif %} {%- endfor %}
{%- block document %}
{%- if not embedded %}{% if not theme_nosidebar|tobool %}
{%- endif %}{% endif %}
{% block body %} {% endblock %}
{%- if not embedded %}{% if not theme_nosidebar|tobool %}
{%- endif %}{% endif %}
{%- endblock %}

{{ _('Contents') }}

{{ toctree() }}

{{ _('Search') }}

{{ _('Enter search terms or a module, class or function name.') }}

{%- block footer %}
{%- for rellink in rellinks %} {{ rellink[3] }} {%- if not loop.last %}{{ reldelim2 }}{% endif %} {%- endfor %} {%- if show_source and has_source and sourcename %}
{{ _('Show Source') }} {%- endif %}
{%- if hasdoc('copyright') %} {% trans path=pathto('copyright'), copyright=copyright|e %}© Copyright {{ copyright }}.{% endtrans %} {%- else %} {% trans copyright=copyright|e %}© Copyright {{ copyright }}.{% endtrans %} {%- endif %} {%- if last_updated %} {% trans last_updated=last_updated|e %}Last updated on {{ last_updated }}.{% endtrans %} {%- endif %} {%- if show_sphinx %}
{% trans sphinx_version=sphinx_version|e %}Created using Sphinx {{ sphinx_version }}.{% endtrans %} {%- endif %}
{%- endblock %}
carrot-0.10.7/docs/_theme/agogo/static/000755 000765 000024 00000000000 11453345106 017570 5ustar00askstaff000000 000000 carrot-0.10.7/docs/_theme/agogo/theme.conf000644 000765 000024 00000000655 11334033664 020261 0ustar00askstaff000000 000000 [theme] inherit = basic stylesheet = agogo.css pygments_style = tango [options] bodyfont = "Verdana", Arial, sans-serif headerfont = "Georgia", "Times New Roman", serif pagewidth = 70em documentwidth = 50em sidebarwidth = 20em bgcolor = #eeeeec headerbg = url(bgtop.png) top left repeat-x footerbg = url(bgfooter.png) top left repeat-x linkcolor = #ce5c00 headercolor1 = #204a87 headercolor2 = #3465a4 headerlinkcolor = #fcaf3e carrot-0.10.7/docs/_theme/agogo/static/agogo.css_t000644 000765 000024 00000011576 11334033664 021734 0ustar00askstaff000000 000000 * { margin: 0px; padding: 0px; } body { font-family: {{ theme_bodyfont }}; line-height: 1.4em; font-size: 14px; color: black; background-color: {{ theme_bgcolor }}; } /* Page layout */ div.header, div.content, div.footer { width: {{ theme_pagewidth }}; margin-left: auto; margin-right: auto; } div.header-wrapper { background: {{ theme_headerbg }}; border-bottom: 3px solid #2e3436; } /* Default body styles */ a { text-decoration: none; color: {{ theme_linkcolor }}; } .clearer { clear: both; } .left { float: left; } .right { float: right; } h1, h2, h3, h4 { font-family: {{ theme_headerfont }}; font-weight: normal; color: {{ theme_headercolor2 }}; margin-bottom: .8em; } h1 { color: {{ theme_headercolor1 }}; } h2 { padding-bottom: .5em; border-bottom: 1px solid {{ theme_headercolor2 }}; } a.headerlink { visibility: hidden; color: #dddddd; padding-left: .3em; } h1:hover > a.headerlink, h2:hover > a.headerlink, h3:hover > a.headerlink, h4:hover > a.headerlink, h5:hover > a.headerlink, h6:hover > a.headerlink, dt:hover > a.headerlink { visibility: visible; } /* Header */ div.header { padding-top: 10px; padding-bottom: 10px; } div.header h1 { font-family: {{ theme_headerfont }}; font-weight: normal; font-size: 160%; letter-spacing: .08em; } div.header h1 a { color: white; } div.header div.rel { margin-top: 1em; } div.header div.rel a { color: {{ theme_headerlinkcolor }}; letter-spacing: .1em; text-transform: uppercase; } /* Content */ div.content-wrapper { background-color: white; padding-top: 20px; padding-bottom: 20px; } div.document { width: {{ theme_documentwidth }}; float: left; } div.body { padding-right: 2em; text-align: justify; } div.document ul { margin-left: 1.2em; list-style-type: square; } div.document dd { margin-left: 1.2em; margin-top: .4em; margin-bottom: 1em; } div.document .section { margin-top: 1.7em; } div.document .section:first-child { margin-top: 0px; } div.document div.highlight { padding: 3px; background-color: #eeeeec; border-top: 2px solid #dddddd; border-bottom: 2px solid #dddddd; margin-top: .8em; margin-bottom: .8em; } div.document h2 { margin-top: .7em; } div.document p { margin-bottom: .5em; } div.document li.toctree-l1 { margin-bottom: 1em; } div.document .descname { font-weight: bold; } div.document .docutils.literal { background-color: #eeeeec; padding: 1px; } div.document .docutils.xref.literal { background-color: transparent; padding: 0px; } /* Sidebar */ div.sidebar { width: {{ theme_sidebarwidth }}; float: right; font-size: .9em; } div.sidebar h3 { color: #2e3436; text-transform: uppercase; font-size: 130%; letter-spacing: .1em; } div.sidebar ul { list-style-type: none; } div.sidebar li.toctree-l1 a { display: block; padding: 1px; border: 1px solid #dddddd; background-color: #eeeeec; margin-bottom: .4em; padding-left: 3px; color: #2e3436; } div.sidebar li.toctree-l2 a { background-color: transparent; border: none; border-bottom: 1px solid #dddddd; } div.sidebar li.toctree-l2:last-child a { border-bottom: none; } div.sidebar li.toctree-l1.current a { border-right: 5px solid {{ theme_headerlinkcolor }}; } div.sidebar li.toctree-l1.current li.toctree-l2 a { border-right: none; } /* Footer */ div.footer-wrapper { background: {{ theme_footerbg }}; border-top: 4px solid #babdb6; padding-top: 10px; padding-bottom: 10px; min-height: 80px; } div.footer, div.footer a { color: #888a85; } div.footer .right { text-align: right; } div.footer .left { text-transform: uppercase; } /* Styles copied form basic theme */ /* -- search page ----------------------------------------------------------- */ ul.search { margin: 10px 0 0 20px; padding: 0; } ul.search li { padding: 5px 0 5px 20px; background-image: url(file.png); background-repeat: no-repeat; background-position: 0 7px; } ul.search li a { font-weight: bold; } ul.search li div.context { color: #888; margin: 2px 0 0 30px; text-align: left; } ul.keywordmatches li.goodmatch a { font-weight: bold; } /* -- index page ------------------------------------------------------------ */ table.contentstable { width: 90%; } table.contentstable p.biglink { line-height: 150%; } a.biglink { font-size: 1.3em; } span.linkdescr { font-style: italic; padding-top: 5px; font-size: 90%; } /* -- general index --------------------------------------------------------- */ table.indextable td { text-align: left; vertical-align: top; } table.indextable dl, table.indextable dd { margin-top: 0; margin-bottom: 0; } table.indextable tr.pcap { height: 10px; } table.indextable tr.cap { margin-top: 10px; background-color: #f2f2f2; } img.toggler { margin-right: 3px; margin-top: 3px; cursor: pointer; } carrot-0.10.7/docs/_theme/agogo/static/bgfooter.png000644 000765 000024 00000000662 11334033664 022112 0ustar00askstaff000000 000000 ‰PNG  IHDRPÜ2ÏæsRGB®ÎébKGDÓ×Ï´ Bb pHYs  šœtIMEÙ4\‰ÖtEXtCommentCreated with GIMPW IDAT8Ëm’;’%13ÜÿjkŒ1Ça >Ò{1m©C¢(’òç÷_Òßáù>ìƒZ7  §Æz§­fH€[vÒ|Ÿ•BTë:m8æ8è›2GK#ì¼#À>k׎9!Ò¾K8s(igîäŒIS³jl;®Ñaå͇Áe]iÝâw Ñä "HŒV£ªyb çÐÁö¬¸_ Ž\i·ÿ …£#l*Š›äå–ˆÙþ±j£ä:h‘Ýv0qÑËÚePzD=Ù¸@æ BÎdô&dútÙ3Ü°Î|ÇžˆS»Ï³Æ6 œó亓ŸøpwÿFÙ/ðOªÞì\éÒˆëìk ÿHy3ÿ°”IEND®B`‚carrot-0.10.7/docs/_theme/agogo/static/bgtop.png000644 000765 000024 00000000656 11334033664 021421 0ustar00askstaff000000 000000 ‰PNG  IHDRPÜ2ÏæsRGB®ÎébKGDUWS’!ÿ pHYs  šœtIMEÙ-”†ô³tEXtCommentCreated with GIMPW IDAT8Ë]’ËŽ@!CÄ_Ÿd2mgAQ¼®y”¶ñû÷#|’q„Iä® Ôw üX­À’/Úõ`ᶮqŽdÏϪfH=Cû&H9M Cúv"ÜíÀQµVýì1´ÆTâ^»“ÎœÊËF#‰lvTت&jÓ0PG§yŸV!.6±›ø`H¢SsVˆf´ÙáCõ¦åöûàÊxØAîÆŽ/êš]„³áMyŒT›…Ú–ÛÄß9vòònµãˆeªªÁ…`cSLFUs­éµ%„éJ±'½öh´ v|¾îmé5kôuü|äðø—*l>Ä*š6DŒå¤»ÏzdœÝþDÞ£”X?»IEND®B`‚carrot-0.10.7/docs/_ext/applyxrefs.py000644 000765 000024 00000004151 11334033664 017454 0ustar00askstaff000000 000000 """Adds xref targets to the top of files.""" import sys import os testing = False DONT_TOUCH = ( './index.txt', ) def target_name(fn): if fn.endswith('.txt'): fn = fn[:-4] return '_' + fn.lstrip('./').replace('/', '-') def process_file(fn, lines): lines.insert(0, '\n') lines.insert(0, '.. %s:\n' % target_name(fn)) try: f = open(fn, 'w') except IOError: print("Can't open %s for writing. Not touching it." % fn) return try: f.writelines(lines) except IOError: print("Can't write to %s. Not touching it." % fn) finally: f.close() def has_target(fn): try: f = open(fn, 'r') except IOError: print("Can't open %s. Not touching it." % fn) return (True, None) readok = True try: lines = f.readlines() except IOError: print("Can't read %s. Not touching it." % fn) readok = False finally: f.close() if not readok: return (True, None) #print fn, len(lines) if len(lines) < 1: print("Not touching empty file %s." % fn) return (True, None) if lines[0].startswith('.. _'): return (True, None) return (False, lines) def main(argv=None): if argv is None: argv = sys.argv if len(argv) == 1: argv.extend('.') files = [] for root in argv[1:]: for (dirpath, dirnames, filenames) in os.walk(root): files.extend([(dirpath, f) for f in filenames]) files.sort() files = [os.path.join(p, fn) for p, fn in files if fn.endswith('.txt')] #print files for fn in files: if fn in DONT_TOUCH: print("Skipping blacklisted file %s." % fn) continue target_found, lines = has_target(fn) if not target_found: if testing: print '%s: %s' % (fn, lines[0]), else: print "Adding xref to %s" % fn process_file(fn, lines) else: print "Skipping %s: already has a xref" % fn if __name__ == '__main__': sys.exit(main()) carrot-0.10.7/docs/_ext/literals_to_xrefs.py000644 000765 000024 00000011155 11334033664 021011 0ustar00askstaff000000 000000 """ Runs through a reST file looking for old-style literals, and helps replace them with new-style references. """ import re import sys import shelve refre = re.compile(r'``([^`\s]+?)``') ROLES = ( 'attr', 'class', "djadmin", 'data', 'exc', 'file', 'func', 'lookup', 'meth', 'mod', "djadminopt", "ref", "setting", "term", "tfilter", "ttag", # special "skip", ) ALWAYS_SKIP = [ "NULL", "True", "False", ] def fixliterals(fname): data = open(fname).read() last = 0 new = [] storage = shelve.open("/tmp/literals_to_xref.shelve") lastvalues = storage.get("lastvalues", {}) for m in refre.finditer(data): new.append(data[last:m.start()]) last = m.end() line_start = data.rfind("\n", 0, m.start()) line_end = data.find("\n", m.end()) prev_start = data.rfind("\n", 0, line_start) next_end = data.find("\n", line_end + 1) # Skip always-skip stuff if m.group(1) in ALWAYS_SKIP: new.append(m.group(0)) continue # skip when the next line is a title next_line = data[m.end():next_end].strip() if next_line[0] in "!-/:-@[-`{-~" and \ all(c == next_line[0] for c in next_line): new.append(m.group(0)) continue sys.stdout.write("\n"+"-"*80+"\n") sys.stdout.write(data[prev_start+1:m.start()]) sys.stdout.write(colorize(m.group(0), fg="red")) sys.stdout.write(data[m.end():next_end]) sys.stdout.write("\n\n") replace_type = None while replace_type is None: replace_type = raw_input( colorize("Replace role: ", fg="yellow")).strip().lower() if replace_type and replace_type not in ROLES: replace_type = None if replace_type == "": new.append(m.group(0)) continue if replace_type == "skip": new.append(m.group(0)) ALWAYS_SKIP.append(m.group(1)) continue default = lastvalues.get(m.group(1), m.group(1)) if default.endswith("()") and \ replace_type in ("class", "func", "meth"): default = default[:-2] replace_value = raw_input( colorize("Text [", fg="yellow") + default + \ colorize("]: ", fg="yellow")).strip() if not replace_value: replace_value = default new.append(":%s:`%s`" % (replace_type, replace_value)) lastvalues[m.group(1)] = replace_value new.append(data[last:]) open(fname, "w").write("".join(new)) storage["lastvalues"] = lastvalues storage.close() def colorize(text='', opts=(), **kwargs): """ Returns your text, enclosed in ANSI graphics codes. Depends on the keyword arguments 'fg' and 'bg', and the contents of the opts tuple/list. Returns the RESET code if no parameters are given. Valid colors: 'black', 'red', 'green', 'yellow', 'blue', 'magenta', 'cyan', 'white' Valid options: 'bold' 'underscore' 'blink' 'reverse' 'conceal' 'noreset' - string will not be auto-terminated with the RESET code Examples: colorize('hello', fg='red', bg='blue', opts=('blink',)) colorize() colorize('goodbye', opts=('underscore',)) print colorize('first line', fg='red', opts=('noreset',)) print 'this should be red too' print colorize('and so should this') print 'this should not be red' """ color_names = ('black', 'red', 'green', 'yellow', 'blue', 'magenta', 'cyan', 'white') foreground = dict([(color_names[x], '3%s' % x) for x in range(8)]) background = dict([(color_names[x], '4%s' % x) for x in range(8)]) RESET = '0' opt_dict = {'bold': '1', 'underscore': '4', 'blink': '5', 'reverse': '7', 'conceal': '8'} text = str(text) code_list = [] if text == '' and len(opts) == 1 and opts[0] == 'reset': return '\x1b[%sm' % RESET for k, v in kwargs.iteritems(): if k == 'fg': code_list.append(foreground[v]) elif k == 'bg': code_list.append(background[v]) for o in opts: if o in opt_dict: code_list.append(opt_dict[o]) if 'noreset' not in opts: text = text + '\x1b[%sm' % RESET return ('\x1b[%sm' % ';'.join(code_list)) + text if __name__ == '__main__': try: fixliterals(sys.argv[1]) except (KeyboardInterrupt, SystemExit): print carrot-0.10.7/docs/.static/.keep000644 000765 000024 00000000000 11401660755 016227 0ustar00askstaff000000 000000 carrot-0.10.7/carrot.egg-info/dependency_links.txt000644 000765 000024 00000000001 11453345105 022045 0ustar00askstaff000000 000000 carrot-0.10.7/carrot.egg-info/not-zip-safe000644 000765 000024 00000000001 11453345105 020225 0ustar00askstaff000000 000000 carrot-0.10.7/carrot.egg-info/PKG-INFO000644 000765 000024 00000040555 11453345105 017105 0ustar00askstaff000000 000000 Metadata-Version: 1.0 Name: carrot Version: 0.10.7 Summary: AMQP Messaging Framework for Python Home-page: http://github.com/ask/carrot/ Author: Ask Solem Author-email: ask@celeryproject.org License: UNKNOWN Description: ############################################## carrot - AMQP Messaging Framework for Python ############################################## :Version: 0.10.7 **NOTE** This release contains backward-incompatible changes. Please read the `Changelog`_ for more information. .. _`Changelog`: http://ask.github.com/carrot/changelog.html Introduction ------------ `carrot` is an `AMQP`_ messaging queue framework. AMQP is the Advanced Message Queuing Protocol, an open standard protocol for message orientation, queuing, routing, reliability and security. The aim of `carrot` is to make messaging in Python as easy as possible by providing a high-level interface for producing and consuming messages. At the same time it is a goal to re-use what is already available as much as possible. `carrot` has pluggable messaging back-ends, so it is possible to support several messaging systems. Currently, there is support for `AMQP`_ (`py-amqplib`_, `pika`_), `STOMP`_ (`python-stomp`_). There's also an in-memory backend for testing purposes, using the `Python queue module`_. Several AMQP message broker implementations exists, including `RabbitMQ`_, `ZeroMQ`_ and `Apache ActiveMQ`_. You'll need to have one of these installed, personally we've been using `RabbitMQ`_. Before you start playing with ``carrot``, you should probably read up on AMQP, and you could start with the excellent article about using RabbitMQ under Python, `Rabbits and warrens`_. For more detailed information, you can refer to the `Wikipedia article about AMQP`_. .. _`RabbitMQ`: http://www.rabbitmq.com/ .. _`ZeroMQ`: http://www.zeromq.org/ .. _`AMQP`: http://amqp.org .. _`STOMP`: http://stomp.codehaus.org .. _`python-stomp`: http://bitbucket.org/asksol/python-stomp .. _`Python Queue module`: http://docs.python.org/library/queue.html .. _`Apache ActiveMQ`: http://activemq.apache.org/ .. _`Django`: http://www.djangoproject.com/ .. _`Rabbits and warrens`: http://blogs.digitar.com/jjww/2009/01/rabbits-and-warrens/ .. _`py-amqplib`: http://barryp.org/software/py-amqplib/ .. _`pika`: http://github.com/tonyg/pika .. _`Wikipedia article about AMQP`: http://en.wikipedia.org/wiki/AMQP Documentation ------------- Carrot is using Sphinx, and the latest documentation is available at GitHub: http://github.com/ask/carrot/ Installation ============ You can install ``carrot`` either via the Python Package Index (PyPI) or from source. To install using ``pip``,:: $ pip install carrot To install using ``easy_install``,:: $ easy_install carrot If you have downloaded a source tarball you can install it by doing the following,:: $ python setup.py build # python setup.py install # as root Terminology =========== There are some concepts you should be familiar with before starting: * Publishers Publishers sends messages to an exchange. * Exchanges Messages are sent to exchanges. Exchanges are named and can be configured to use one of several routing algorithms. The exchange routes the messages to consumers by matching the routing key in the message with the routing key the consumer provides when binding to the exchange. * Consumers Consumers declares a queue, binds it to a exchange and receives messages from it. * Queues Queues receive messages sent to exchanges. The queues are declared by consumers. * Routing keys Every message has a routing key. The interpretation of the routing key depends on the exchange type. There are four default exchange types defined by the AMQP standard, and vendors can define custom types (so see your vendors manual for details). These are the default exchange types defined by AMQP/0.8: * Direct exchange Matches if the routing key property of the message and the ``routing_key`` attribute of the consumer are identical. * Fan-out exchange Always matches, even if the binding does not have a routing key. * Topic exchange Matches the routing key property of the message by a primitive pattern matching scheme. The message routing key then consists of words separated by dots (``"."``, like domain names), and two special characters are available; star (``"*"``) and hash (``"#"``). The star matches any word, and the hash matches zero or more words. For example ``"*.stock.#"`` matches the routing keys ``"usd.stock"`` and ``"eur.stock.db"`` but not ``"stock.nasdaq"``. Examples ======== Creating a connection --------------------- You can set up a connection by creating an instance of ``carrot.messaging.BrokerConnection``, with the appropriate options for your broker: >>> from carrot.connection import BrokerConnection >>> conn = BrokerConnection(hostname="localhost", port=5672, ... userid="test", password="test", ... virtual_host="test") If you're using Django you can use the ``carrot.connection.DjangoBrokerConnection`` class instead, which loads the connection settings from your ``settings.py``:: BROKER_HOST = "localhost" BROKER_PORT = 5672 BROKER_USER = "test" BROKER_PASSWORD = "secret" BROKER_VHOST = "/test" Then create a connection by doing: >>> from carrot.connection import DjangoBrokerConnection >>> conn = DjangoBrokerConnection() Receiving messages using a Consumer ----------------------------------- First we open up a Python shell and start a message consumer. This consumer declares a queue named ``"feed"``, receiving messages with the routing key ``"importer"`` from the ``"feed"`` exchange. The example then uses the consumers ``wait()`` method to go into consume mode, where it continuously polls the queue for new messages, and when a message is received it passes the message to all registered callbacks. >>> from carrot.messaging import Consumer >>> consumer = Consumer(connection=conn, queue="feed", ... exchange="feed", routing_key="importer") >>> def import_feed_callback(message_data, message): ... feed_url = message_data["import_feed"] ... print("Got feed import message for: %s" % feed_url) ... # something importing this feed url ... # import_feed(feed_url) ... message.ack() >>> consumer.register_callback(import_feed_callback) >>> consumer.wait() # Go into the consumer loop. Sending messages using a Publisher ---------------------------------- Then we open up another Python shell to send some messages to the consumer defined in the last section. >>> from carrot.messaging import Publisher >>> publisher = Publisher(connection=conn, ... exchange="feed", routing_key="importer") >>> publisher.send({"import_feed": "http://cnn.com/rss/edition.rss"}) >>> publisher.close() Look in the first Python shell again (where ``consumer.wait()`` is running), where the following text has been printed to the screen:: Got feed import message for: http://cnn.com/rss/edition.rss Serialization of Data ----------------------- By default every message is encoded using `JSON`_, so sending Python data structures like dictionaries and lists works. `YAML`_, `msgpack`_ and Python's built-in ``pickle`` module is also supported, and if needed you can register any custom serialization scheme you want to use. .. _`JSON`: http://www.json.org/ .. _`YAML`: http://yaml.org/ .. _`msgpack`: http://msgpack.sourceforge.net/ Each option has its advantages and disadvantages. ``json`` -- JSON is supported in many programming languages, is now a standard part of Python (since 2.6), and is fairly fast to decode using the modern Python libraries such as ``cjson or ``simplejson``. The primary disadvantage to ``JSON`` is that it limits you to the following data types: strings, unicode, floats, boolean, dictionaries, and lists. Decimals and dates are notably missing. Also, binary data will be transferred using base64 encoding, which will cause the transferred data to be around 34% larger than an encoding which supports native binary types. However, if your data fits inside the above constraints and you need cross-language support, the default setting of ``JSON`` is probably your best choice. ``pickle`` -- If you have no desire to support any language other than Python, then using the ``pickle`` encoding will gain you the support of all built-in Python data types (except class instances), smaller messages when sending binary files, and a slight speedup over ``JSON`` processing. ``yaml`` -- YAML has many of the same characteristics as ``json``, except that it natively supports more data types (including dates, recursive references, etc.) However, the Python libraries for YAML are a good bit slower than the libraries for JSON. If you need a more expressive set of data types and need to maintain cross-language compatibility, then ``YAML`` may be a better fit than the above. To instruct carrot to use an alternate serialization method, use one of the following options. 1. Set the serialization option on a per-Publisher basis: >>> from carrot.messaging import Publisher >>> publisher = Publisher(connection=conn, ... exchange="feed", routing_key="importer", ... serializer="yaml") 2. Set the serialization option on a per-call basis >>> from carrot.messaging import Publisher >>> publisher = Publisher(connection=conn, ... exchange="feed", routing_key="importer") >>> publisher.send({"import_feed": "http://cnn.com/rss/edition.rss"}, ... serializer="pickle") >>> publisher.close() Note that ``Consumer``s do not need the serialization method specified in their code. They can auto-detect the serialization method since we supply the ``Content-type`` header as part of the AMQP message. Sending raw data without Serialization --------------------------------------- In some cases, you don't need your message data to be serialized. If you pass in a plain string or unicode object as your message, then carrot will not waste cycles serializing/deserializing the data. You can optionally specify a ``content_type`` and ``content_encoding`` for the raw data: >>> from carrot.messaging import Publisher >>> publisher = Publisher(connection=conn, ... exchange="feed", routing_key="import_pictures") >>> publisher.send(open('~/my_picture.jpg','rb').read(), content_type="image/jpeg", content_encoding="binary") >>> publisher.close() The ``message`` object returned by the ``Consumer`` class will have a ``content_type`` and ``content_encoding`` attribute. Receiving messages without a callback -------------------------------------- You can also poll the queue manually, by using the ``fetch`` method. This method returns a ``Message`` object, from where you can get the message body, de-serialize the body to get the data, acknowledge, reject or re-queue the message. >>> consumer = Consumer(connection=conn, queue="feed", ... exchange="feed", routing_key="importer") >>> message = consumer.fetch() >>> if message: ... message_data = message.payload ... message.ack() ... else: ... # No messages waiting on the queue. >>> consumer.close() Sub-classing the messaging classes ---------------------------------- The ``Consumer``, and ``Publisher`` classes can also be sub classed. Thus you can define the above publisher and consumer like so: >>> from carrot.messaging import Publisher, Consumer >>> class FeedPublisher(Publisher): ... exchange = "feed" ... routing_key = "importer" ... ... def import_feed(self, feed_url): ... return self.send({"action": "import_feed", ... "feed_url": feed_url}) >>> class FeedConsumer(Consumer): ... queue = "feed" ... exchange = "feed" ... routing_key = "importer" ... ... def receive(self, message_data, message): ... action = message_data["action"] ... if action == "import_feed": ... # something importing this feed ... # import_feed(message_data["feed_url"]) message.ack() ... else: ... raise Exception("Unknown action: %s" % action) >>> publisher = FeedPublisher(connection=conn) >>> publisher.import_feed("http://cnn.com/rss/edition.rss") >>> publisher.close() >>> consumer = FeedConsumer(connection=conn) >>> consumer.wait() # Go into the consumer loop. Getting Help ============ Mailing list ------------ Join the `carrot-users`_ mailing list. .. _`carrot-users`: http://groups.google.com/group/carrot-users/ Bug tracker =========== If you have any suggestions, bug reports or annoyances please report them to our issue tracker at http://github.com/ask/carrot/issues/ Contributing ============ Development of ``carrot`` happens at Github: http://github.com/ask/carrot You are highly encouraged to participate in the development. If you don't like Github (for some reason) you're welcome to send regular patches. License ======= This software is licensed under the ``New BSD License``. See the ``LICENSE`` file in the top distribution directory for the full license text. Platform: any Classifier: Development Status :: 4 - Beta Classifier: Framework :: Django Classifier: Operating System :: OS Independent Classifier: Programming Language :: Python Classifier: License :: OSI Approved :: BSD License Classifier: Intended Audience :: Developers Classifier: Topic :: Communications Classifier: Topic :: System :: Distributed Computing Classifier: Topic :: Software Development :: Libraries :: Python Modules carrot-0.10.7/carrot.egg-info/requires.txt000644 000765 000024 00000000024 11453345105 020373 0ustar00askstaff000000 000000 anyjson amqplib>=0.6carrot-0.10.7/carrot.egg-info/SOURCES.txt000644 000765 000024 00000003202 11453345106 017661 0ustar00askstaff000000 000000 AUTHORS Changelog FAQ INSTALL LICENSE MANIFEST.in README README.rst THANKS TODO setup.cfg setup.py carrot/__init__.py carrot/connection.py carrot/messaging.py carrot/serialization.py carrot/utils.py carrot.egg-info/PKG-INFO carrot.egg-info/SOURCES.txt carrot.egg-info/dependency_links.txt carrot.egg-info/not-zip-safe carrot.egg-info/requires.txt carrot.egg-info/top_level.txt carrot/backends/__init__.py carrot/backends/base.py carrot/backends/librabbitmq.py carrot/backends/pikachu.py carrot/backends/pyamqplib.py carrot/backends/pystomp.py carrot/backends/queue.py docs/Makefile docs/changelog.rst docs/conf.py docs/faq.rst docs/index.rst docs/introduction.rst docs/.static/.keep docs/_ext/applyxrefs.py docs/_ext/literals_to_xrefs.py docs/_theme/agogo/layout.html docs/_theme/agogo/theme.conf docs/_theme/agogo/static/agogo.css_t docs/_theme/agogo/static/bgfooter.png docs/_theme/agogo/static/bgtop.png docs/_theme/nature/theme.conf docs/_theme/nature/static/nature.css_t docs/_theme/nature/static/pygments.css docs/reference/carrot.backends.base.rst docs/reference/carrot.backends.pikachu.rst docs/reference/carrot.backends.pyamqplib.rst docs/reference/carrot.backends.pystomp.rst docs/reference/carrot.backends.queue.rst docs/reference/carrot.backends.rst docs/reference/carrot.connection.rst docs/reference/carrot.messaging.rst docs/reference/carrot.serialization.rst docs/reference/carrot.utils.rst docs/reference/index.rst tests/__init__.py tests/backend.py tests/test_django.py tests/test_examples.py tests/test_pyamqplib.py tests/test_pyqueue.py tests/test_serialization.py tests/test_utils.py tests/test_with_statement.py tests/utils.py tests/xxxstmop.pycarrot-0.10.7/carrot.egg-info/top_level.txt000644 000765 000024 00000000007 11453345105 020526 0ustar00askstaff000000 000000 carrot carrot-0.10.7/carrot/__init__.py000644 000765 000024 00000000372 11453344767 016435 0ustar00askstaff000000 000000 """AMQP Messaging Framework for Python""" VERSION = (0, 10, 7) __version__ = ".".join(map(str, VERSION)) __author__ = "Ask Solem" __contact__ = "ask@celeryproject.org" __homepage__ = "http://github.com/ask/carrot/" __docformat__ = "restructuredtext" carrot-0.10.7/carrot/backends/000755 000765 000024 00000000000 11453345106 016060 5ustar00askstaff000000 000000 carrot-0.10.7/carrot/connection.py000644 000765 000024 00000022611 11446074070 017022 0ustar00askstaff000000 000000 """ Getting a connection to the AMQP server. """ import socket import warnings from collections import deque from copy import copy from Queue import Queue, Empty as QueueEmpty from amqplib.client_0_8.connection import AMQPConnectionException from carrot.backends import get_backend_cls DEFAULT_CONNECT_TIMEOUT = 5 # seconds SETTING_PREFIX = "BROKER" COMPAT_SETTING_PREFIX = "AMQP" ARG_TO_DJANGO_SETTING = { "hostname": "HOST", "userid": "USER", "password": "PASSWORD", "virtual_host": "VHOST", "port": "PORT", } SETTING_DEPRECATED_FMT = "Setting %s has been renamed to %s and is " \ "scheduled for removal in version 1.0." class BrokerConnection(object): """A network/socket connection to an AMQP message broker. :param hostname: see :attr:`hostname`. :param userid: see :attr:`userid`. :param password: see :attr:`password`. :keyword virtual_host: see :attr:`virtual_host`. :keyword port: see :attr:`port`. :keyword insist: see :attr:`insist`. :keyword connect_timeout: see :attr:`connect_timeout`. :keyword ssl: see :attr:`ssl`. .. attribute:: hostname The hostname to the AMQP server .. attribute:: userid A valid username used to authenticate to the server. .. attribute:: password The password used to authenticate to the server. .. attribute:: virtual_host The name of the virtual host to work with. This virtual host must exist on the server, and the user must have access to it. Consult your brokers manual for help with creating, and mapping users to virtual hosts. Default is ``"/"``. .. attribute:: port The port of the AMQP server. Default is ``5672`` (amqp). .. attribute:: insist Insist on connecting to a server. In a configuration with multiple load-sharing servers, the insist option tells the server that the client is insisting on a connection to the specified server. Default is ``False``. .. attribute:: connect_timeout The timeout in seconds before we give up connecting to the server. The default is no timeout. .. attribute:: ssl Use SSL to connect to the server. The default is ``False``. .. attribute:: backend_cls The messaging backend class used. Defaults to the ``pyamqplib`` backend. """ virtual_host = "/" port = None insist = False connect_timeout = DEFAULT_CONNECT_TIMEOUT ssl = False _closed = True backend_cls = None ConnectionException = AMQPConnectionException @property def host(self): """The host as a hostname/port pair separated by colon.""" return ":".join([self.hostname, str(self.port)]) def __init__(self, hostname=None, userid=None, password=None, virtual_host=None, port=None, pool=None, **kwargs): self.hostname = hostname self.userid = userid self.password = password self.virtual_host = virtual_host or self.virtual_host self.port = port or self.port self.insist = kwargs.get("insist", self.insist) self.pool = pool self.connect_timeout = kwargs.get("connect_timeout", self.connect_timeout) self.ssl = kwargs.get("ssl", self.ssl) self.backend_cls = kwargs.get("backend_cls", None) self._closed = None self._connection = None def __copy__(self): return self.__class__(self.hostname, self.userid, self.password, self.virtual_host, self.port, insist=self.insist, connect_timeout=self.connect_timeout, ssl=self.ssl, backend_cls=self.backend_cls, pool=self.pool) @property def connection(self): if self._closed == True: return if not self._connection: self._connection = self._establish_connection() self._closed = False return self._connection def __enter__(self): return self def __exit__(self, e_type, e_value, e_trace): if e_type: raise e_type(e_value) self.close() def _establish_connection(self): return self.create_backend().establish_connection() def get_backend_cls(self): """Get the currently used backend class.""" backend_cls = self.backend_cls if not backend_cls or isinstance(backend_cls, basestring): backend_cls = get_backend_cls(backend_cls) return backend_cls def create_backend(self): """Create a new instance of the current backend in :attr:`backend_cls`.""" backend_cls = self.get_backend_cls() return backend_cls(connection=self) def get_channel(self): """Request a new AMQP channel.""" return self.connection.channel() def connect(self): """Establish a connection to the AMQP server.""" self._closed = False return self.connection def drain_events(self, **kwargs): return self.connection.drain_events(**kwargs) def close(self): """Close the currently open connection.""" try: if self._connection: backend = self.create_backend() backend.close_connection(self._connection) except socket.error: pass self._closed = True def release(self): if not self.pool: raise NotImplementedError( "Trying to release connection not part of a pool") self.pool.release(self) # For backwards compatability. AMQPConnection = BrokerConnection class ConnectionLimitExceeded(Exception): """The maximum number of pool connections has been exceeded.""" class ConnectionPool(object): def __init__(self, source_connection, min=2, max=None, preload=True): self.source_connection = source_connection self.min = min self.max = max self.preload = preload self.source_connection.pool = self self._connections = Queue() self._dirty = deque() self._connections.put(self.source_connection) for i in range(min - 1): self._connections.put_nowait(self._new_connection()) def acquire(self, block=False, timeout=None, connect_timeout=None): try: conn = self._connections.get(block=block, timeout=timeout) except QueueEmpty: conn = self._new_connection() self._dirty.append(conn) if connect_timeout is not None: conn.connect_timeout = connect_timeout return conn def release(self, connection): self._dirty.remove(connection) self._connections.put_nowait(connection) def _new_connection(self): if len(self._dirty) >= self.max: raise ConnectionLimitExceeded(self.max) return copy(self.source_connection) def get_django_conninfo(settings=None): # FIXME can't wait to remove this mess in 1.0 [askh] ci = {} if settings is None: from django.conf import settings ci["backend_cls"] = getattr(settings, "CARROT_BACKEND", None) for arg_name, setting_name in ARG_TO_DJANGO_SETTING.items(): setting = "%s_%s" % (SETTING_PREFIX, setting_name) compat_setting = "%s_%s" % (COMPAT_SETTING_PREFIX, setting_name) if hasattr(settings, setting): ci[arg_name] = getattr(settings, setting, None) elif hasattr(settings, compat_setting): ci[arg_name] = getattr(settings, compat_setting, None) warnings.warn(DeprecationWarning(SETTING_DEPRECATED_FMT % ( compat_setting, setting))) if "hostname" not in ci: if hasattr(settings, "AMQP_SERVER"): ci["hostname"] = settings.AMQP_SERVER warnings.warn(DeprecationWarning( "AMQP_SERVER has been renamed to BROKER_HOST and is" "scheduled for removal in version 1.0.")) return ci class DjangoBrokerConnection(BrokerConnection): """A version of :class:`BrokerConnection` that takes configuration from the Django ``settings.py`` module. :keyword hostname: The hostname of the AMQP server to connect to, if not provided this is taken from ``settings.BROKER_HOST``. :keyword userid: The username of the user to authenticate to the server as. If not provided this is taken from ``settings.BROKER_USER``. :keyword password: The users password. If not provided this is taken from ``settings.BROKER_PASSWORD``. :keyword virtual_host: The name of the virtual host to work with. This virtual host must exist on the server, and the user must have access to it. Consult your brokers manual for help with creating, and mapping users to virtual hosts. If not provided this is taken from ``settings.BROKER_VHOST``. :keyword port: The port the AMQP server is running on. If not provided this is taken from ``settings.BROKER_PORT``, or if that is not set, the default is ``5672`` (amqp). """ def __init__(self, *args, **kwargs): settings = kwargs.pop("settings", None) kwargs = dict(get_django_conninfo(settings), **kwargs) super(DjangoBrokerConnection, self).__init__(*args, **kwargs) # For backwards compatability. DjangoAMQPConnection = DjangoBrokerConnection carrot-0.10.7/carrot/messaging.py000644 000765 000024 00000110310 11451422605 016627 0ustar00askstaff000000 000000 """ Sending/Receiving Messages. """ from itertools import count from carrot.utils import gen_unique_id import warnings from carrot import serialization class Consumer(object): """Message consumer. :param connection: see :attr:`connection`. :param queue: see :attr:`queue`. :param exchange: see :attr:`exchange`. :param routing_key: see :attr:`routing_key`. :keyword durable: see :attr:`durable`. :keyword auto_delete: see :attr:`auto_delete`. :keyword exclusive: see :attr:`exclusive`. :keyword exchange_type: see :attr:`exchange_type`. :keyword auto_ack: see :attr:`auto_ack`. :keyword no_ack: see :attr:`no_ack`. :keyword auto_declare: see :attr:`auto_declare`. .. attribute:: connection The connection to the broker. A :class:`carrot.connection.BrokerConnection` instance. .. attribute:: queue Name of the queue. .. attribute:: exchange Name of the exchange the queue binds to. .. attribute:: routing_key The routing key (if any). The interpretation of the routing key depends on the value of the :attr:`exchange_type` attribute: * direct exchange Matches if the routing key property of the message and the :attr:`routing_key` attribute are identical. * fanout exchange Always matches, even if the binding does not have a key. * topic exchange Matches the routing key property of the message by a primitive pattern matching scheme. The message routing key then consists of words separated by dots (``"."``, like domain names), and two special characters are available; star (``"*"``) and hash (``"#"``). The star matches any word, and the hash matches zero or more words. For example ``"*.stock.#"`` matches the routing keys ``"usd.stock"`` and ``"eur.stock.db"`` but not ``"stock.nasdaq"``. .. attribute:: durable Durable exchanges remain active when a server restarts. Non-durable exchanges (transient exchanges) are purged when a server restarts. Default is ``True``. .. attribute:: auto_delete If set, the exchange is deleted when all queues have finished using it. Default is ``False``. .. attribute:: exclusive Exclusive queues may only be consumed from by the current connection. When :attr:`exclusive` is on, this also implies :attr:`auto_delete`. Default is ``False``. .. attribute:: exchange_type AMQP defines four default exchange types (routing algorithms) that covers most of the common messaging use cases. An AMQP broker can also define additional exchange types, so see your message brokers manual for more information about available exchange types. * Direct Direct match between the routing key in the message, and the routing criteria used when a queue is bound to this exchange. * Topic Wildcard match between the routing key and the routing pattern specified in the binding. The routing key is treated as zero or more words delimited by ``"."`` and supports special wildcard characters. ``"*"`` matches a single word and ``"#"`` matches zero or more words. * Fanout Queues are bound to this exchange with no arguments. Hence any message sent to this exchange will be forwarded to all queues bound to this exchange. * Headers Queues are bound to this exchange with a table of arguments containing headers and values (optional). A special argument named "x-match" determines the matching algorithm, where ``"all"`` implies an ``AND`` (all pairs must match) and ``"any"`` implies ``OR`` (at least one pair must match). Use the :attr:`routing_key`` is used to specify the arguments, the same when sending messages. This description of AMQP exchange types was shamelessly stolen from the blog post `AMQP in 10 minutes: Part 4`_ by Rajith Attapattu. Recommended reading. .. _`AMQP in 10 minutes: Part 4`: http://bit.ly/amqp-exchange-types .. attribute:: callbacks List of registered callbacks to trigger when a message is received by :meth:`wait`, :meth:`process_next` or :meth:`iterqueue`. .. attribute:: warn_if_exists Emit a warning if the queue has already been declared. If a queue already exists, and you try to redeclare the queue with new settings, the new settings will be silently ignored, so this can be useful if you've recently changed the :attr:`routing_key` attribute or other settings. .. attribute:: auto_ack Acknowledgement is handled automatically once messages are received. This means that the :meth:`carrot.backends.base.BaseMessage.ack` and :meth:`carrot.backends.base.BaseMessage.reject` methods on the message object are no longer valid. By default :attr:`auto_ack` is set to ``False``, and the receiver is required to manually handle acknowledgment. .. attribute:: no_ack Disable acknowledgement on the server-side. This is different from :attr:`auto_ack` in that acknowledgement is turned off altogether. This functionality increases performance but at the cost of reliability. Messages can get lost if a client dies before it can deliver them to the application. .. attribute auto_declare If this is ``True`` the following will be automatically declared: * The queue if :attr:`queue` is set. * The exchange if :attr:`exchange` is set. * The :attr:`queue` will be bound to the :attr:`exchange`. This is the default behaviour. :raises `amqplib.client_0_8.channel.AMQPChannelException`: if the queue is exclusive and the queue already exists and is owned by another connection. Example Usage >>> consumer = Consumer(connection=DjangoBrokerConnection(), ... queue="foo", exchange="foo", routing_key="foo") >>> def process_message(message_data, message): ... print("Got message %s: %s" % ( ... message.delivery_tag, message_data)) >>> consumer.register_callback(process_message) >>> consumer.wait() # Go into receive loop """ queue = "" exchange = "" routing_key = "" durable = True exclusive = False auto_delete = False exchange_type = "direct" channel_open = False warn_if_exists = False auto_declare = True auto_ack = False queue_arguments = None no_ack = False _closed = True _init_opts = ("durable", "exclusive", "auto_delete", "exchange_type", "warn_if_exists", "auto_ack", "auto_declare", "queue_arguments") _next_consumer_tag = count(1).next def __init__(self, connection, queue=None, exchange=None, routing_key=None, **kwargs): self.connection = connection self.backend = kwargs.get("backend", None) if not self.backend: self.backend = self.connection.create_backend() self.queue = queue or self.queue # Binding. self.queue = queue or self.queue self.exchange = exchange or self.exchange self.routing_key = routing_key or self.routing_key self.callbacks = [] # Options for opt_name in self._init_opts: opt_value = kwargs.get(opt_name) if opt_value is not None: setattr(self, opt_name, opt_value) # exclusive implies auto-delete. if self.exclusive: self.auto_delete = True self.consumer_tag = self._generate_consumer_tag() if self.auto_declare: self.declare() def __enter__(self): return self def __exit__(self, e_type, e_value, e_trace): if e_type: raise e_type(e_value) self.close() def __iter__(self): """iter(Consumer) -> Consumer.iterqueue(infinite=True)""" return self.iterqueue(infinite=True) def _generate_consumer_tag(self): """Generate a unique consumer tag. :rtype string: """ return "%s.%s%s" % ( self.__class__.__module__, self.__class__.__name__, self._next_consumer_tag()) def declare(self): """Declares the queue, the exchange and binds the queue to the exchange.""" arguments = None routing_key = self.routing_key if self.exchange_type == "headers": arguments, routing_key = routing_key, "" if self.queue: self.backend.queue_declare(queue=self.queue, durable=self.durable, exclusive=self.exclusive, auto_delete=self.auto_delete, arguments=self.queue_arguments, warn_if_exists=self.warn_if_exists) if self.exchange: self.backend.exchange_declare(exchange=self.exchange, type=self.exchange_type, durable=self.durable, auto_delete=self.auto_delete) if self.queue: self.backend.queue_bind(queue=self.queue, exchange=self.exchange, routing_key=routing_key, arguments=arguments) self._closed = False return self def _receive_callback(self, raw_message): """Internal method used when a message is received in consume mode.""" message = self.backend.message_to_python(raw_message) if self.auto_ack and not message.acknowledged: message.ack() self.receive(message.payload, message) def fetch(self, no_ack=None, auto_ack=None, enable_callbacks=False): """Receive the next message waiting on the queue. :returns: A :class:`carrot.backends.base.BaseMessage` instance, or ``None`` if there's no messages to be received. :keyword enable_callbacks: Enable callbacks. The message will be processed with all registered callbacks. Default is disabled. :keyword auto_ack: Override the default :attr:`auto_ack` setting. :keyword no_ack: Override the default :attr:`no_ack` setting. """ no_ack = no_ack or self.no_ack auto_ack = auto_ack or self.auto_ack message = self.backend.get(self.queue, no_ack=no_ack) if message: if auto_ack and not message.acknowledged: message.ack() if enable_callbacks: self.receive(message.payload, message) return message def process_next(self): """**DEPRECATED** Use :meth:`fetch` like this instead: >>> message = self.fetch(enable_callbacks=True) """ warnings.warn(DeprecationWarning( "Consumer.process_next has been deprecated in favor of \ Consumer.fetch(enable_callbacks=True)")) return self.fetch(enable_callbacks=True) def receive(self, message_data, message): """This method is called when a new message is received by running :meth:`wait`, :meth:`process_next` or :meth:`iterqueue`. When a message is received, it passes the message on to the callbacks listed in the :attr:`callbacks` attribute. You can register callbacks using :meth:`register_callback`. :param message_data: The deserialized message data. :param message: The :class:`carrot.backends.base.BaseMessage` instance. :raises NotImplementedError: If no callbacks has been registered. """ if not self.callbacks: raise NotImplementedError("No consumer callbacks registered") for callback in self.callbacks: callback(message_data, message) def register_callback(self, callback): """Register a callback function to be triggered by :meth:`receive`. The ``callback`` function must take two arguments: * message_data The deserialized message data * message The :class:`carrot.backends.base.BaseMessage` instance. """ self.callbacks.append(callback) def discard_all(self, filterfunc=None): """Discard all waiting messages. :param filterfunc: A filter function to only discard the messages this filter returns. :returns: the number of messages discarded. *WARNING*: All incoming messages will be ignored and not processed. Example using filter: >>> def waiting_feeds_only(message): ... try: ... message_data = message.decode() ... except: # Should probably be more specific. ... pass ... ... if message_data.get("type") == "feed": ... return True ... else: ... return False """ if not filterfunc: return self.backend.queue_purge(self.queue) if self.no_ack or self.auto_ack: raise Exception("discard_all: Can't use filter with auto/no-ack.") discarded_count = 0 while True: message = self.fetch() if message is None: return discarded_count if filterfunc(message): message.ack() discarded_count += 1 def iterconsume(self, limit=None, no_ack=None): """Iterator processing new messages as they arrive. Every new message will be passed to the callbacks, and the iterator returns ``True``. The iterator is infinite unless the ``limit`` argument is specified or someone closes the consumer. :meth:`iterconsume` uses transient requests for messages on the server, while :meth:`iterequeue` uses synchronous access. In most cases you want :meth:`iterconsume`, but if your environment does not support this behaviour you can resort to using :meth:`iterqueue` instead. Also, :meth:`iterconsume` does not return the message at each step, something which :meth:`iterqueue` does. :keyword limit: Maximum number of messages to process. :raises StopIteration: if limit is set and the message limit has been reached. """ self.consume(no_ack=no_ack) return self.backend.consume(limit=limit) def consume(self, no_ack=None): """Declare consumer.""" no_ack = no_ack or self.no_ack self.backend.declare_consumer(queue=self.queue, no_ack=no_ack, callback=self._receive_callback, consumer_tag=self.consumer_tag, nowait=True) self.channel_open = True def wait(self, limit=None): """Go into consume mode. Mostly for testing purposes and simple programs, you probably want :meth:`iterconsume` or :meth:`iterqueue` instead. This runs an infinite loop, processing all incoming messages using :meth:`receive` to apply the message to all registered callbacks. """ it = self.iterconsume(limit) while True: it.next() def iterqueue(self, limit=None, infinite=False): """Infinite iterator yielding pending messages, by using synchronous direct access to the queue (``basic_get``). :meth:`iterqueue` is used where synchronous functionality is more important than performance. If you can, use :meth:`iterconsume` instead. :keyword limit: If set, the iterator stops when it has processed this number of messages in total. :keyword infinite: Don't raise :exc:`StopIteration` if there is no messages waiting, but return ``None`` instead. If infinite you obviously shouldn't consume the whole iterator at once without using a ``limit``. :raises StopIteration: If there is no messages waiting, and the iterator is not infinite. """ for items_since_start in count(): item = self.fetch() if (not infinite and item is None) or \ (limit and items_since_start >= limit): raise StopIteration yield item def cancel(self): """Cancel a running :meth:`iterconsume` session.""" if self.channel_open: try: self.backend.cancel(self.consumer_tag) except KeyError: pass def close(self): """Close the channel to the queue.""" self.cancel() self.backend.close() self._closed = True def flow(self, active): """This method asks the peer to pause or restart the flow of content data. This is a simple flow-control mechanism that a peer can use to avoid oveflowing its queues or otherwise finding itself receiving more messages than it can process. Note that this method is not intended for window control. The peer that receives a request to stop sending content should finish sending the current content, if any, and then wait until it receives the ``flow(active=True)`` restart method. """ self.backend.flow(active) def qos(self, prefetch_size=0, prefetch_count=0, apply_global=False): """Request specific Quality of Service. This method requests a specific quality of service. The QoS can be specified for the current channel or for all channels on the connection. The particular properties and semantics of a qos method always depend on the content class semantics. Though the qos method could in principle apply to both peers, it is currently meaningful only for the server. :param prefetch_size: Prefetch window in octets. The client can request that messages be sent in advance so that when the client finishes processing a message, the following message is already held locally, rather than needing to be sent down the channel. Prefetching gives a performance improvement. This field specifies the prefetch window size in octets. The server will send a message in advance if it is equal to or smaller in size than the available prefetch size (and also falls into other prefetch limits). May be set to zero, meaning "no specific limit", although other prefetch limits may still apply. The ``prefetch_size`` is ignored if the :attr:`no_ack` option is set. :param prefetch_count: Specifies a prefetch window in terms of whole messages. This field may be used in combination with ``prefetch_size``; A message will only be sent in advance if both prefetch windows (and those at the channel and connection level) allow it. The prefetch- count is ignored if the :attr:`no_ack` option is set. :keyword apply_global: By default the QoS settings apply to the current channel only. If this is set, they are applied to the entire connection. """ return self.backend.qos(prefetch_size, prefetch_count, apply_global) class Publisher(object): """Message publisher. :param connection: see :attr:`connection`. :param exchange: see :attr:`exchange`. :param routing_key: see :attr:`routing_key`. :keyword exchange_type: see :attr:`Consumer.exchange_type`. :keyword durable: see :attr:`Consumer.durable`. :keyword auto_delete: see :attr:`Consumer.auto_delete`. :keyword serializer: see :attr:`serializer`. :keyword auto_declare: See :attr:`auto_declare`. .. attribute:: connection The connection to the broker. A :class:`carrot.connection.BrokerConnection` instance. .. attribute:: exchange Name of the exchange we send messages to. .. attribute:: routing_key The default routing key for messages sent using this publisher. See :attr:`Consumer.routing_key` for more information. You can override the routing key by passing an explicit ``routing_key`` argument to :meth:`send`. .. attribute:: delivery_mode The default delivery mode used for messages. The value is an integer. The following delivery modes are supported by (at least) RabbitMQ: * 1 or "transient" The message is transient. Which means it is stored in memory only, and is lost if the server dies or restarts. * 2 or "persistent" The message is persistent. Which means the message is stored both in-memory, and on disk, and therefore preserved if the server dies or restarts. The default value is ``2`` (persistent). .. attribute:: exchange_type See :attr:`Consumer.exchange_type`. .. attribute:: durable See :attr:`Consumer.durable`. .. attribute:: auto_delete See :attr:`Consumer.auto_delete`. .. attribute:: auto_declare If this is ``True`` and the :attr:`exchange` name is set, the exchange will be automatically declared at instantiation. You can manually the declare the exchange by using the :meth:`declare` method. Auto declare is on by default. .. attribute:: serializer A string identifying the default serialization method to use. Defaults to ``json``. Can be ``json`` (default), ``raw``, ``pickle``, ``hessian``, ``yaml``, or any custom serialization methods that have been registered with :mod:`carrot.serialization.registry`. """ NONE_PERSISTENT_DELIVERY_MODE = 1 TRANSIENT_DELIVERY_MODE = 1 PERSISTENT_DELIVERY_MODE = 2 DELIVERY_MODES = { "transient": TRANSIENT_DELIVERY_MODE, "persistent": PERSISTENT_DELIVERY_MODE, "non-persistent": TRANSIENT_DELIVERY_MODE, } exchange = "" routing_key = "" delivery_mode = PERSISTENT_DELIVERY_MODE _closed = True exchange_type = "direct" durable = True auto_delete = False auto_declare = True serializer = None _init_opts = ("exchange_type", "durable", "auto_delete", "serializer", "delivery_mode", "auto_declare") def __init__(self, connection, exchange=None, routing_key=None, **kwargs): self.connection = connection self.backend = self.connection.create_backend() self.exchange = exchange or self.exchange self.routing_key = routing_key or self.routing_key for opt_name in self._init_opts: opt_value = kwargs.get(opt_name) if opt_value is not None: setattr(self, opt_name, opt_value) self.delivery_mode = self.DELIVERY_MODES.get(self.delivery_mode, self.delivery_mode) self._closed = False if self.auto_declare and self.exchange: self.declare() def declare(self): """Declare the exchange. Creates the exchange on the broker. """ self.backend.exchange_declare(exchange=self.exchange, type=self.exchange_type, durable=self.durable, auto_delete=self.auto_delete) def __enter__(self): return self def __exit__(self, e_type, e_value, e_trace): self.close() def create_message(self, message_data, delivery_mode=None, priority=None, content_type=None, content_encoding=None, serializer=None): """With any data, serialize it and encapsulate it in a AMQP message with the proper headers set.""" delivery_mode = delivery_mode or self.delivery_mode # No content_type? Then we're serializing the data internally. if not content_type: serializer = serializer or self.serializer (content_type, content_encoding, message_data) = serialization.encode(message_data, serializer=serializer) else: # If the programmer doesn't want us to serialize, # make sure content_encoding is set. if isinstance(message_data, unicode): if not content_encoding: content_encoding = 'utf-8' message_data = message_data.encode(content_encoding) # If they passed in a string, we can't know anything # about it. So assume it's binary data. elif not content_encoding: content_encoding = 'binary' return self.backend.prepare_message(message_data, delivery_mode, priority=priority, content_type=content_type, content_encoding=content_encoding) def send(self, message_data, routing_key=None, delivery_mode=None, mandatory=False, immediate=False, priority=0, content_type=None, content_encoding=None, serializer=None, exchange=None): """Send a message. :param message_data: The message data to send. Can be a list, dictionary or a string. :keyword routing_key: A custom routing key for the message. If not set, the default routing key set in the :attr:`routing_key` attribute is used. :keyword mandatory: If set, the message has mandatory routing. By default the message is silently dropped by the server if it can't be routed to a queue. However - If the message is mandatory, an exception will be raised instead. :keyword immediate: Request immediate delivery. If the message cannot be routed to a queue consumer immediately, an exception will be raised. This is instead of the default behaviour, where the server will accept and queue the message, but with no guarantee that the message will ever be consumed. :keyword delivery_mode: Override the default :attr:`delivery_mode`. :keyword priority: The message priority, ``0`` to ``9``. :keyword content_type: The messages content_type. If content_type is set, no serialization occurs as it is assumed this is either a binary object, or you've done your own serialization. Leave blank if using built-in serialization as our library properly sets content_type. :keyword content_encoding: The character set in which this object is encoded. Use "binary" if sending in raw binary objects. Leave blank if using built-in serialization as our library properly sets content_encoding. :keyword serializer: Override the default :attr:`serializer`. :keyword exchange: Override the exchange to publish to. Note that this exchange must have been declared. """ headers = None routing_key = routing_key or self.routing_key if self.exchange_type == "headers": headers, routing_key = routing_key, "" exchange = exchange or self.exchange message = self.create_message(message_data, priority=priority, delivery_mode=delivery_mode, content_type=content_type, content_encoding=content_encoding, serializer=serializer) self.backend.publish(message, exchange=exchange, routing_key=routing_key, mandatory=mandatory, immediate=immediate, headers=headers) def close(self): """Close connection to queue.""" self.backend.close() self._closed = True class Messaging(object): """A combined message publisher and consumer.""" queue = "" exchange = "" routing_key = "" publisher_cls = Publisher consumer_cls = Consumer _closed = True def __init__(self, connection, **kwargs): self.connection = connection self.exchange = kwargs.get("exchange", self.exchange) self.queue = kwargs.get("queue", self.queue) self.routing_key = kwargs.get("routing_key", self.routing_key) self.publisher = self.publisher_cls(connection, exchange=self.exchange, routing_key=self.routing_key) self.consumer = self.consumer_cls(connection, queue=self.queue, exchange=self.exchange, routing_key=self.routing_key) self.consumer.register_callback(self.receive) self.callbacks = [] self._closed = False def __enter__(self): return self def __exit__(self, e_type, e_value, e_trace): if e_type: raise e_type(e_value) self.close() def register_callback(self, callback): """See :meth:`Consumer.register_callback`""" self.callbacks.append(callback) def receive(self, message_data, message): """See :meth:`Consumer.receive`""" if not self.callbacks: raise NotImplementedError("No consumer callbacks registered") for callback in self.callbacks: callback(message_data, message) def send(self, message_data, delivery_mode=None): """See :meth:`Publisher.send`""" self.publisher.send(message_data, delivery_mode=delivery_mode) def fetch(self, **kwargs): """See :meth:`Consumer.fetch`""" return self.consumer.fetch(**kwargs) def close(self): """Close any open channels.""" self.consumer.close() self.publisher.close() self._closed = True class ConsumerSet(object): """Receive messages from multiple consumers. :param connection: see :attr:`connection`. :param from_dict: see :attr:`from_dict`. :param consumers: see :attr:`consumers`. :param callbacks: see :attr:`callbacks`. .. attribute:: connection The connection to the broker. A :class:`carrot.connection.BrokerConnection` instance. .. attribute:: callbacks A list of callbacks to be called when a message is received. See :class:`Consumer.register_callback`. .. attribute:: from_dict Add consumers from a dictionary configuration:: { "webshot": { "exchange": "link_exchange", "exchange_type": "topic", "binding_key": "links.webshot", "default_routing_key": "links.webshot", }, "retrieve": { "exchange": "link_exchange", "exchange_type" = "topic", "binding_key": "links.*", "default_routing_key": "links.retrieve", "auto_delete": True, # ... }, } .. attribute:: consumers Add consumers from a list of :class:`Consumer` instances. .. attribute:: auto_ack Default value for the :attr:`Consumer.auto_ack` attribute. """ auto_ack = False def __init__(self, connection, from_dict=None, consumers=None, callbacks=None, **options): self.connection = connection self.options = options self.from_dict = from_dict or {} self.consumers = [] self.callbacks = callbacks or [] self._open_consumers = {} self.backend = self.connection.create_backend() self.auto_ack = options.get("auto_ack", self.auto_ack) if consumers: [self.add_consumer(consumer) for consumer in consumers] [self.add_consumer_from_dict(queue_name, **queue_options) for queue_name, queue_options in self.from_dict.items()] def _receive_callback(self, raw_message): """Internal method used when a message is received in consume mode.""" message = self.backend.message_to_python(raw_message) if self.auto_ack and not message.acknowledged: message.ack() self.receive(message.decode(), message) def add_consumer_from_dict(self, queue, **options): """Add another consumer from dictionary configuration.""" consumer = Consumer(self.connection, queue=queue, backend=self.backend, **options) self.consumers.append(consumer) return consumer def add_consumer(self, consumer): """Add another consumer from a :class:`Consumer` instance.""" consumer.backend = self.backend self.consumers.append(consumer) def register_callback(self, callback): """Register new callback to be called when a message is received. See :meth:`Consumer.register_callback`""" self.callbacks.append(callback) def receive(self, message_data, message): """What to do when a message is received. See :meth:`Consumer.receive`.""" if not self.callbacks: raise NotImplementedError("No consumer callbacks registered") for callback in self.callbacks: callback(message_data, message) def _declare_consumer(self, consumer, nowait=False): """Declare consumer so messages can be received from it using :meth:`iterconsume`.""" if consumer.queue not in self._open_consumers: # Use the ConsumerSet's consumer by default, but if the # child consumer has a callback, honor it. callback = consumer.callbacks and \ consumer._receive_callback or self._receive_callback self.backend.declare_consumer(queue=consumer.queue, no_ack=consumer.no_ack, nowait=nowait, callback=callback, consumer_tag=consumer.consumer_tag) self._open_consumers[consumer.queue] = consumer.consumer_tag def consume(self): """Declare consumers.""" head = self.consumers[:-1] tail = self.consumers[-1] [self._declare_consumer(consumer, nowait=True) for consumer in head] self._declare_consumer(tail, nowait=False) def iterconsume(self, limit=None): """Cycle between all consumers in consume mode. See :meth:`Consumer.iterconsume`. """ self.consume() return self.backend.consume(limit=limit) def discard_all(self): """Discard all messages. Does not support filtering. See :meth:`Consumer.discard_all`.""" return sum([consumer.discard_all() for consumer in self.consumers]) def flow(self, active): """This method asks the peer to pause or restart the flow of content data. See :meth:`Consumer.flow`. """ self.backend.flow(active) def qos(self, prefetch_size=0, prefetch_count=0, apply_global=False): """Request specific Quality of Service. See :meth:`Consumer.cos`. """ self.backend.qos(prefetch_size, prefetch_count, apply_global) def cancel(self): """Cancel a running :meth:`iterconsume` session.""" for consumer_tag in self._open_consumers.values(): try: self.backend.cancel(consumer_tag) except KeyError: pass self._open_consumers.clear() def cancel_by_queue(self, queue): consumer_tag = self._open_consumers.pop(queue) self.backend.cancel(consumer_tag) def close(self): """Close all consumers.""" self.cancel() for consumer in self.consumers: consumer.close() carrot-0.10.7/carrot/serialization.py000644 000765 000024 00000022724 11401660116 017536 0ustar00askstaff000000 000000 """ Centralized support for encoding/decoding of data structures. Requires a json library (`cjson`_, `simplejson`_, or `Python 2.6+`_). Pickle support is built-in. Optionally installs support for ``YAML`` if the `PyYAML`_ package is installed. Optionally installs support for `msgpack`_ if the `msgpack-python`_ package is installed. .. _`cjson`: http://pypi.python.org/pypi/python-cjson/ .. _`simplejson`: http://code.google.com/p/simplejson/ .. _`Python 2.6+`: http://docs.python.org/library/json.html .. _`PyYAML`: http://pyyaml.org/ .. _`msgpack`: http://msgpack.sourceforge.net/ .. _`msgpack-python`: http://pypi.python.org/pypi/msgpack-python/ """ import codecs __all__ = ['SerializerNotInstalled', 'registry'] class SerializerNotInstalled(StandardError): """Support for the requested serialization type is not installed""" class SerializerRegistry(object): """The registry keeps track of serialization methods.""" def __init__(self): self._encoders = {} self._decoders = {} self._default_encode = None self._default_content_type = None self._default_content_encoding = None def register(self, name, encoder, decoder, content_type, content_encoding='utf-8'): """Register a new encoder/decoder. :param name: A convenience name for the serialization method. :param encoder: A method that will be passed a python data structure and should return a string representing the serialized data. If ``None``, then only a decoder will be registered. Encoding will not be possible. :param decoder: A method that will be passed a string representing serialized data and should return a python data structure. If ``None``, then only an encoder will be registered. Decoding will not be possible. :param content_type: The mime-type describing the serialized structure. :param content_encoding: The content encoding (character set) that the :param:`decoder` method will be returning. Will usually be ``utf-8``, ``us-ascii``, or ``binary``. """ if encoder: self._encoders[name] = (content_type, content_encoding, encoder) if decoder: self._decoders[content_type] = decoder def _set_default_serializer(self, name): """ Set the default serialization method used by this library. :param name: The name of the registered serialization method. For example, ``json`` (default), ``pickle``, ``yaml``, or any custom methods registered using :meth:`register`. :raises SerializerNotInstalled: If the serialization method requested is not available. """ try: (self._default_content_type, self._default_content_encoding, self._default_encode) = self._encoders[name] except KeyError: raise SerializerNotInstalled( "No encoder installed for %s" % name) def encode(self, data, serializer=None): """ Serialize a data structure into a string suitable for sending as an AMQP message body. :param data: The message data to send. Can be a list, dictionary or a string. :keyword serializer: An optional string representing the serialization method you want the data marshalled into. (For example, ``json``, ``raw``, or ``pickle``). If ``None`` (default), then `JSON`_ will be used, unless ``data`` is a ``str`` or ``unicode`` object. In this latter case, no serialization occurs as it would be unnecessary. Note that if ``serializer`` is specified, then that serialization method will be used even if a ``str`` or ``unicode`` object is passed in. :returns: A three-item tuple containing the content type (e.g., ``application/json``), content encoding, (e.g., ``utf-8``) and a string containing the serialized data. :raises SerializerNotInstalled: If the serialization method requested is not available. """ if serializer == "raw": return raw_encode(data) if serializer and not self._encoders.get(serializer): raise SerializerNotInstalled( "No encoder installed for %s" % serializer) # If a raw string was sent, assume binary encoding # (it's likely either ASCII or a raw binary file, but 'binary' # charset will encompass both, even if not ideal. if not serializer and isinstance(data, str): # In Python 3+, this would be "bytes"; allow binary data to be # sent as a message without getting encoder errors return "application/data", "binary", data # For unicode objects, force it into a string if not serializer and isinstance(data, unicode): payload = data.encode("utf-8") return "text/plain", "utf-8", payload if serializer: content_type, content_encoding, encoder = \ self._encoders[serializer] else: encoder = self._default_encode content_type = self._default_content_type content_encoding = self._default_content_encoding payload = encoder(data) return content_type, content_encoding, payload def decode(self, data, content_type, content_encoding): """Deserialize a data stream as serialized using ``encode`` based on :param:`content_type`. :param data: The message data to deserialize. :param content_type: The content-type of the data. (e.g., ``application/json``). :param content_encoding: The content-encoding of the data. (e.g., ``utf-8``, ``binary``, or ``us-ascii``). :returns: The unserialized data. """ content_type = content_type or 'application/data' content_encoding = (content_encoding or 'utf-8').lower() # Don't decode 8-bit strings or unicode objects if content_encoding not in ('binary', 'ascii-8bit') and \ not isinstance(data, unicode): data = codecs.decode(data, content_encoding) try: decoder = self._decoders[content_type] except KeyError: return data return decoder(data) """ .. data:: registry Global registry of serializers/deserializers. """ registry = SerializerRegistry() """ .. function:: encode(data, serializer=default_serializer) Encode data using the registry's default encoder. """ encode = registry.encode """ .. function:: decode(data, content_type, content_encoding): Decode data using the registry's default decoder. """ decode = registry.decode def raw_encode(data): """Special case serializer.""" content_type = 'application/data' payload = data if isinstance(payload, unicode): content_encoding = 'utf-8' payload = payload.encode(content_encoding) else: content_encoding = 'binary' return content_type, content_encoding, payload def register_json(): """Register a encoder/decoder for JSON serialization.""" from anyjson import serialize as json_serialize from anyjson import deserialize as json_deserialize registry.register('json', json_serialize, json_deserialize, content_type='application/json', content_encoding='utf-8') def register_yaml(): """Register a encoder/decoder for YAML serialization. It is slower than JSON, but allows for more data types to be serialized. Useful if you need to send data such as dates""" try: import yaml registry.register('yaml', yaml.safe_dump, yaml.safe_load, content_type='application/x-yaml', content_encoding='utf-8') except ImportError: def not_available(*args, **kwargs): """In case a client receives a yaml message, but yaml isn't installed.""" raise SerializerNotInstalled( "No decoder installed for YAML. Install the PyYAML library") registry.register('yaml', None, not_available, 'application/x-yaml') def register_pickle(): """The fastest serialization method, but restricts you to python clients.""" import cPickle registry.register('pickle', cPickle.dumps, cPickle.loads, content_type='application/x-python-serialize', content_encoding='binary') def register_msgpack(): """See http://msgpack.sourceforge.net/""" try: import msgpack registry.register('msgpack', msgpack.packs, msgpack.unpacks, content_type='application/x-msgpack', content_encoding='utf-8') except ImportError: def not_available(*args, **kwargs): """In case a client receives a msgpack message, but yaml isn't installed.""" raise SerializerNotInstalled( "No decoder installed for msgpack. " "Install the msgpack library") registry.register('msgpack', None, not_available, 'application/x-msgpack') # Register the base serialization methods. register_json() register_pickle() register_yaml() register_msgpack() # JSON is assumed to always be available, so is the default. # (this matches the historical use of carrot.) registry._set_default_serializer('json') carrot-0.10.7/carrot/utils.py000644 000765 000024 00000003435 11401714645 016026 0ustar00askstaff000000 000000 from uuid import UUID, uuid4, _uuid_generate_random try: import ctypes except ImportError: ctypes = None def gen_unique_id(): """Generate a unique id, having - hopefully - a very small chance of collission. For now this is provided by :func:`uuid.uuid4`. """ # Workaround for http://bugs.python.org/issue4607 if ctypes and _uuid_generate_random: buffer = ctypes.create_string_buffer(16) _uuid_generate_random(buffer) return str(UUID(bytes=buffer.raw)) return str(uuid4()) def _compat_rl_partition(S, sep, direction=None): if direction is None: direction = S.split items = direction(sep, 1) if len(items) == 1: return items[0], sep, '' return items[0], sep, items[1] def _compat_partition(S, sep): """``partition(S, sep) -> (head, sep, tail)`` Search for the separator ``sep`` in ``S``, and return the part before it, the separator itself, and the part after it. If the separator is not found, return ``S`` and two empty strings. """ return _compat_rl_partition(S, sep, direction=S.split) def _compat_rpartition(S, sep): """``rpartition(S, sep) -> (tail, sep, head)`` Search for the separator ``sep`` in ``S``, starting at the end of ``S``, and return the part before it, the separator itself, and the part after it. If the separator is not found, return two empty strings and ``S``. """ return _compat_rl_partition(S, sep, direction=S.rsplit) def partition(S, sep): if hasattr(S, 'partition'): return S.partition(sep) else: # Python <= 2.4: return _compat_partition(S, sep) def rpartition(S, sep): if hasattr(S, 'rpartition'): return S.rpartition(sep) else: # Python <= 2.4: return _compat_rpartition(S, sep) carrot-0.10.7/carrot/backends/__init__.py000644 000765 000024 00000003043 11401660116 020163 0ustar00askstaff000000 000000 """ Working with Backends. """ import sys from carrot.utils import rpartition DEFAULT_BACKEND = "carrot.backends.pyamqplib.Backend" BACKEND_ALIASES = { "amqp": "carrot.backends.pyamqplib.Backend", "amqplib": "carrot.backends.pyamqplib.Backend", "stomp": "carrot.backends.pystomp.Backend", "stompy": "carrot.backends.pystomp.Backend", "memory": "carrot.backends.queue.Backend", "mem": "carrot.backends.queue.Backend", "pika": "carrot.backends.pikachu.AsyncoreBackend", "pikachu": "carrot.backends.pikachu.AsyncoreBackend", "syncpika": "carrot.backends.pikachu.SyncBackend", } _backend_cache = {} def resolve_backend(backend=None): backend = BACKEND_ALIASES.get(backend, backend) backend_module_name, _, backend_cls_name = rpartition(backend, ".") return backend_module_name, backend_cls_name def _get_backend_cls(backend=None): backend_module_name, backend_cls_name = resolve_backend(backend) __import__(backend_module_name) backend_module = sys.modules[backend_module_name] return getattr(backend_module, backend_cls_name) def get_backend_cls(backend=None): """Get backend class by name. The backend string is the full path to a backend class, e.g.:: "carrot.backends.pyamqplib.Backend" If the name does not include "``.``" (is not fully qualified), the alias table will be consulted. """ backend = backend or DEFAULT_BACKEND if backend not in _backend_cache: _backend_cache[backend] = _get_backend_cls(backend) return _backend_cache[backend] carrot-0.10.7/carrot/backends/base.py000644 000765 000024 00000012142 11404434721 017342 0ustar00askstaff000000 000000 """ Backend base classes. """ from carrot import serialization ACKNOWLEDGED_STATES = frozenset(["ACK", "REJECTED", "REQUEUED"]) class MessageStateError(Exception): """The message has already been acknowledged.""" class BaseMessage(object): """Base class for received messages.""" _state = None MessageStateError = MessageStateError def __init__(self, backend, **kwargs): self.backend = backend self.body = kwargs.get("body") self.delivery_tag = kwargs.get("delivery_tag") self.content_type = kwargs.get("content_type") self.content_encoding = kwargs.get("content_encoding") self.delivery_info = kwargs.get("delivery_info", {}) self._decoded_cache = None self._state = "RECEIVED" def decode(self): """Deserialize the message body, returning the original python structure sent by the publisher.""" return serialization.decode(self.body, self.content_type, self.content_encoding) @property def payload(self): """The decoded message.""" if not self._decoded_cache: self._decoded_cache = self.decode() return self._decoded_cache def ack(self): """Acknowledge this message as being processed., This will remove the message from the queue. :raises MessageStateError: If the message has already been acknowledged/requeued/rejected. """ if self.acknowledged: raise self.MessageStateError( "Message already acknowledged with state: %s" % self._state) self.backend.ack(self.delivery_tag) self._state = "ACK" def reject(self): """Reject this message. The message will be discarded by the server. :raises MessageStateError: If the message has already been acknowledged/requeued/rejected. """ if self.acknowledged: raise self.MessageStateError( "Message already acknowledged with state: %s" % self._state) self.backend.reject(self.delivery_tag) self._state = "REJECTED" def requeue(self): """Reject this message and put it back on the queue. You must not use this method as a means of selecting messages to process. :raises MessageStateError: If the message has already been acknowledged/requeued/rejected. """ if self.acknowledged: raise self.MessageStateError( "Message already acknowledged with state: %s" % self._state) self.backend.requeue(self.delivery_tag) self._state = "REQUEUED" @property def acknowledged(self): return self._state in ACKNOWLEDGED_STATES class BaseBackend(object): """Base class for backends.""" default_port = None extra_options = None def __init__(self, connection, **kwargs): self.connection = connection self.extra_options = kwargs.get("extra_options") def queue_declare(self, *args, **kwargs): """Declare a queue by name.""" pass def queue_delete(self, *args, **kwargs): """Delete a queue by name.""" pass def exchange_declare(self, *args, **kwargs): """Declare an exchange by name.""" pass def queue_bind(self, *args, **kwargs): """Bind a queue to an exchange.""" pass def get(self, *args, **kwargs): """Pop a message off the queue.""" pass def declare_consumer(self, *args, **kwargs): pass def consume(self, *args, **kwargs): """Iterate over the declared consumers.""" pass def cancel(self, *args, **kwargs): """Cancel the consumer.""" pass def ack(self, delivery_tag): """Acknowledge the message.""" pass def queue_purge(self, queue, **kwargs): """Discard all messages in the queue. This will delete the messages and results in an empty queue.""" return 0 def reject(self, delivery_tag): """Reject the message.""" pass def requeue(self, delivery_tag): """Requeue the message.""" pass def purge(self, queue, **kwargs): """Discard all messages in the queue.""" pass def message_to_python(self, raw_message): """Convert received message body to a python datastructure.""" return raw_message def prepare_message(self, message_data, delivery_mode, **kwargs): """Prepare message for sending.""" return message_data def publish(self, message, exchange, routing_key, **kwargs): """Publish a message.""" pass def close(self): """Close the backend.""" pass def establish_connection(self): """Establish a connection to the backend.""" pass def close_connection(self, connection): """Close the connection.""" pass def flow(self, active): """Enable/disable flow from peer.""" pass def qos(self, prefetch_size, prefetch_count, apply_global=False): """Request specific Quality of Service.""" pass carrot-0.10.7/carrot/backends/librabbitmq.py000644 000765 000024 00000020531 11405425416 020723 0ustar00askstaff000000 000000 """ `amqplib`_ backend for carrot. .. _`amqplib`: http://barryp.org/software/py-amqplib/ """ import pylibrabbitmq as amqp from pylibrabbitmq import ChannelError, ConnectionError from carrot.backends.base import BaseMessage, BaseBackend from itertools import count import warnings import weakref DEFAULT_PORT = 5672 class Message(BaseMessage): """A message received by the broker. Usually you don't insantiate message objects yourself, but receive them using a :class:`carrot.messaging.Consumer`. :param backend: see :attr:`backend`. :param amqp_message: see :attr:`_amqp_message`. .. attribute:: body The message body. .. attribute:: delivery_tag The message delivery tag, uniquely identifying this message. .. attribute:: backend The message backend used. A subclass of :class:`carrot.backends.base.BaseBackend`. .. attribute:: _amqp_message A :class:`amqplib.client_0_8.basic_message.Message` instance. This is a private attribute and should not be accessed by production code. """ def __init__(self, backend, amqp_message, **kwargs): self._amqp_message = amqp_message self.backend = backend kwargs["body"] = amqp_message.body properties = amqp_message.properties kwargs["content_type"] = properties["content_type"] kwargs["content_encoding"] = properties["content_encoding"] kwargs["delivery_info"] = amqp_message.delivery_info kwargs["delivery_tag"] = amqp_message.delivery_info["delivery_tag"] super(Message, self).__init__(backend, **kwargs) class Backend(BaseBackend): """amqplib backend :param connection: see :attr:`connection`. .. attribute:: connection A :class:`carrot.connection.BrokerConnection` instance. An established connection to the broker. """ default_port = DEFAULT_PORT Message = Message def __init__(self, connection, **kwargs): self.connection = connection self.default_port = kwargs.get("default_port", self.default_port) self._channel_ref = None @property def _channel(self): return callable(self._channel_ref) and self._channel_ref() @property def channel(self): """If no channel exists, a new one is requested.""" if not self._channel: self._channel_ref = weakref.ref(self.connection.get_channel()) return self._channel def establish_connection(self): """Establish connection to the AMQP broker.""" conninfo = self.connection if not conninfo.hostname: raise KeyError("Missing hostname for AMQP connection.") if conninfo.userid is None: raise KeyError("Missing user id for AMQP connection.") if conninfo.password is None: raise KeyError("Missing password for AMQP connection.") if not conninfo.port: conninfo.port = self.default_port conn = amqp.Connection(host=conninfo.hostname, port=conninfo.port, userid=conninfo.userid, password=conninfo.password, virtual_host=conninfo.virtual_host) return conn def close_connection(self, connection): """Close the AMQP broker connection.""" connection.close() def queue_exists(self, queue): return True def queue_delete(self, queue, if_unused=False, if_empty=False): """Delete queue by name.""" pass def queue_purge(self, queue, **kwargs): """Discard all messages in the queue. This will delete the messages and results in an empty queue.""" return self.channel.queue_purge(queue=queue) def queue_declare(self, queue, durable, exclusive, auto_delete, warn_if_exists=False): """Declare a named queue.""" return self.channel.queue_declare(queue=queue, durable=durable, exclusive=exclusive, auto_delete=auto_delete) def exchange_declare(self, exchange, type, durable, auto_delete): """Declare an named exchange.""" return self.channel.exchange_declare(exchange=exchange, type=type, durable=durable, auto_delete=auto_delete) def queue_bind(self, queue, exchange, routing_key, arguments=None): """Bind queue to an exchange using a routing key.""" return self.channel.queue_bind(queue=queue, exchange=exchange, routing_key=routing_key, arguments=arguments) def message_to_python(self, raw_message): """Convert encoded message body back to a Python value.""" return self.Message(backend=self, amqp_message=raw_message) def get(self, queue, no_ack=False): """Receive a message from a declared queue by name. :returns: A :class:`Message` object if a message was received, ``None`` otherwise. If ``None`` was returned, it probably means there was no messages waiting on the queue. """ raw_message = self.channel.basic_get(queue, no_ack=no_ack) if not raw_message: return None return self.message_to_python(raw_message) def declare_consumer(self, queue, no_ack, callback, consumer_tag, nowait=False): """Declare a consumer.""" return self.channel.basic_consume(queue=queue, no_ack=no_ack, callback=callback, consumer_tag=consumer_tag) def consume(self, limit=None): """Returns an iterator that waits for one message at a time.""" for total_message_count in count(): if limit and total_message_count >= limit: raise StopIteration if not self.channel.is_open: raise StopIteration self.channel.conn.drain_events() yield True def cancel(self, consumer_tag): """Cancel a channel by consumer tag.""" if not self.channel.conn: return self.channel.basic_cancel(consumer_tag) def close(self): """Close the channel if open.""" if self._channel and self._channel.is_open: self._channel.close() self._channel_ref = None def ack(self, delivery_tag): """Acknowledge a message by delivery tag.""" return self.channel.basic_ack(delivery_tag) def reject(self, delivery_tag): """Reject a message by deliver tag.""" return self.channel.basic_reject(delivery_tag, requeue=False) def requeue(self, delivery_tag): """Reject and requeue a message by delivery tag.""" return self.channel.basic_reject(delivery_tag, requeue=True) def prepare_message(self, message_data, delivery_mode, priority=None, content_type=None, content_encoding=None): """Encapsulate data into a AMQP message.""" return amqp.Message(message_data, properties={ "delivery_mode": delivery_mode, "priority": priority, "content_type": content_type, "content_encoding": content_encoding}) def publish(self, message, exchange, routing_key, mandatory=None, immediate=None, headers=None): """Publish a message to a named exchange.""" if headers: message.properties["headers"] = headers ret = self.channel.basic_publish(message, exchange=exchange, routing_key=routing_key, mandatory=mandatory, immediate=immediate) if mandatory or immediate: self.close() def qos(self, prefetch_size, prefetch_count, apply_global=False): """Request specific Quality of Service.""" pass #self.channel.basic_qos(prefetch_size, prefetch_count, # apply_global) def flow(self, active): """Enable/disable flow from peer.""" pass #self.channel.flow(active) carrot-0.10.7/carrot/backends/pikachu.py000644 000765 000024 00000020027 11432724640 020060 0ustar00askstaff000000 000000 import asyncore import weakref import functools import itertools import pika from carrot.backends.base import BaseMessage, BaseBackend DEFAULT_PORT = 5672 class Message(BaseMessage): def __init__(self, backend, amqp_message, **kwargs): channel, method, header, body = amqp_message self._channel = channel self._method = method self._header = header self.backend = backend kwargs.update({"body": body, "delivery_tag": method.delivery_tag, "content_type": header.content_type, "content_encoding": header.content_encoding, "delivery_info": dict( consumer_tag=method.consumer_tag, routing_key=method.routing_key, delivery_tag=method.delivery_tag, exchange=method.exchange)}) super(Message, self).__init__(backend, **kwargs) class SyncBackend(BaseBackend): default_port = DEFAULT_PORT _connection_cls = pika.BlockingConnection Message = Message def __init__(self, connection, **kwargs): self.connection = connection self.default_port = kwargs.get("default_port", self.default_port) self._channel_ref = None @property def _channel(self): return callable(self._channel_ref) and self._channel_ref() @property def channel(self): """If no channel exists, a new one is requested.""" if not self._channel: self._channel_ref = weakref.ref(self.connection.get_channel()) return self._channel def establish_connection(self): """Establish connection to the AMQP broker.""" conninfo = self.connection if not conninfo.port: conninfo.port = self.default_port credentials = pika.PlainCredentials(conninfo.userid, conninfo.password) return self._connection_cls(pika.ConnectionParameters( conninfo.hostname, port=conninfo.port, virtual_host=conninfo.virtual_host, credentials=credentials)) def close_connection(self, connection): """Close the AMQP broker connection.""" connection.close() def queue_exists(self, queue): return False # FIXME def queue_delete(self, queue, if_unused=False, if_empty=False): """Delete queue by name.""" return self.channel.queue_delete(queue=queue, if_unused=if_unused, if_empty=if_empty) def queue_purge(self, queue, **kwargs): """Discard all messages in the queue. This will delete the messages and results in an empty queue.""" return self.channel.queue_purge(queue=queue).message_count def queue_declare(self, queue, durable, exclusive, auto_delete, warn_if_exists=False, arguments=None): """Declare a named queue.""" return self.channel.queue_declare(queue=queue, durable=durable, exclusive=exclusive, auto_delete=auto_delete, arguments=arguments) def exchange_declare(self, exchange, type, durable, auto_delete): """Declare an named exchange.""" return self.channel.exchange_declare(exchange=exchange, type=type, durable=durable, auto_delete=auto_delete) def queue_bind(self, queue, exchange, routing_key, arguments={}): """Bind queue to an exchange using a routing key.""" if not arguments: arguments = {} return self.channel.queue_bind(queue=queue, exchange=exchange, routing_key=routing_key, arguments=arguments) def message_to_python(self, raw_message): """Convert encoded message body back to a Python value.""" return self.Message(backend=self, amqp_message=raw_message) def get(self, queue, no_ack=False): """Receive a message from a declared queue by name. :returns: A :class:`Message` object if a message was received, ``None`` otherwise. If ``None`` was returned, it probably means there was no messages waiting on the queue. """ raw_message = self.channel.basic_get(queue=queue, no_ack=no_ack) if not raw_message: return None return self.message_to_python(raw_message) def declare_consumer(self, queue, no_ack, callback, consumer_tag, nowait=False): """Declare a consumer.""" @functools.wraps(callback) def _callback_decode(channel, method, header, body): return callback((channel, method, header, body)) return self.channel.basic_consume(_callback_decode, queue=queue, no_ack=no_ack, consumer_tag=consumer_tag) def consume(self, limit=None): """Returns an iterator that waits for one message at a time.""" for total_message_count in itertools.count(): if limit and total_message_count >= limit: raise StopIteration self.connection.connection.drain_events() yield True def cancel(self, consumer_tag): """Cancel a channel by consumer tag.""" if not self._channel: return self.channel.basic_cancel(consumer_tag) def close(self): """Close the channel if open.""" if self._channel and not self._channel.handler.channel_close: self._channel.close() self._channel_ref = None def ack(self, delivery_tag): """Acknowledge a message by delivery tag.""" return self.channel.basic_ack(delivery_tag) def reject(self, delivery_tag): """Reject a message by deliver tag.""" return self.channel.basic_reject(delivery_tag, requeue=False) def requeue(self, delivery_tag): """Reject and requeue a message by delivery tag.""" return self.channel.basic_reject(delivery_tag, requeue=True) def prepare_message(self, message_data, delivery_mode, priority=None, content_type=None, content_encoding=None): """Encapsulate data into a AMQP message.""" properties = pika.BasicProperties(priority=priority, content_type=content_type, content_encoding=content_encoding, delivery_mode=delivery_mode) return message_data, properties def publish(self, message, exchange, routing_key, mandatory=None, immediate=None, headers=None): """Publish a message to a named exchange.""" body, properties = message if headers: properties.headers = headers ret = self.channel.basic_publish(body=body, properties=properties, exchange=exchange, routing_key=routing_key, mandatory=mandatory, immediate=immediate) if mandatory or immediate: self.close() def qos(self, prefetch_size, prefetch_count, apply_global=False): """Request specific Quality of Service.""" self.channel.basic_qos(prefetch_size, prefetch_count, apply_global) def flow(self, active): """Enable/disable flow from peer.""" self.channel.flow(active) class AsyncoreBackend(SyncBackend): _connection_cls = pika.AsyncoreConnection carrot-0.10.7/carrot/backends/pyamqplib.py000644 000765 000024 00000031110 11446066105 020425 0ustar00askstaff000000 000000 """ `amqplib`_ backend for carrot. .. _`amqplib`: http://barryp.org/software/py-amqplib/ """ from amqplib.client_0_8 import transport # amqplib's handshake mistakenly identifies as protocol version 1191, # this breaks in RabbitMQ tip, which no longer falls back to # 0-8 for unknown ids. transport.AMQP_PROTOCOL_HEADER = "AMQP\x01\x01\x08\x00" from amqplib import client_0_8 as amqp from amqplib.client_0_8.exceptions import AMQPChannelException from amqplib.client_0_8.serialization import AMQPReader, AMQPWriter from carrot.backends.base import BaseMessage, BaseBackend from itertools import count import warnings import weakref DEFAULT_PORT = 5672 class Connection(amqp.Connection): def drain_events(self, allowed_methods=None, timeout=None): """Wait for an event on any channel.""" return self.wait_multi(self.channels.values(), timeout=timeout) def wait_multi(self, channels, allowed_methods=None, timeout=None): """Wait for an event on a channel.""" chanmap = dict((chan.channel_id, chan) for chan in channels) chanid, method_sig, args, content = self._wait_multiple( chanmap.keys(), allowed_methods, timeout=timeout) channel = chanmap[chanid] if content \ and channel.auto_decode \ and hasattr(content, 'content_encoding'): try: content.body = content.body.decode(content.content_encoding) except Exception: pass amqp_method = channel._METHOD_MAP.get(method_sig, None) if amqp_method is None: raise Exception('Unknown AMQP method (%d, %d)' % method_sig) if content is None: return amqp_method(channel, args) else: return amqp_method(channel, args, content) def read_timeout(self, timeout=None): if timeout is None: return self.method_reader.read_method() sock = self.transport.sock prev = sock.gettimeout() sock.settimeout(timeout) try: return self.method_reader.read_method() finally: sock.settimeout(prev) def _wait_multiple(self, channel_ids, allowed_methods, timeout=None): for channel_id in channel_ids: method_queue = self.channels[channel_id].method_queue for queued_method in method_queue: method_sig = queued_method[0] if (allowed_methods is None) \ or (method_sig in allowed_methods) \ or (method_sig == (20, 40)): method_queue.remove(queued_method) method_sig, args, content = queued_method return channel_id, method_sig, args, content # Nothing queued, need to wait for a method from the peer while True: channel, method_sig, args, content = self.read_timeout(timeout) if (channel in channel_ids) \ and ((allowed_methods is None) \ or (method_sig in allowed_methods) \ or (method_sig == (20, 40))): return channel, method_sig, args, content # Not the channel and/or method we were looking for. Queue # this method for later self.channels[channel].method_queue.append((method_sig, args, content)) # # If we just queued up a method for channel 0 (the Connection # itself) it's probably a close method in reaction to some # error, so deal with it right away. # if channel == 0: self.wait() class QueueAlreadyExistsWarning(UserWarning): """A queue with that name already exists, so a recently changed ``routing_key`` or other settings might be ignored unless you rename the queue or restart the broker.""" class Message(BaseMessage): """A message received by the broker. Usually you don't insantiate message objects yourself, but receive them using a :class:`carrot.messaging.Consumer`. :param backend: see :attr:`backend`. :param amqp_message: see :attr:`_amqp_message`. .. attribute:: body The message body. .. attribute:: delivery_tag The message delivery tag, uniquely identifying this message. .. attribute:: backend The message backend used. A subclass of :class:`carrot.backends.base.BaseBackend`. .. attribute:: _amqp_message A :class:`amqplib.client_0_8.basic_message.Message` instance. This is a private attribute and should not be accessed by production code. """ def __init__(self, backend, amqp_message, **kwargs): self._amqp_message = amqp_message self.backend = backend for attr_name in ("body", "delivery_tag", "content_type", "content_encoding", "delivery_info"): kwargs[attr_name] = getattr(amqp_message, attr_name, None) super(Message, self).__init__(backend, **kwargs) class Backend(BaseBackend): """amqplib backend :param connection: see :attr:`connection`. .. attribute:: connection A :class:`carrot.connection.BrokerConnection` instance. An established connection to the broker. """ default_port = DEFAULT_PORT Message = Message def __init__(self, connection, **kwargs): self.connection = connection self.default_port = kwargs.get("default_port", self.default_port) self._channel_ref = None @property def _channel(self): return callable(self._channel_ref) and self._channel_ref() @property def channel(self): """If no channel exists, a new one is requested.""" if not self._channel: connection = self.connection.connection self._channel_ref = weakref.ref(connection.channel()) return self._channel def establish_connection(self): """Establish connection to the AMQP broker.""" conninfo = self.connection if not conninfo.hostname: raise KeyError("Missing hostname for AMQP connection.") if conninfo.userid is None: raise KeyError("Missing user id for AMQP connection.") if conninfo.password is None: raise KeyError("Missing password for AMQP connection.") if not conninfo.port: conninfo.port = self.default_port return Connection(host=conninfo.host, userid=conninfo.userid, password=conninfo.password, virtual_host=conninfo.virtual_host, insist=conninfo.insist, ssl=conninfo.ssl, connect_timeout=conninfo.connect_timeout) def close_connection(self, connection): """Close the AMQP broker connection.""" connection.close() def queue_exists(self, queue): """Check if a queue has been declared. :rtype bool: """ try: self.channel.queue_declare(queue=queue, passive=True) except AMQPChannelException, e: if e.amqp_reply_code == 404: return False raise e else: return True def queue_delete(self, queue, if_unused=False, if_empty=False): """Delete queue by name.""" return self.channel.queue_delete(queue, if_unused, if_empty) def queue_purge(self, queue, **kwargs): """Discard all messages in the queue. This will delete the messages and results in an empty queue.""" return self.channel.queue_purge(queue=queue) def queue_declare(self, queue, durable, exclusive, auto_delete, warn_if_exists=False, arguments=None): """Declare a named queue.""" if warn_if_exists and self.queue_exists(queue): warnings.warn(QueueAlreadyExistsWarning( QueueAlreadyExistsWarning.__doc__)) return self.channel.queue_declare(queue=queue, durable=durable, exclusive=exclusive, auto_delete=auto_delete, arguments=arguments) def exchange_declare(self, exchange, type, durable, auto_delete): """Declare an named exchange.""" return self.channel.exchange_declare(exchange=exchange, type=type, durable=durable, auto_delete=auto_delete) def queue_bind(self, queue, exchange, routing_key, arguments=None): """Bind queue to an exchange using a routing key.""" return self.channel.queue_bind(queue=queue, exchange=exchange, routing_key=routing_key, arguments=arguments) def message_to_python(self, raw_message): """Convert encoded message body back to a Python value.""" return self.Message(backend=self, amqp_message=raw_message) def get(self, queue, no_ack=False): """Receive a message from a declared queue by name. :returns: A :class:`Message` object if a message was received, ``None`` otherwise. If ``None`` was returned, it probably means there was no messages waiting on the queue. """ raw_message = self.channel.basic_get(queue, no_ack=no_ack) if not raw_message: return None return self.message_to_python(raw_message) def declare_consumer(self, queue, no_ack, callback, consumer_tag, nowait=False): """Declare a consumer.""" return self.channel.basic_consume(queue=queue, no_ack=no_ack, callback=callback, consumer_tag=consumer_tag, nowait=nowait) def consume(self, limit=None): """Returns an iterator that waits for one message at a time.""" for total_message_count in count(): if limit and total_message_count >= limit: raise StopIteration if not self.channel.is_open: raise StopIteration self.channel.wait() yield True def cancel(self, consumer_tag): """Cancel a channel by consumer tag.""" if not self.channel.connection: return self.channel.basic_cancel(consumer_tag) def close(self): """Close the channel if open.""" if self._channel and self._channel.is_open: self._channel.close() self._channel_ref = None def ack(self, delivery_tag): """Acknowledge a message by delivery tag.""" return self.channel.basic_ack(delivery_tag) def reject(self, delivery_tag): """Reject a message by deliver tag.""" return self.channel.basic_reject(delivery_tag, requeue=False) def requeue(self, delivery_tag): """Reject and requeue a message by delivery tag.""" return self.channel.basic_reject(delivery_tag, requeue=True) def prepare_message(self, message_data, delivery_mode, priority=None, content_type=None, content_encoding=None): """Encapsulate data into a AMQP message.""" message = amqp.Message(message_data, priority=priority, content_type=content_type, content_encoding=content_encoding) message.properties["delivery_mode"] = delivery_mode return message def publish(self, message, exchange, routing_key, mandatory=None, immediate=None, headers=None): """Publish a message to a named exchange.""" if headers: message.properties["headers"] = headers ret = self.channel.basic_publish(message, exchange=exchange, routing_key=routing_key, mandatory=mandatory, immediate=immediate) if mandatory or immediate: self.close() def qos(self, prefetch_size, prefetch_count, apply_global=False): """Request specific Quality of Service.""" self.channel.basic_qos(prefetch_size, prefetch_count, apply_global) def flow(self, active): """Enable/disable flow from peer.""" self.channel.flow(active) carrot-0.10.7/carrot/backends/pystomp.py000644 000765 000024 00000014146 11444077101 020150 0ustar00askstaff000000 000000 import time import socket from itertools import count from stompy import Client from stompy import Empty as QueueEmpty from carrot.backends.base import BaseMessage, BaseBackend DEFAULT_PORT = 61613 class Message(BaseMessage): """A message received by the STOMP broker. Usually you don't insantiate message objects yourself, but receive them using a :class:`carrot.messaging.Consumer`. :param backend: see :attr:`backend`. :param frame: see :attr:`_frame`. .. attribute:: body The message body. .. attribute:: delivery_tag The message delivery tag, uniquely identifying this message. .. attribute:: backend The message backend used. A subclass of :class:`carrot.backends.base.BaseBackend`. .. attribute:: _frame The frame received by the STOMP client. This is considered a private variable and should never be used in production code. """ def __init__(self, backend, frame, **kwargs): self._frame = frame self.backend = backend kwargs["body"] = frame.body kwargs["delivery_tag"] = frame.headers["message-id"] kwargs["content_type"] = frame.headers.get("content-type") kwargs["content_encoding"] = frame.headers.get("content-encoding") kwargs["priority"] = frame.headers.get("priority") super(Message, self).__init__(backend, **kwargs) def ack(self): """Acknowledge this message as being processed., This will remove the message from the queue. :raises MessageStateError: If the message has already been acknowledged/requeued/rejected. """ if self.acknowledged: raise self.MessageStateError( "Message already acknowledged with state: %s" % self._state) self.backend.ack(self._frame) self._state = "ACK" def reject(self): raise NotImplementedError( "The STOMP backend does not implement basic.reject") def requeue(self): raise NotImplementedError( "The STOMP backend does not implement requeue") class Backend(BaseBackend): Stomp = Client Message = Message default_port = DEFAULT_PORT def __init__(self, connection, **kwargs): self.connection = connection self.default_port = kwargs.get("default_port", self.default_port) self._channel = None self._consumers = {} # open consumers by consumer tag self._callbacks = {} def establish_connection(self): conninfo = self.connection if not conninfo.port: conninfo.port = self.default_port stomp = self.Stomp(conninfo.hostname, conninfo.port) stomp.connect(username=conninfo.userid, password=conninfo.password) stomp.drain_events = self.drain_events return stomp def close_connection(self, connection): try: connection.disconnect() except socket.error: pass def queue_exists(self, queue): return True def queue_purge(self, queue, **kwargs): for purge_count in count(0): try: frame = self.channel.get_nowait() except QueueEmpty: return purge_count else: self.channel.ack(frame) def declare_consumer(self, queue, no_ack, callback, consumer_tag, **kwargs): ack = no_ack and "auto" or "client" self.channel.subscribe(queue, ack=ack) self._consumers[consumer_tag] = queue self._callbacks[queue] = callback def drain_events(self, timeout=None): start_time = time.time() while True: frame = self.channel.get() if frame: break if time.time() - time_start > timeout: raise socket.timeout("the operation timed out.") queue = frame.headers.get("destination") if not queue or queue not in self._callbacks: return self._callbacks[queue](frame) def consume(self, limit=None): """Returns an iterator that waits for one message at a time.""" for total_message_count in count(): if limit and total_message_count >= limit: raise StopIteration self.drain_events() yield True def queue_declare(self, queue, *args, **kwargs): self.channel.subscribe(queue, ack="client") def get(self, queue, no_ack=False): try: frame = self.channel.get_nowait() except QueueEmpty: return None else: return self.message_to_python(frame) def ack(self, frame): self.channel.ack(frame) def message_to_python(self, raw_message): """Convert encoded message body back to a Python value.""" return self.Message(backend=self, frame=raw_message) def prepare_message(self, message_data, delivery_mode, priority=0, content_type=None, content_encoding=None): persistent = "false" if delivery_mode == 2: persistent = "true" priority = priority or 0 return {"body": message_data, "persistent": persistent, "priority": priority, "content-encoding": content_encoding, "content-type": content_type} def publish(self, message, exchange, routing_key, **kwargs): message["destination"] = exchange self.channel.stomp.send(message) def cancel(self, consumer_tag): if not self._channel or consumer_tag not in self._consumers: return queue = self._consumers.pop(consumer_tag) self.channel.unsubscribe(queue) def close(self): for consumer_tag in self._consumers.keys(): self.cancel(consumer_tag) if self._channel: try: self._channel.disconnect() except socket.error: pass @property def channel(self): if not self._channel: # Sorry, but the python-stomp library needs one connection # for each channel. self._channel = self.establish_connection() return self._channel carrot-0.10.7/carrot/backends/queue.py000644 000765 000024 00000004670 11450640640 017563 0ustar00askstaff000000 000000 """ Backend for unit-tests, using the Python :mod:`Queue` module. """ from Queue import Queue from carrot.backends.base import BaseMessage, BaseBackend import time import itertools mqueue = Queue() class Message(BaseMessage): """Message received from the backend. See :class:`carrot.backends.base.BaseMessage`. """ class Backend(BaseBackend): """Backend using the Python :mod:`Queue` library. Usually only used while executing unit tests. Please not that this backend does not support queues, exchanges or routing keys, so *all messages will be sent to all consumers*. """ Message = Message def get(self, *args, **kwargs): """Get the next waiting message from the queue. :returns: A :class:`Message` instance, or ``None`` if there is no messages waiting. """ if not mqueue.qsize(): return None message_data, content_type, content_encoding = mqueue.get() return self.Message(backend=self, body=message_data, content_type=content_type, content_encoding=content_encoding) def establish_connection(self): # for drain_events return self def drain_events(self, timeout=None): message = self.get() if message: self.callback(message) else: time.sleep(0.1) def consume(self, limit=None): """Go into consume mode.""" for total_message_count in itertools.count(): if limit and total_message_count >= limit: raise StopIteration self.drain_events() yield True def declare_consumer(self, queue, no_ack, callback, consumer_tag, nowait=False): self.queue = queue self.no_ack = no_ack self.callback = callback self.consumer_tag = consumer_tag self.nowait = nowait def queue_purge(self, queue, **kwargs): """Discard all messages in the queue.""" qsize = mqueue.qsize() mqueue.queue.clear() return qsize def prepare_message(self, message_data, delivery_mode, content_type, content_encoding, **kwargs): """Prepare message for sending.""" return (message_data, content_type, content_encoding) def publish(self, message, exchange, routing_key, **kwargs): """Publish a message to the queue.""" mqueue.put(message)