kafka-1.3.2/0000755001271300127130000000000013031057517012362 5ustar dpowers00000000000000kafka-1.3.2/AUTHORS.md0000644001271300127130000000471412650776450014051 0ustar dpowers00000000000000# Current Maintainer * Dana Powers, [@dpkp](https://github.com/dpkp) # Original Author and First Commit * David Arthur, [@mumrah](https://github.com/mumrah) # Contributors - 2015 (alpha by username) * Alex Couture-Beil, [@alexcb](https://github.com/alexcb) * Ali-Akber Saifee, [@alisaifee](https://github.com/alisaifee) * Christophe-Marie Duquesne, [@chmduquesne](https://github.com/chmduquesne) * Thomas Dimson, [@cosbynator](https://github.com/cosbynator) * Kasper Jacobsen, [@Dinoshauer](https://github.com/Dinoshauer) * Ross Duggan, [@duggan](https://github.com/duggan) * Enrico Canzonieri, [@ecanzonieri](https://github.com/ecanzonieri) * haosdent, [@haosdent](https://github.com/haosdent) * Arturo Filastò, [@hellais](https://github.com/hellais) * Job Evers‐Meltzer, [@jobevers](https://github.com/jobevers) * Martin Olveyra, [@kalessin](https://github.com/kalessin) * Kubilay Kocak, [@koobs](https://github.com/koobs) * Matthew L Daniel * Eric Hewitt, [@meandthewallaby](https://github.com/meandthewallaby) * Oliver Jowett [@mutability](https://github.com/mutability) * Shaolei Zhou, [@reAsOn2010](https://github.com/reAsOn2010) * Oskari Saarenmaa, [@saaros](https://github.com/saaros) * John Anderson, [@sontek](https://github.com/sontek) * Eduard Iskandarov, [@toidi](https://github.com/toidi) * Todd Palino, [@toddpalino](https://github.com/toddpalino) * trbs, [@trbs](https://github.com/trbs) * Viktor Shlapakov, [@vshlapakov](https://github.com/vshlapakov) * Will Daly, [@wedaly](https://github.com/wedaly) * Warren Kiser, [@wkiser](https://github.com/wkiser) * William Ting, [@wting](https://github.com/wting) * Zack Dever, [@zackdever](https://github.com/zackdever) # More Contributors * Bruno Renié, [@brutasse](https://github.com/brutasse) * Thomas Dimson, [@cosbynator](https://github.com/cosbynator) * Jesse Myers, [@jessemyers](https://github.com/jessemyers) * Mahendra M, [@mahendra](https://github.com/mahendra) * Miguel Eduardo Gil Biraud, [@mgilbir](https://github.com/mgilbir) * Marc Labbé, [@mrtheb](https://github.com/mrtheb) * Patrick Lucas, [@patricklucas](https://github.com/patricklucas) * Omar Ghishan, [@rdiomar](https://github.com/rdiomar) - RIP, Omar. 2014 * Ivan Pouzyrevsky, [@sandello](https://github.com/sandello) * Lou Marvin Caraig, [@se7entyse7en](https://github.com/se7entyse7en) * waliaashish85, [@waliaashish85](https://github.com/waliaashish85) * Mark Roberts, [@wizzat](https://github.com/wizzat) Thanks to all who have contributed! kafka-1.3.2/CHANGES.md0000644001271300127130000006072413031057471013764 0ustar dpowers00000000000000# 1.3.2 (Dec 28, 2016) Core * Add kafka.serializer interfaces (dpkp 912) Consumer * KAFKA-3007: KafkaConsumer max_poll_records (dpkp 831) * Raise exception if given a non-str topic (ssaamm 824) * Immediately update metadata for pattern subscription (laz2 915) Producer * Update Partitioners for use with KafkaProducer (barrotsteindev 827) * Sort partitions before calling partitioner (ms7s 905) Client * Always check for request timeouts (dpkp 887) * When hostname lookup is necessary, do every connect (benauthor 812) Bugfixes * Fix errorcode check when socket.connect_ex raises an exception (guojh 907) * Fix fetcher bug when processing offset out of range (sibiryakov 860) * Fix possible request draining in ensure_active_group (dpkp 896) * Fix metadata refresh handling with 0.10+ brokers when topic list is empty (sibiryakov 867) * KafkaProducer should set timestamp in Message if provided (Drizzt1991 875) * Fix murmur2 bug handling python2 bytes that do not ascii encode (dpkp 815) * Monkeypatch max_in_flight_requests_per_connection when checking broker version (dpkp 834) * Fix message timestamp_type (qix 828) * Added ssl_password config option to KafkaProducer class (kierkegaard13 830) * from kafka import ConsumerRebalanceListener, OffsetAndMetadata * Use 0.10.0.1 for integration tests (dpkp 803) Logging / Error Messages * Always include an error for logging when the coordinator is marked dead (dpkp 890) * Only string-ify BrokerResponseError args if provided (dpkp 889) * Update warning re advertised.listeners / advertised.host.name (jeffwidman 878) * Fix unrecognized sasl_mechanism error message (sharego 883) Documentation * Add docstring for max_records (jeffwidman 897) * Fixup doc references to max_in_flight_requests_per_connection * Fix typo: passowrd --> password (jeffwidman 901) * Fix documentation typo 'Defualt' -> 'Default'. (rolando 895) * Added doc for `max_poll_records` option (Drizzt1991 881) * Remove old design notes from Kafka 8 era (jeffwidman 876) * Fix documentation typos (jeffwidman 874) * Fix quota violation exception message (dpkp 809) * Add comment for round robin partitioner with different subscriptions * Improve KafkaProducer docstring for retries configuration # 1.3.1 (Aug 8, 2016) Bugfixes * Fix AttributeError in BrokerConnectionMetrics after reconnecting # 1.3.0 (Aug 4, 2016) Incompatible Changes * Delete KafkaConnection class (dpkp 769) * Rename partition_assignment -> assignment in MemberMetadata for consistency * Move selectors34 and socketpair to kafka.vendor (dpkp 785) * Change api_version config to tuple; deprecate str with warning (dpkp 761) * Rename _DEFAULT_CONFIG -> DEFAULT_CONFIG in KafkaProducer (dpkp 788) Improvements * Vendor six 1.10.0 to eliminate runtime dependency (dpkp 785) * Add KafkaProducer and KafkaConsumer.metrics() with instrumentation similar to java client (dpkp 754 / 772 / 794) * Support Sasl PLAIN authentication (larsjsol PR 779) * Add checksum and size to RecordMetadata and ConsumerRecord (KAFKA-3196 / 770 / 594) * Use MetadataRequest v1 for 0.10+ api_version (dpkp 762) * Fix KafkaConsumer autocommit for 0.8 brokers (dpkp 756 / 706) * Improve error logging (dpkp 760 / 759) * Adapt benchmark scripts from https://github.com/mrafayaleem/kafka-jython (dpkp 754) * Add api_version config to KafkaClient (dpkp 761) * New Metadata method with_partitions() (dpkp 787) * Use socket_options configuration to setsockopts(). Default TCP_NODELAY (dpkp 783) * Expose selector type as config option (dpkp 764) * Drain pending requests to the coordinator before initiating group rejoin (dpkp 798) * Send combined size and payload bytes to socket to avoid potentially split packets with TCP_NODELAY (dpkp 797) Bugfixes * Ignore socket.error when checking for protocol out of sync prior to socket close (dpkp 792) * Fix offset fetch when partitions are manually assigned (KAFKA-3960 / 786) * Change pickle_method to use python3 special attributes (jpaulodit 777) * Fix ProduceResponse v2 throttle_time_ms * Always encode size with MessageSet (#771) * Avoid buffer overread when compressing messageset in KafkaProducer * Explicit format string argument indices for python 2.6 compatibility * Simplify RecordMetadata; short circuit callbacks (#768) * Fix autocommit when partitions assigned manually (KAFKA-3486 / #767 / #626) * Handle metadata updates during consumer rebalance (KAFKA-3117 / #766 / #701) * Add a consumer config option to exclude internal topics (KAFKA-2832 / #765) * Protect writes to wakeup socket with threading lock (#763 / #709) * Fetcher spending unnecessary time during metrics recording (KAFKA-3785) * Always use absolute_import (dpkp) Test / Fixtures * Catch select errors while capturing test fixture logs * Fix consumer group test race condition (dpkp 795) * Retry fixture failures on a different port (dpkp 796) * Dump fixture logs on failure Documentation * Fix misspelling of password (ssaamm 793) * Document the ssl_password config option (ssaamm 780) * Fix typo in KafkaConsumer documentation (ssaamm 775) * Expand consumer.fetcher inline comments * Update kafka configuration links -> 0.10.0.0 docs * Fixup metrics_sample_window_ms docstring in consumer # 1.2.5 (July 15, 2016) Bugfixes * Fix bug causing KafkaProducer to double-compress message batches on retry * Check for double-compressed messages in KafkaConsumer, log warning and optionally skip * Drop recursion in _unpack_message_set; only decompress once # 1.2.4 (July 8, 2016) Bugfixes * Update consumer_timeout_ms docstring - KafkaConsumer raises StopIteration, no longer ConsumerTimeout * Use explicit subscription state flag to handle seek() during message iteration * Fix consumer iteration on compacted topics (dpkp PR 752) * Support ssl_password config when loading cert chains (amckemie PR 750) # 1.2.3 (July 2, 2016) Patch Improvements * Fix gc error log: avoid AttributeError in _unregister_cleanup (dpkp PR 747) * Wakeup socket optimizations (dpkp PR 740) * Assert will be disabled by "python -O" (tyronecai PR 736) * Randomize order of topics/partitions processed by fetcher to improve balance (dpkp PR 732) * Allow client.check_version timeout to be set in Producer and Consumer constructors (eastlondoner PR 647) # 1.2.2 (June 21, 2016) Bugfixes * Clarify timeout unit in KafkaProducer close and flush (ms7s PR 734) * Avoid busy poll during metadata refresh failure with retry_backoff_ms (dpkp PR 733) * Check_version should scan nodes until version found or timeout (dpkp PR 731) * Fix bug which could cause least_loaded_node to always return the same unavailable node (dpkp PR 730) * Fix producer garbage collection with weakref in atexit handler (dpkp PR 728) * Close client selector to fix fd leak (msmith PR 729) * Tweak spelling mistake in error const (steve8918 PR 719) * Rearrange connection tests to separate legacy KafkaConnection # 1.2.1 (June 1, 2016) Bugfixes * Fix regression in MessageSet decoding wrt PartialMessages (#716) * Catch response decode errors and log details (#715) * Fix Legacy support url (#712 - JonasGroeger) * Update sphinx docs re 0.10 broker support # 1.2.0 (May 24, 2016) This release officially adds support for Kafka 0.10 * Add protocol support for ApiVersionRequest (dpkp PR 678) * KAFKA-3025: Message v1 -- add timetamp and relative offsets (dpkp PR 693) * Use Fetch/Produce API v2 for brokers >= 0.10 (uses message format v1) (dpkp PR 694) * Use standard LZ4 framing for v1 messages / kafka 0.10 (dpkp PR 695) Consumers * Update SimpleConsumer / legacy protocol to handle compressed messages (paulcavallaro PR 684) Producers * KAFKA-3388: Fix expiration of batches sitting in the accumulator (dpkp PR 699) * KAFKA-3197: when max.in.flight.request.per.connection = 1, attempt to guarantee ordering (dpkp PR 698) * Don't use soon-to-be-reserved keyword await as function name (FutureProduceResult) (dpkp PR 697) Clients * Fix socket leaks in KafkaClient (dpkp PR 696) Documentation Internals * Support SSL CRL [requires python 2.7.9+ / 3.4+] (vincentbernat PR 683) * Use original hostname for SSL checks (vincentbernat PR 682) * Always pass encoded message bytes to MessageSet.encode() * Raise ValueError on protocol encode/decode errors * Supplement socket.gaierror exception in BrokerConnection.connect() (erikbeebe PR 687) * BrokerConnection check_version: expect 0.9 to fail with CorrelationIdError * Fix small bug in Sensor (zackdever PR 679) # 1.1.1 (Apr 26, 2016) quick bugfixes * fix throttle_time_ms sensor handling (zackdever pr 667) * improve handling of disconnected sockets (easypost pr 666 / dpkp) * disable standard metadata refresh triggers during bootstrap (dpkp) * more predictable future callback/errback exceptions (zackdever pr 670) * avoid some exceptions in coordinator.__del__ (dpkp pr 668) # 1.1.0 (Apr 25, 2016) Consumers * Avoid resending FetchRequests that are pending on internal queue * Log debug messages when skipping fetched messages due to offset checks * KAFKA-3013: Include topic-partition in exception for expired batches * KAFKA-3318: clean up consumer logging and error messages * Improve unknown coordinator error handling * Improve auto-commit error handling when group_id is None * Add paused() API (zackdever PR 602) * Add default_offset_commit_callback to KafkaConsumer DEFAULT_CONFIGS Producers Clients * Support SSL connections * Use selectors module for non-blocking IO * Refactor KafkaClient connection management * Fix AttributeError in __del__ * SimpleClient: catch errors thrown by _get_leader_for_partition (zackdever PR 606) Documentation * Fix serializer/deserializer examples in README * Update max.block.ms docstring * Remove errant next(consumer) from consumer documentation * Add producer.flush() to usage docs Internals * Add initial metrics implementation (zackdever PR 637) * KAFKA-2136: support Fetch and Produce v1 (throttle_time_ms) * Use version-indexed lists for request/response protocol structs (dpkp PR 630) * Split kafka.common into kafka.structs and kafka.errors * Handle partial socket send() (dpkp PR 611) * Fix windows support (dpkp PR 603) * IPv6 support (TimEvens PR 615; Roguelazer PR 642) # 1.0.2 (Mar 14, 2016) Consumers * Improve KafkaConsumer Heartbeat handling (dpkp PR 583) * Fix KafkaConsumer.position bug (stefanth PR 578) * Raise TypeError when partition is not a TopicPartition (dpkp PR 587) * KafkaConsumer.poll should sleep to prevent tight-loops (dpkp PR 597) Producers * Fix producer threading bug that can crash sender (dpkp PR 590) * Fix bug in producer buffer pool reallocation (dpkp PR 585) * Remove spurious warnings when closing sync SimpleProducer (twm PR 567) * Fix FutureProduceResult.await() on python2.6 (dpkp) * Add optional timeout parameter to KafkaProducer.flush() (dpkp) * KafkaProducer Optimizations (zackdever PR 598) Clients * Improve error handling in SimpleClient.load_metadata_for_topics (dpkp) * Improve handling of KafkaClient.least_loaded_node failure (dpkp PR 588) Documentation * Fix KafkaError import error in docs (shichao-an PR 564) * Fix serializer / deserializer examples (scribu PR 573) Internals * Update to Kafka 0.9.0.1 for integration testing * Fix ifr.future.failure in conn.py (mortenlj PR 566) * Improve Zookeeper / Kafka Fixture management (dpkp) # 1.0.1 (Feb 19, 2016) Consumers * Add RangePartitionAssignor (and use as default); add assignor tests (dpkp PR 550) * Make sure all consumers are in same generation before stopping group test * Verify node ready before sending offset fetch request from coordinator * Improve warning when offset fetch request returns unknown topic / partition Producers * Warn if pending batches failed during flush * Fix concurrency bug in RecordAccumulator.ready() * Fix bug in SimpleBufferPool memory condition waiting / timeout * Support batch_size = 0 in producer buffers (dpkp PR 558) * Catch duplicate batch.done() calls [e.g., maybe_expire then a response errback] Clients Documentation * Improve kafka.cluster docstrings * Migrate load_example.py to KafkaProducer / KafkaConsumer Internals * Don't override system rcvbuf or sndbuf unless configured explicitly (dpkp PR 557) * Some attributes may not exist in __del__ if we failed assertions * Break up some circular references and close client wake pipes on __del__ (aisch PR 554) # 1.0.0 (Feb 15, 2016) This release includes significant code changes. Users of older kafka-python versions are encouraged to test upgrades before deploying to production as some interfaces and configuration options have changed. Users of SimpleConsumer / SimpleProducer / SimpleClient (formerly KafkaClient) from prior releases should migrate to KafkaConsumer / KafkaProducer. Low-level APIs (Simple*) are no longer being actively maintained and will be removed in a future release. For comprehensive API documentation, please see python help() / docstrings, kafka-python.readthedocs.org, or run `tox -e docs` from source to build documentation locally. Consumers * KafkaConsumer re-written to emulate the new 0.9 kafka consumer (java client) and support coordinated consumer groups (feature requires >= 0.9.0.0 brokers) * Methods no longer available: * configure [initialize a new consumer instead] * set_topic_partitions [use subscribe() or assign()] * fetch_messages [use poll() or iterator interface] * get_partition_offsets * offsets [use committed(partition)] * task_done [handled internally by auto-commit; or commit offsets manually] * Configuration changes (consistent with updated java client): * lots of new configuration parameters -- see docs for details * auto_offset_reset: previously values were 'smallest' or 'largest', now values are 'earliest' or 'latest' * fetch_wait_max_ms is now fetch_max_wait_ms * max_partition_fetch_bytes is now max_partition_fetch_bytes * deserializer_class is now value_deserializer and key_deserializer * auto_commit_enable is now enable_auto_commit * auto_commit_interval_messages was removed * socket_timeout_ms was removed * refresh_leader_backoff_ms was removed * SimpleConsumer and MultiProcessConsumer are now deprecated and will be removed in a future release. Users are encouraged to migrate to KafkaConsumer. Producers * new producer class: KafkaProducer. Exposes the same interface as official java client. Async by default; returned future.get() can be called for synchronous blocking * SimpleProducer is now deprecated and will be removed in a future release. Users are encouraged to migrate to KafkaProducer. Clients * synchronous KafkaClient renamed to SimpleClient. For backwards compatibility, you will get a SimpleClient via `from kafka import KafkaClient`. This will change in a future release. * All client calls use non-blocking IO under the hood. * Add probe method check_version() to infer broker versions. Documentation * Updated README and sphinx documentation to address new classes. * Docstring improvements to make python help() easier to use. Internals * Old protocol stack is deprecated. It has been moved to kafka.protocol.legacy and may be removed in a future release. * Protocol layer re-written using Type classes, Schemas and Structs (modeled on the java client). * Add support for LZ4 compression (including broken framing header checksum). # 0.9.5 (Dec 6, 2015) Consumers * Initial support for consumer coordinator: offsets only (toddpalino PR 420) * Allow blocking until some messages are received in SimpleConsumer (saaros PR 457) * Support subclass config changes in KafkaConsumer (zackdever PR 446) * Support retry semantics in MultiProcessConsumer (barricadeio PR 456) * Support partition_info in MultiProcessConsumer (scrapinghub PR 418) * Enable seek() to an absolute offset in SimpleConsumer (haosdent PR 412) * Add KafkaConsumer.close() (ucarion PR 426) Producers * Catch client.reinit() exceptions in async producer (dpkp) * Producer.stop() now blocks until async thread completes (dpkp PR 485) * Catch errors during load_metadata_for_topics in async producer (bschopman PR 467) * Add compression-level support for codecs that support it (trbs PR 454) * Fix translation of Java murmur2 code, fix byte encoding for Python 3 (chrischamberlin PR 439) * Only call stop() on not-stopped producer objects (docker-hub PR 435) * Allow null payload for deletion feature (scrapinghub PR 409) Clients * Use non-blocking io for broker aware requests (ecanzonieri PR 473) * Use debug logging level for metadata request (ecanzonieri PR 415) * Catch KafkaUnavailableError in _send_broker_aware_request (mutability PR 436) * Lower logging level on replica not available and commit (ecanzonieri PR 415) Documentation * Update docs and links wrt maintainer change (mumrah -> dpkp) Internals * Add py35 to tox testing * Update travis config to use container infrastructure * Add 0.8.2.2 and 0.9.0.0 resources for integration tests; update default official releases * new pylint disables for pylint 1.5.1 (zackdever PR 481) * Fix python3 / python2 comments re queue/Queue (dpkp) * Add Murmur2Partitioner to kafka __all__ imports (dpkp Issue 471) * Include LICENSE in PyPI sdist (koobs PR 441) # 0.9.4 (June 11, 2015) Consumers * Refactor SimpleConsumer internal fetch handling (dpkp PR 399) * Handle exceptions in SimpleConsumer commit() and reset_partition_offset() (dpkp PR 404) * Improve FailedPayloadsError handling in KafkaConsumer (dpkp PR 398) * KafkaConsumer: avoid raising KeyError in task_done (dpkp PR 389) * MultiProcessConsumer -- support configured partitions list (dpkp PR 380) * Fix SimpleConsumer leadership change handling (dpkp PR 393) * Fix SimpleConsumer connection error handling (reAsOn2010 PR 392) * Improve Consumer handling of 'falsy' partition values (wting PR 342) * Fix _offsets call error in KafkaConsumer (hellais PR 376) * Fix str/bytes bug in KafkaConsumer (dpkp PR 365) * Register atexit handlers for consumer and producer thread/multiprocess cleanup (dpkp PR 360) * Always fetch commit offsets in base consumer unless group is None (dpkp PR 356) * Stop consumer threads on delete (dpkp PR 357) * Deprecate metadata_broker_list in favor of bootstrap_servers in KafkaConsumer (dpkp PR 340) * Support pass-through parameters in multiprocess consumer (scrapinghub PR 336) * Enable offset commit on SimpleConsumer.seek (ecanzonieri PR 350) * Improve multiprocess consumer partition distribution (scrapinghub PR 335) * Ignore messages with offset less than requested (wkiser PR 328) * Handle OffsetOutOfRange in SimpleConsumer (ecanzonieri PR 296) Producers * Add Murmur2Partitioner (dpkp PR 378) * Log error types in SimpleProducer and SimpleConsumer (dpkp PR 405) * SimpleProducer support configuration of fail_on_error (dpkp PR 396) * Deprecate KeyedProducer.send() (dpkp PR 379) * Further improvements to async producer code (dpkp PR 388) * Add more configuration parameters for async producer (dpkp) * Deprecate SimpleProducer batch_send=True in favor of async (dpkp) * Improve async producer error handling and retry logic (vshlapakov PR 331) * Support message keys in async producer (vshlapakov PR 329) * Use threading instead of multiprocessing for Async Producer (vshlapakov PR 330) * Stop threads on __del__ (chmduquesne PR 324) * Fix leadership failover handling in KeyedProducer (dpkp PR 314) KafkaClient * Add .topics property for list of known topics (dpkp) * Fix request / response order guarantee bug in KafkaClient (dpkp PR 403) * Improve KafkaClient handling of connection failures in _get_conn (dpkp) * Client clears local metadata cache before updating from server (dpkp PR 367) * KafkaClient should return a response or error for each request - enable better retry handling (dpkp PR 366) * Improve str/bytes conversion in KafkaClient and KafkaConsumer (dpkp PR 332) * Always return sorted partition ids in client.get_partition_ids_for_topic() (dpkp PR 315) Documentation * Cleanup Usage Documentation * Improve KafkaConsumer documentation (dpkp PR 341) * Update consumer documentation (sontek PR 317) * Add doc configuration for tox (sontek PR 316) * Switch to .rst doc format (sontek PR 321) * Fixup google groups link in README (sontek PR 320) * Automate documentation at kafka-python.readthedocs.org Internals * Switch integration testing from 0.8.2.0 to 0.8.2.1 (dpkp PR 402) * Fix most flaky tests, improve debug logging, improve fixture handling (dpkp) * General style cleanups (dpkp PR 394) * Raise error on duplicate topic-partition payloads in protocol grouping (dpkp) * Use module-level loggers instead of simply 'kafka' (dpkp) * Remove pkg_resources check for __version__ at runtime (dpkp PR 387) * Make external API consistently support python3 strings for topic (kecaps PR 361) * Fix correlation id overflow (dpkp PR 355) * Cleanup kafka/common structs (dpkp PR 338) * Use context managers in gzip_encode / gzip_decode (dpkp PR 337) * Save failed request as FailedPayloadsError attribute (jobevers PR 302) * Remove unused kafka.queue (mumrah) # 0.9.3 (Feb 3, 2015) * Add coveralls.io support (sontek PR 307) * Fix python2.6 threading.Event bug in ReentrantTimer (dpkp PR 312) * Add kafka 0.8.2.0 to travis integration tests (dpkp PR 310) * Auto-convert topics to utf-8 bytes in Producer (sontek PR 306) * Fix reference cycle between SimpleConsumer and ReentrantTimer (zhaopengzp PR 309) * Add Sphinx API docs (wedaly PR 282) * Handle additional error cases exposed by 0.8.2.0 kafka server (dpkp PR 295) * Refactor error class management (alexcb PR 289) * Expose KafkaConsumer in __all__ for easy imports (Dinoshauer PR 286) * SimpleProducer starts on random partition by default (alexcb PR 288) * Add keys to compressed messages (meandthewallaby PR 281) * Add new high-level KafkaConsumer class based on java client api (dpkp PR 234) * Add KeyedProducer.send_messages api (pubnub PR 277) * Fix consumer pending() method (jettify PR 276) * Update low-level demo in README (sunisdown PR 274) * Include key in KeyedProducer messages (se7entyse7en PR 268) * Fix SimpleConsumer timeout behavior in get_messages (dpkp PR 238) * Fix error in consumer.py test against max_buffer_size (rthille/wizzat PR 225/242) * Improve string concat performance on pypy / py3 (dpkp PR 233) * Reorg directory layout for consumer/producer/partitioners (dpkp/wizzat PR 232/243) * Add OffsetCommitContext (locationlabs PR 217) * Metadata Refactor (dpkp PR 223) * Add Python 3 support (brutasse/wizzat - PR 227) * Minor cleanups - imports / README / PyPI classifiers (dpkp - PR 221) * Fix socket test (dpkp - PR 222) * Fix exception catching bug in test_failover_integration (zever - PR 216) # 0.9.2 (Aug 26, 2014) * Warn users that async producer does not reliably handle failures (dpkp - PR 213) * Fix spurious ConsumerFetchSizeTooSmall error in consumer (DataDog - PR 136) * Use PyLint for static error checking (dpkp - PR 208) * Strictly enforce str message type in producer.send_messages (dpkp - PR 211) * Add test timers via nose-timer plugin; list 10 slowest timings by default (dpkp) * Move fetching last known offset logic to a stand alone function (zever - PR 177) * Improve KafkaConnection and add more tests (dpkp - PR 196) * Raise TypeError if necessary when encoding strings (mdaniel - PR 204) * Use Travis-CI to publish tagged releases to pypi (tkuhlman / mumrah) * Use official binary tarballs for integration tests and parallelize travis tests (dpkp - PR 193) * Improve new-topic creation handling (wizzat - PR 174) # 0.9.1 (Aug 10, 2014) * Add codec parameter to Producers to enable compression (patricklucas - PR 166) * Support IPv6 hosts and network (snaury - PR 169) * Remove dependency on distribute (patricklucas - PR 163) * Fix connection error timeout and improve tests (wizzat - PR 158) * SimpleProducer randomization of initial round robin ordering (alexcb - PR 139) * Fix connection timeout in KafkaClient and KafkaConnection (maciejkula - PR 161) * Fix seek + commit behavior (wizzat - PR 148) # 0.9.0 (Mar 21, 2014) * Connection refactor and test fixes (wizzat - PR 134) * Fix when partition has no leader (mrtheb - PR 109) * Change Producer API to take topic as send argument, not as instance variable (rdiomar - PR 111) * Substantial refactor and Test Fixing (rdiomar - PR 88) * Fix Multiprocess Consumer on windows (mahendra - PR 62) * Improve fault tolerance; add integration tests (jimjh) * PEP8 / Flakes / Style cleanups (Vetoshkin Nikita; mrtheb - PR 59) * Setup Travis CI (jimjh - PR 53/54) * Fix import of BufferUnderflowError (jimjh - PR 49) * Fix code examples in README (StevenLeRoux - PR 47/48) # 0.8.0 * Changing auto_commit to False in [SimpleConsumer](kafka/consumer.py), until 0.8.1 is release offset commits are unsupported * Adding fetch_size_bytes to SimpleConsumer constructor to allow for user-configurable fetch sizes * Allow SimpleConsumer to automatically increase the fetch size if a partial message is read and no other messages were read during that fetch request. The increase factor is 1.5 * Exception classes moved to kafka.common kafka-1.3.2/kafka/0000755001271300127130000000000013031057517013437 5ustar dpowers00000000000000kafka-1.3.2/kafka/__init__.py0000644001271300127130000000365113031057471015554 0ustar dpowers00000000000000from __future__ import absolute_import __title__ = 'kafka' from .version import __version__ __author__ = 'Dana Powers' __license__ = 'Apache License 2.0' __copyright__ = 'Copyright 2016 Dana Powers, David Arthur, and Contributors' # Set default logging handler to avoid "No handler found" warnings. import logging try: # Python 2.7+ from logging import NullHandler except ImportError: class NullHandler(logging.Handler): def emit(self, record): pass logging.getLogger(__name__).addHandler(NullHandler()) from kafka.consumer import KafkaConsumer from kafka.consumer.subscription_state import ConsumerRebalanceListener from kafka.producer import KafkaProducer from kafka.conn import BrokerConnection from kafka.protocol import ( create_message, create_gzip_message, create_snappy_message) from kafka.partitioner import RoundRobinPartitioner, HashedPartitioner, Murmur2Partitioner from kafka.structs import TopicPartition, OffsetAndMetadata from kafka.serializer import Serializer, Deserializer # To be deprecated when KafkaProducer interface is released from kafka.client import SimpleClient from kafka.producer import SimpleProducer, KeyedProducer # deprecated in favor of KafkaConsumer from kafka.consumer import SimpleConsumer, MultiProcessConsumer import warnings class KafkaClient(SimpleClient): def __init__(self, *args, **kwargs): warnings.warn('The legacy KafkaClient interface has been moved to' ' kafka.SimpleClient - this import will break in a' ' future release', DeprecationWarning) super(KafkaClient, self).__init__(*args, **kwargs) __all__ = [ 'KafkaConsumer', 'KafkaProducer', 'KafkaClient', 'BrokerConnection', 'SimpleClient', 'SimpleProducer', 'KeyedProducer', 'RoundRobinPartitioner', 'HashedPartitioner', 'create_message', 'create_gzip_message', 'create_snappy_message', 'SimpleConsumer', 'MultiProcessConsumer', ] kafka-1.3.2/kafka/client.py0000644001271300127130000006730113031057471015275 0ustar dpowers00000000000000from __future__ import absolute_import import collections import copy import functools import logging import random import time from kafka.vendor import six import kafka.errors from kafka.errors import (UnknownError, ConnectionError, FailedPayloadsError, KafkaTimeoutError, KafkaUnavailableError, LeaderNotAvailableError, UnknownTopicOrPartitionError, NotLeaderForPartitionError, ReplicaNotAvailableError) from kafka.structs import TopicPartition, BrokerMetadata from kafka.conn import ( collect_hosts, BrokerConnection, ConnectionStates, get_ip_port_afi) from kafka.protocol import KafkaProtocol # New KafkaClient # this is not exposed in top-level imports yet, # due to conflicts with legacy SimpleConsumer / SimpleProducer usage from kafka.client_async import KafkaClient log = logging.getLogger(__name__) # Legacy KafkaClient interface -- will be deprecated soon class SimpleClient(object): CLIENT_ID = b'kafka-python' DEFAULT_SOCKET_TIMEOUT_SECONDS = 120 # NOTE: The timeout given to the client should always be greater than the # one passed to SimpleConsumer.get_message(), otherwise you can get a # socket timeout. def __init__(self, hosts, client_id=CLIENT_ID, timeout=DEFAULT_SOCKET_TIMEOUT_SECONDS, correlation_id=0): # We need one connection to bootstrap self.client_id = client_id self.timeout = timeout self.hosts = collect_hosts(hosts) self.correlation_id = correlation_id self._conns = {} self.brokers = {} # broker_id -> BrokerMetadata self.topics_to_brokers = {} # TopicPartition -> BrokerMetadata self.topic_partitions = {} # topic -> partition -> leader self.load_metadata_for_topics() # bootstrap with all metadata ################## # Private API # ################## def _get_conn(self, host, port, afi): """Get or create a connection to a broker using host and port""" host_key = (host, port) if host_key not in self._conns: self._conns[host_key] = BrokerConnection( host, port, afi, request_timeout_ms=self.timeout * 1000, client_id=self.client_id ) conn = self._conns[host_key] conn.connect() if conn.connected(): return conn timeout = time.time() + self.timeout while time.time() < timeout and conn.connecting(): if conn.connect() is ConnectionStates.CONNECTED: break else: time.sleep(0.05) else: conn.close() raise ConnectionError("%s:%s (%s)" % (host, port, afi)) return conn def _get_leader_for_partition(self, topic, partition): """ Returns the leader for a partition or None if the partition exists but has no leader. UnknownTopicOrPartitionError will be raised if the topic or partition is not part of the metadata. LeaderNotAvailableError is raised if server has metadata, but there is no current leader """ key = TopicPartition(topic, partition) # Use cached metadata if it is there if self.topics_to_brokers.get(key) is not None: return self.topics_to_brokers[key] # Otherwise refresh metadata # If topic does not already exist, this will raise # UnknownTopicOrPartitionError if not auto-creating # LeaderNotAvailableError otherwise until partitions are created self.load_metadata_for_topics(topic) # If the partition doesn't actually exist, raise if partition not in self.topic_partitions.get(topic, []): raise UnknownTopicOrPartitionError(key) # If there's no leader for the partition, raise leader = self.topic_partitions[topic][partition] if leader == -1: raise LeaderNotAvailableError((topic, partition)) # Otherwise return the BrokerMetadata return self.brokers[leader] def _get_coordinator_for_group(self, group): """ Returns the coordinator broker for a consumer group. GroupCoordinatorNotAvailableError will be raised if the coordinator does not currently exist for the group. GroupLoadInProgressError is raised if the coordinator is available but is still loading offsets from the internal topic """ resp = self.send_consumer_metadata_request(group) # If there's a problem with finding the coordinator, raise the # provided error kafka.errors.check_error(resp) # Otherwise return the BrokerMetadata return BrokerMetadata(resp.nodeId, resp.host, resp.port, None) def _next_id(self): """Generate a new correlation id""" # modulo to keep w/i int32 self.correlation_id = (self.correlation_id + 1) % 2**31 return self.correlation_id def _send_broker_unaware_request(self, payloads, encoder_fn, decoder_fn): """ Attempt to send a broker-agnostic request to one of the available brokers. Keep trying until you succeed. """ hosts = set() for broker in self.brokers.values(): host, port, afi = get_ip_port_afi(broker.host) hosts.add((host, broker.port, afi)) hosts.update(self.hosts) hosts = list(hosts) random.shuffle(hosts) for (host, port, afi) in hosts: try: conn = self._get_conn(host, port, afi) except ConnectionError: log.warning("Skipping unconnected connection: %s:%s (AFI %s)", host, port, afi) continue request = encoder_fn(payloads=payloads) future = conn.send(request) # Block while not future.is_done: conn.recv() if future.failed(): log.error("Request failed: %s", future.exception) continue return decoder_fn(future.value) raise KafkaUnavailableError('All servers failed to process request: %s' % hosts) def _payloads_by_broker(self, payloads): payloads_by_broker = collections.defaultdict(list) for payload in payloads: try: leader = self._get_leader_for_partition(payload.topic, payload.partition) except (KafkaUnavailableError, LeaderNotAvailableError, UnknownTopicOrPartitionError): leader = None payloads_by_broker[leader].append(payload) return dict(payloads_by_broker) def _send_broker_aware_request(self, payloads, encoder_fn, decoder_fn): """ Group a list of request payloads by topic+partition and send them to the leader broker for that partition using the supplied encode/decode functions Arguments: payloads: list of object-like entities with a topic (str) and partition (int) attribute; payloads with duplicate topic-partitions are not supported. encode_fn: a method to encode the list of payloads to a request body, must accept client_id, correlation_id, and payloads as keyword arguments decode_fn: a method to decode a response body into response objects. The response objects must be object-like and have topic and partition attributes Returns: List of response objects in the same order as the supplied payloads """ # encoders / decoders do not maintain ordering currently # so we need to keep this so we can rebuild order before returning original_ordering = [(p.topic, p.partition) for p in payloads] # Connection errors generally mean stale metadata # although sometimes it means incorrect api request # Unfortunately there is no good way to tell the difference # so we'll just reset metadata on all errors to be safe refresh_metadata = False # For each broker, send the list of request payloads # and collect the responses and errors payloads_by_broker = self._payloads_by_broker(payloads) responses = {} def failed_payloads(payloads): for payload in payloads: topic_partition = (str(payload.topic), payload.partition) responses[(topic_partition)] = FailedPayloadsError(payload) # For each BrokerConnection keep the real socket so that we can use # a select to perform unblocking I/O connections_by_future = {} for broker, broker_payloads in six.iteritems(payloads_by_broker): if broker is None: failed_payloads(broker_payloads) continue host, port, afi = get_ip_port_afi(broker.host) try: conn = self._get_conn(host, broker.port, afi) except ConnectionError: refresh_metadata = True failed_payloads(broker_payloads) continue request = encoder_fn(payloads=broker_payloads) # decoder_fn=None signal that the server is expected to not # send a response. This probably only applies to # ProduceRequest w/ acks = 0 expect_response = (decoder_fn is not None) future = conn.send(request, expect_response=expect_response) if future.failed(): refresh_metadata = True failed_payloads(broker_payloads) continue if not expect_response: for payload in broker_payloads: topic_partition = (str(payload.topic), payload.partition) responses[topic_partition] = None continue connections_by_future[future] = (conn, broker) conn = None while connections_by_future: futures = list(connections_by_future.keys()) for future in futures: if not future.is_done: conn, _ = connections_by_future[future] conn.recv() continue _, broker = connections_by_future.pop(future) if future.failed(): refresh_metadata = True failed_payloads(payloads_by_broker[broker]) else: for payload_response in decoder_fn(future.value): topic_partition = (str(payload_response.topic), payload_response.partition) responses[topic_partition] = payload_response if refresh_metadata: self.reset_all_metadata() # Return responses in the same order as provided return [responses[tp] for tp in original_ordering] def _send_consumer_aware_request(self, group, payloads, encoder_fn, decoder_fn): """ Send a list of requests to the consumer coordinator for the group specified using the supplied encode/decode functions. As the payloads that use consumer-aware requests do not contain the group (e.g. OffsetFetchRequest), all payloads must be for a single group. Arguments: group: the name of the consumer group (str) the payloads are for payloads: list of object-like entities with topic (str) and partition (int) attributes; payloads with duplicate topic+partition are not supported. encode_fn: a method to encode the list of payloads to a request body, must accept client_id, correlation_id, and payloads as keyword arguments decode_fn: a method to decode a response body into response objects. The response objects must be object-like and have topic and partition attributes Returns: List of response objects in the same order as the supplied payloads """ # encoders / decoders do not maintain ordering currently # so we need to keep this so we can rebuild order before returning original_ordering = [(p.topic, p.partition) for p in payloads] broker = self._get_coordinator_for_group(group) # Send the list of request payloads and collect the responses and # errors responses = {} requestId = self._next_id() log.debug('Request %s to %s: %s', requestId, broker, payloads) request = encoder_fn(client_id=self.client_id, correlation_id=requestId, payloads=payloads) # Send the request, recv the response try: host, port, afi = get_ip_port_afi(broker.host) conn = self._get_conn(host, broker.port, afi) conn.send(requestId, request) except ConnectionError as e: log.warning('ConnectionError attempting to send request %s ' 'to server %s: %s', requestId, broker, e) for payload in payloads: topic_partition = (payload.topic, payload.partition) responses[topic_partition] = FailedPayloadsError(payload) # No exception, try to get response else: # decoder_fn=None signal that the server is expected to not # send a response. This probably only applies to # ProduceRequest w/ acks = 0 if decoder_fn is None: log.debug('Request %s does not expect a response ' '(skipping conn.recv)', requestId) for payload in payloads: topic_partition = (payload.topic, payload.partition) responses[topic_partition] = None return [] try: response = conn.recv(requestId) except ConnectionError as e: log.warning('ConnectionError attempting to receive a ' 'response to request %s from server %s: %s', requestId, broker, e) for payload in payloads: topic_partition = (payload.topic, payload.partition) responses[topic_partition] = FailedPayloadsError(payload) else: _resps = [] for payload_response in decoder_fn(response): topic_partition = (payload_response.topic, payload_response.partition) responses[topic_partition] = payload_response _resps.append(payload_response) log.debug('Response %s: %s', requestId, _resps) # Return responses in the same order as provided return [responses[tp] for tp in original_ordering] def __repr__(self): return '' % (self.client_id) def _raise_on_response_error(self, resp): # Response can be an unraised exception object (FailedPayloadsError) if isinstance(resp, Exception): raise resp # Or a server api error response try: kafka.errors.check_error(resp) except (UnknownTopicOrPartitionError, NotLeaderForPartitionError): self.reset_topic_metadata(resp.topic) raise # Return False if no error to enable list comprehensions return False ################# # Public API # ################# def close(self): for conn in self._conns.values(): conn.close() def copy(self): """ Create an inactive copy of the client object, suitable for passing to a separate thread. Note that the copied connections are not initialized, so reinit() must be called on the returned copy. """ _conns = self._conns self._conns = {} c = copy.deepcopy(self) self._conns = _conns return c def reinit(self): timeout = time.time() + self.timeout conns = set(self._conns.values()) for conn in conns: conn.close() conn.connect() while time.time() < timeout: for conn in list(conns): conn.connect() if conn.connected(): conns.remove(conn) if not conns: break def reset_topic_metadata(self, *topics): for topic in topics: for topic_partition in list(self.topics_to_brokers.keys()): if topic_partition.topic == topic: del self.topics_to_brokers[topic_partition] if topic in self.topic_partitions: del self.topic_partitions[topic] def reset_all_metadata(self): self.topics_to_brokers.clear() self.topic_partitions.clear() def has_metadata_for_topic(self, topic): return ( topic in self.topic_partitions and len(self.topic_partitions[topic]) > 0 ) def get_partition_ids_for_topic(self, topic): if topic not in self.topic_partitions: return [] return sorted(list(self.topic_partitions[topic])) @property def topics(self): return list(self.topic_partitions.keys()) def ensure_topic_exists(self, topic, timeout = 30): start_time = time.time() while not self.has_metadata_for_topic(topic): if time.time() > start_time + timeout: raise KafkaTimeoutError('Unable to create topic {0}'.format(topic)) self.load_metadata_for_topics(topic, ignore_leadernotavailable=True) time.sleep(.5) def load_metadata_for_topics(self, *topics, **kwargs): """Fetch broker and topic-partition metadata from the server. Updates internal data: broker list, topic/partition list, and topic/parition -> broker map. This method should be called after receiving any error. Note: Exceptions *will not* be raised in a full refresh (i.e. no topic list). In this case, error codes will be logged as errors. Partition-level errors will also not be raised here (a single partition w/o a leader, for example). Arguments: *topics (optional): If a list of topics is provided, the metadata refresh will be limited to the specified topics only. ignore_leadernotavailable (bool): suppress LeaderNotAvailableError so that metadata is loaded correctly during auto-create. Default: False. Raises: UnknownTopicOrPartitionError: Raised for topics that do not exist, unless the broker is configured to auto-create topics. LeaderNotAvailableError: Raised for topics that do not exist yet, when the broker is configured to auto-create topics. Retry after a short backoff (topics/partitions are initializing). """ if 'ignore_leadernotavailable' in kwargs: ignore_leadernotavailable = kwargs['ignore_leadernotavailable'] else: ignore_leadernotavailable = False if topics: self.reset_topic_metadata(*topics) else: self.reset_all_metadata() resp = self.send_metadata_request(topics) log.debug('Updating broker metadata: %s', resp.brokers) log.debug('Updating topic metadata: %s', [topic for _, topic, _ in resp.topics]) self.brokers = dict([(nodeId, BrokerMetadata(nodeId, host, port, None)) for nodeId, host, port in resp.brokers]) for error, topic, partitions in resp.topics: # Errors expected for new topics if error: error_type = kafka.errors.kafka_errors.get(error, UnknownError) if error_type in (UnknownTopicOrPartitionError, LeaderNotAvailableError): log.error('Error loading topic metadata for %s: %s (%s)', topic, error_type, error) if topic not in topics: continue elif (error_type is LeaderNotAvailableError and ignore_leadernotavailable): continue raise error_type(topic) self.topic_partitions[topic] = {} for error, partition, leader, _, _ in partitions: self.topic_partitions[topic][partition] = leader # Populate topics_to_brokers dict topic_part = TopicPartition(topic, partition) # Check for partition errors if error: error_type = kafka.errors.kafka_errors.get(error, UnknownError) # If No Leader, topics_to_brokers topic_partition -> None if error_type is LeaderNotAvailableError: log.error('No leader for topic %s partition %d', topic, partition) self.topics_to_brokers[topic_part] = None continue # If one of the replicas is unavailable -- ignore # this error code is provided for admin purposes only # we never talk to replicas, only the leader elif error_type is ReplicaNotAvailableError: log.debug('Some (non-leader) replicas not available for topic %s partition %d', topic, partition) else: raise error_type(topic_part) # If Known Broker, topic_partition -> BrokerMetadata if leader in self.brokers: self.topics_to_brokers[topic_part] = self.brokers[leader] # If Unknown Broker, fake BrokerMetadata so we don't lose the id # (not sure how this could happen. server could be in bad state) else: self.topics_to_brokers[topic_part] = BrokerMetadata( leader, None, None, None ) def send_metadata_request(self, payloads=[], fail_on_error=True, callback=None): encoder = KafkaProtocol.encode_metadata_request decoder = KafkaProtocol.decode_metadata_response return self._send_broker_unaware_request(payloads, encoder, decoder) def send_consumer_metadata_request(self, payloads=[], fail_on_error=True, callback=None): encoder = KafkaProtocol.encode_consumer_metadata_request decoder = KafkaProtocol.decode_consumer_metadata_response return self._send_broker_unaware_request(payloads, encoder, decoder) def send_produce_request(self, payloads=[], acks=1, timeout=1000, fail_on_error=True, callback=None): """ Encode and send some ProduceRequests ProduceRequests will be grouped by (topic, partition) and then sent to a specific broker. Output is a list of responses in the same order as the list of payloads specified Arguments: payloads (list of ProduceRequest): produce requests to send to kafka ProduceRequest payloads must not contain duplicates for any topic-partition. acks (int, optional): how many acks the servers should receive from replica brokers before responding to the request. If it is 0, the server will not send any response. If it is 1, the server will wait until the data is written to the local log before sending a response. If it is -1, the server will wait until the message is committed by all in-sync replicas before sending a response. For any value > 1, the server will wait for this number of acks to occur (but the server will never wait for more acknowledgements than there are in-sync replicas). defaults to 1. timeout (int, optional): maximum time in milliseconds the server can await the receipt of the number of acks, defaults to 1000. fail_on_error (bool, optional): raise exceptions on connection and server response errors, defaults to True. callback (function, optional): instead of returning the ProduceResponse, first pass it through this function, defaults to None. Returns: list of ProduceResponses, or callback results if supplied, in the order of input payloads """ encoder = functools.partial( KafkaProtocol.encode_produce_request, acks=acks, timeout=timeout) if acks == 0: decoder = None else: decoder = KafkaProtocol.decode_produce_response resps = self._send_broker_aware_request(payloads, encoder, decoder) return [resp if not callback else callback(resp) for resp in resps if resp is not None and (not fail_on_error or not self._raise_on_response_error(resp))] def send_fetch_request(self, payloads=[], fail_on_error=True, callback=None, max_wait_time=100, min_bytes=4096): """ Encode and send a FetchRequest Payloads are grouped by topic and partition so they can be pipelined to the same brokers. """ encoder = functools.partial(KafkaProtocol.encode_fetch_request, max_wait_time=max_wait_time, min_bytes=min_bytes) resps = self._send_broker_aware_request( payloads, encoder, KafkaProtocol.decode_fetch_response) return [resp if not callback else callback(resp) for resp in resps if not fail_on_error or not self._raise_on_response_error(resp)] def send_offset_request(self, payloads=[], fail_on_error=True, callback=None): resps = self._send_broker_aware_request( payloads, KafkaProtocol.encode_offset_request, KafkaProtocol.decode_offset_response) return [resp if not callback else callback(resp) for resp in resps if not fail_on_error or not self._raise_on_response_error(resp)] def send_offset_commit_request(self, group, payloads=[], fail_on_error=True, callback=None): encoder = functools.partial(KafkaProtocol.encode_offset_commit_request, group=group) decoder = KafkaProtocol.decode_offset_commit_response resps = self._send_broker_aware_request(payloads, encoder, decoder) return [resp if not callback else callback(resp) for resp in resps if not fail_on_error or not self._raise_on_response_error(resp)] def send_offset_fetch_request(self, group, payloads=[], fail_on_error=True, callback=None): encoder = functools.partial(KafkaProtocol.encode_offset_fetch_request, group=group) decoder = KafkaProtocol.decode_offset_fetch_response resps = self._send_broker_aware_request(payloads, encoder, decoder) return [resp if not callback else callback(resp) for resp in resps if not fail_on_error or not self._raise_on_response_error(resp)] def send_offset_fetch_request_kafka(self, group, payloads=[], fail_on_error=True, callback=None): encoder = functools.partial(KafkaProtocol.encode_offset_fetch_request, group=group, from_kafka=True) decoder = KafkaProtocol.decode_offset_fetch_response resps = self._send_consumer_aware_request(group, payloads, encoder, decoder) return [resp if not callback else callback(resp) for resp in resps if not fail_on_error or not self._raise_on_response_error(resp)] kafka-1.3.2/kafka/client_async.py0000644001271300127130000011451313031057471016470 0ustar dpowers00000000000000from __future__ import absolute_import, division import copy import functools import heapq import itertools import logging import random import threading # selectors in stdlib as of py3.4 try: import selectors # pylint: disable=import-error except ImportError: # vendored backport module from .vendor import selectors34 as selectors import socket import time from kafka.vendor import six from .cluster import ClusterMetadata from .conn import BrokerConnection, ConnectionStates, collect_hosts, get_ip_port_afi from . import errors as Errors from .future import Future from .metrics import AnonMeasurable from .metrics.stats import Avg, Count, Rate from .metrics.stats.rate import TimeUnit from .protocol.metadata import MetadataRequest from .protocol.produce import ProduceRequest from .vendor import socketpair from .version import __version__ if six.PY2: ConnectionError = None log = logging.getLogger('kafka.client') class KafkaClient(object): """ A network client for asynchronous request/response network i/o. This is an internal class used to implement the user-facing producer and consumer clients. This class is not thread-safe! """ DEFAULT_CONFIG = { 'bootstrap_servers': 'localhost', 'client_id': 'kafka-python-' + __version__, 'request_timeout_ms': 40000, 'reconnect_backoff_ms': 50, 'max_in_flight_requests_per_connection': 5, 'receive_buffer_bytes': None, 'send_buffer_bytes': None, 'socket_options': [(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)], 'retry_backoff_ms': 100, 'metadata_max_age_ms': 300000, 'security_protocol': 'PLAINTEXT', 'ssl_context': None, 'ssl_check_hostname': True, 'ssl_cafile': None, 'ssl_certfile': None, 'ssl_keyfile': None, 'ssl_password': None, 'ssl_crlfile': None, 'api_version': None, 'api_version_auto_timeout_ms': 2000, 'selector': selectors.DefaultSelector, 'metrics': None, 'metric_group_prefix': '', 'sasl_mechanism': None, 'sasl_plain_username': None, 'sasl_plain_password': None, } API_VERSIONS = [ (0, 10), (0, 9), (0, 8, 2), (0, 8, 1), (0, 8, 0) ] def __init__(self, **configs): """Initialize an asynchronous kafka client Keyword Arguments: bootstrap_servers: 'host[:port]' string (or list of 'host[:port]' strings) that the consumer should contact to bootstrap initial cluster metadata. This does not have to be the full node list. It just needs to have at least one broker that will respond to a Metadata API Request. Default port is 9092. If no servers are specified, will default to localhost:9092. client_id (str): a name for this client. This string is passed in each request to servers and can be used to identify specific server-side log entries that correspond to this client. Also submitted to GroupCoordinator for logging with respect to consumer group administration. Default: 'kafka-python-{version}' reconnect_backoff_ms (int): The amount of time in milliseconds to wait before attempting to reconnect to a given host. Default: 50. request_timeout_ms (int): Client request timeout in milliseconds. Default: 40000. retry_backoff_ms (int): Milliseconds to backoff when retrying on errors. Default: 100. max_in_flight_requests_per_connection (int): Requests are pipelined to kafka brokers up to this number of maximum requests per broker connection. Default: 5. receive_buffer_bytes (int): The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. Default: None (relies on system defaults). Java client defaults to 32768. send_buffer_bytes (int): The size of the TCP send buffer (SO_SNDBUF) to use when sending data. Default: None (relies on system defaults). Java client defaults to 131072. socket_options (list): List of tuple-arguments to socket.setsockopt to apply to broker connection sockets. Default: [(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)] metadata_max_age_ms (int): The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions. Default: 300000 security_protocol (str): Protocol used to communicate with brokers. Valid values are: PLAINTEXT, SSL. Default: PLAINTEXT. ssl_context (ssl.SSLContext): pre-configured SSLContext for wrapping socket connections. If provided, all other ssl_* configurations will be ignored. Default: None. ssl_check_hostname (bool): flag to configure whether ssl handshake should verify that the certificate matches the brokers hostname. default: true. ssl_cafile (str): optional filename of ca file to use in certificate veriication. default: none. ssl_certfile (str): optional filename of file in pem format containing the client certificate, as well as any ca certificates needed to establish the certificate's authenticity. default: none. ssl_keyfile (str): optional filename containing the client private key. default: none. ssl_password (str): optional password to be used when loading the certificate chain. default: none. ssl_crlfile (str): optional filename containing the CRL to check for certificate expiration. By default, no CRL check is done. When providing a file, only the leaf certificate will be checked against this CRL. The CRL can only be checked with Python 3.4+ or 2.7.9+. default: none. api_version (tuple): specify which kafka API version to use. Accepted values are: (0, 8, 0), (0, 8, 1), (0, 8, 2), (0, 9), (0, 10) If None, KafkaClient will attempt to infer the broker version by probing various APIs. Default: None api_version_auto_timeout_ms (int): number of milliseconds to throw a timeout exception from the constructor when checking the broker api version. Only applies if api_version is None selector (selectors.BaseSelector): Provide a specific selector implementation to use for I/O multiplexing. Default: selectors.DefaultSelector metrics (kafka.metrics.Metrics): Optionally provide a metrics instance for capturing network IO stats. Default: None. metric_group_prefix (str): Prefix for metric names. Default: '' sasl_mechanism (str): string picking sasl mechanism when security_protocol is SASL_PLAINTEXT or SASL_SSL. Currently only PLAIN is supported. Default: None sasl_plain_username (str): username for sasl PLAIN authentication. Default: None sasl_plain_password (str): password for sasl PLAIN authentication. Default: None """ self.config = copy.copy(self.DEFAULT_CONFIG) for key in self.config: if key in configs: self.config[key] = configs[key] if self.config['api_version'] is not None: assert self.config['api_version'] in self.API_VERSIONS, ( 'api_version [{0}] must be one of: {1}'.format( self.config['api_version'], str(self.API_VERSIONS))) self.cluster = ClusterMetadata(**self.config) self._topics = set() # empty set will fetch all topic metadata self._metadata_refresh_in_progress = False self._last_no_node_available_ms = 0 self._selector = self.config['selector']() self._conns = {} self._connecting = set() self._refresh_on_disconnects = True self._delayed_tasks = DelayedTaskQueue() self._last_bootstrap = 0 self._bootstrap_fails = 0 self._wake_r, self._wake_w = socket.socketpair() self._wake_r.setblocking(False) self._wake_lock = threading.Lock() self._selector.register(self._wake_r, selectors.EVENT_READ) self._closed = False self._sensors = None if self.config['metrics']: self._sensors = KafkaClientMetrics(self.config['metrics'], self.config['metric_group_prefix'], self._conns) self._bootstrap(collect_hosts(self.config['bootstrap_servers'])) # Check Broker Version if not set explicitly if self.config['api_version'] is None: check_timeout = self.config['api_version_auto_timeout_ms'] / 1000 self.config['api_version'] = self.check_version(timeout=check_timeout) def _bootstrap(self, hosts): # Exponential backoff if bootstrap fails backoff_ms = self.config['reconnect_backoff_ms'] * 2 ** self._bootstrap_fails next_at = self._last_bootstrap + backoff_ms / 1000.0 self._refresh_on_disconnects = False now = time.time() if next_at > now: log.debug("Sleeping %0.4f before bootstrapping again", next_at - now) time.sleep(next_at - now) self._last_bootstrap = time.time() if self.config['api_version'] is None or self.config['api_version'] < (0, 10): metadata_request = MetadataRequest[0]([]) else: metadata_request = MetadataRequest[1](None) for host, port, afi in hosts: log.debug("Attempting to bootstrap via node at %s:%s", host, port) cb = functools.partial(self._conn_state_change, 'bootstrap') bootstrap = BrokerConnection(host, port, afi, state_change_callback=cb, node_id='bootstrap', **self.config) bootstrap.connect() while bootstrap.connecting(): bootstrap.connect() if bootstrap.state is not ConnectionStates.CONNECTED: bootstrap.close() continue future = bootstrap.send(metadata_request) while not future.is_done: bootstrap.recv() if future.failed(): bootstrap.close() continue self.cluster.update_metadata(future.value) # A cluster with no topics can return no broker metadata # in that case, we should keep the bootstrap connection if not len(self.cluster.brokers()): self._conns['bootstrap'] = bootstrap else: bootstrap.close() self._bootstrap_fails = 0 break # No bootstrap found... else: log.error('Unable to bootstrap from %s', hosts) # Max exponential backoff is 2^12, x4000 (50ms -> 200s) self._bootstrap_fails = min(self._bootstrap_fails + 1, 12) self._refresh_on_disconnects = True def _can_connect(self, node_id): if node_id not in self._conns: if self.cluster.broker_metadata(node_id): return True return False conn = self._conns[node_id] return conn.state is ConnectionStates.DISCONNECTED and not conn.blacked_out() def _conn_state_change(self, node_id, conn): if conn.connecting(): # SSL connections can enter this state 2x (second during Handshake) if node_id not in self._connecting: self._connecting.add(node_id) self._selector.register(conn._sock, selectors.EVENT_WRITE) elif conn.connected(): log.debug("Node %s connected", node_id) if node_id in self._connecting: self._connecting.remove(node_id) try: self._selector.unregister(conn._sock) except KeyError: pass self._selector.register(conn._sock, selectors.EVENT_READ, conn) if self._sensors: self._sensors.connection_created.record() if 'bootstrap' in self._conns and node_id != 'bootstrap': bootstrap = self._conns.pop('bootstrap') # XXX: make conn.close() require error to cause refresh self._refresh_on_disconnects = False bootstrap.close() self._refresh_on_disconnects = True # Connection failures imply that our metadata is stale, so let's refresh elif conn.state is ConnectionStates.DISCONNECTING: if node_id in self._connecting: self._connecting.remove(node_id) try: self._selector.unregister(conn._sock) except KeyError: pass if self._sensors: self._sensors.connection_closed.record() if self._refresh_on_disconnects and not self._closed: log.warning("Node %s connection failed -- refreshing metadata", node_id) self.cluster.request_update() def _maybe_connect(self, node_id): """Idempotent non-blocking connection attempt to the given node id.""" if node_id not in self._conns: broker = self.cluster.broker_metadata(node_id) assert broker, 'Broker id %s not in current metadata' % node_id log.debug("Initiating connection to node %s at %s:%s", node_id, broker.host, broker.port) host, port, afi = get_ip_port_afi(broker.host) cb = functools.partial(self._conn_state_change, node_id) self._conns[node_id] = BrokerConnection(host, broker.port, afi, state_change_callback=cb, node_id=node_id, **self.config) conn = self._conns[node_id] if conn.connected(): return True conn.connect() return conn.connected() def ready(self, node_id): """Check whether a node is connected and ok to send more requests. Arguments: node_id (int): the id of the node to check Returns: bool: True if we are ready to send to the given node """ self._maybe_connect(node_id) return self.is_ready(node_id) def connected(self, node_id): """Return True iff the node_id is connected.""" if node_id not in self._conns: return False return self._conns[node_id].connected() def close(self, node_id=None): """Closes one or all broker connections. Arguments: node_id (int, optional): the id of the node to close """ if node_id is None: self._closed = True for conn in self._conns.values(): conn.close() self._wake_r.close() self._wake_w.close() self._selector.close() elif node_id in self._conns: self._conns[node_id].close() else: log.warning("Node %s not found in current connection list; skipping", node_id) return def is_disconnected(self, node_id): """Check whether the node connection has been disconnected or failed. A disconnected node has either been closed or has failed. Connection failures are usually transient and can be resumed in the next ready() call, but there are cases where transient failures need to be caught and re-acted upon. Arguments: node_id (int): the id of the node to check Returns: bool: True iff the node exists and is disconnected """ if node_id not in self._conns: return False return self._conns[node_id].disconnected() def connection_delay(self, node_id): """ Returns the number of milliseconds to wait, based on the connection state, before attempting to send data. When disconnected, this respects the reconnect backoff time. When connecting, returns 0 to allow non-blocking connect to finish. When connected, returns a very large number to handle slow/stalled connections. Arguments: node_id (int): The id of the node to check Returns: int: The number of milliseconds to wait. """ if node_id not in self._conns: return 0 conn = self._conns[node_id] time_waited_ms = time.time() - (conn.last_attempt or 0) if conn.state is ConnectionStates.DISCONNECTED: return max(self.config['reconnect_backoff_ms'] - time_waited_ms, 0) elif conn.connecting(): return 0 else: return 999999999 def is_ready(self, node_id): """Check whether a node is ready to send more requests. In addition to connection-level checks, this method also is used to block additional requests from being sent during a metadata refresh. Arguments: node_id (int): id of the node to check Returns: bool: True if the node is ready and metadata is not refreshing """ # if we need to update our metadata now declare all requests unready to # make metadata requests first priority if not self._metadata_refresh_in_progress and not self.cluster.ttl() == 0: if self._can_send_request(node_id): return True return False def _can_send_request(self, node_id): if node_id not in self._conns: return False conn = self._conns[node_id] return conn.connected() and conn.can_send_more() def send(self, node_id, request): """Send a request to a specific node. Arguments: node_id (int): destination node request (Struct): request object (not-encoded) Raises: AssertionError: if node_id is not in current cluster metadata Returns: Future: resolves to Response struct or Error """ if not self._maybe_connect(node_id): return Future().failure(Errors.NodeNotReadyError(node_id)) # Every request gets a response, except one special case: expect_response = True if isinstance(request, tuple(ProduceRequest)) and request.required_acks == 0: expect_response = False return self._conns[node_id].send(request, expect_response=expect_response) def poll(self, timeout_ms=None, future=None, sleep=True, delayed_tasks=True): """Try to read and write to sockets. This method will also attempt to complete node connections, refresh stale metadata, and run previously-scheduled tasks. Arguments: timeout_ms (int, optional): maximum amount of time to wait (in ms) for at least one response. Must be non-negative. The actual timeout will be the minimum of timeout, request timeout and metadata timeout. Default: request_timeout_ms future (Future, optional): if provided, blocks until future.is_done sleep (bool): if True and there is nothing to do (no connections or requests in flight), will sleep for duration timeout before returning empty results. Default: False. Returns: list: responses received (can be empty) """ if timeout_ms is None: timeout_ms = self.config['request_timeout_ms'] responses = [] # Loop for futures, break after first loop if None while True: # Attempt to complete pending connections for node_id in list(self._connecting): self._maybe_connect(node_id) # Send a metadata request if needed metadata_timeout_ms = self._maybe_refresh_metadata() # Send scheduled tasks if delayed_tasks: for task, task_future in self._delayed_tasks.pop_ready(): try: result = task() except Exception as e: log.error("Task %s failed: %s", task, e) task_future.failure(e) else: task_future.success(result) # If we got a future that is already done, don't block in _poll if future and future.is_done: timeout = 0 else: timeout = min( timeout_ms, metadata_timeout_ms, self._delayed_tasks.next_at() * 1000, self.config['request_timeout_ms']) timeout = max(0, timeout / 1000.0) # avoid negative timeouts responses.extend(self._poll(timeout, sleep=sleep)) # If all we had was a timeout (future is None) - only do one poll # If we do have a future, we keep looping until it is done if not future or future.is_done: break return responses def _poll(self, timeout, sleep=True): # select on reads across all connected sockets, blocking up to timeout assert self.in_flight_request_count() > 0 or self._connecting or sleep responses = [] processed = set() start_select = time.time() ready = self._selector.select(timeout) end_select = time.time() if self._sensors: self._sensors.select_time.record((end_select - start_select) * 1000000000) for key, events in ready: if key.fileobj is self._wake_r: self._clear_wake_fd() continue elif not (events & selectors.EVENT_READ): continue conn = key.data processed.add(conn) if not conn.in_flight_requests: # if we got an EVENT_READ but there were no in-flight requests, one of # two things has happened: # # 1. The remote end closed the connection (because it died, or because # a firewall timed out, or whatever) # 2. The protocol is out of sync. # # either way, we can no longer safely use this connection # # Do a 1-byte read to check protocol didnt get out of sync, and then close the conn try: unexpected_data = key.fileobj.recv(1) if unexpected_data: # anything other than a 0-byte read means protocol issues log.warning('Protocol out of sync on %r, closing', conn) except socket.error: pass conn.close() continue # Accumulate as many responses as the connection has pending while conn.in_flight_requests: response = conn.recv() # Note: conn.recv runs callbacks / errbacks # Incomplete responses are buffered internally # while conn.in_flight_requests retains the request if not response: break responses.append(response) # Check for additional pending SSL bytes if self.config['security_protocol'] in ('SSL', 'SASL_SSL'): # TODO: optimize for conn in self._conns.values(): if conn not in processed and conn.connected() and conn._sock.pending(): response = conn.recv() if response: responses.append(response) for conn in six.itervalues(self._conns): if conn.requests_timed_out(): log.warning('%s timed out after %s ms. Closing connection.', conn, conn.config['request_timeout_ms']) conn.close(error=Errors.RequestTimedOutError( 'Request timed out after %s ms' % conn.config['request_timeout_ms'])) if self._sensors: self._sensors.io_time.record((time.time() - end_select) * 1000000000) return responses def in_flight_request_count(self, node_id=None): """Get the number of in-flight requests for a node or all nodes. Arguments: node_id (int, optional): a specific node to check. If unspecified, return the total for all nodes Returns: int: pending in-flight requests for the node, or all nodes if None """ if node_id is not None: if node_id not in self._conns: return 0 return len(self._conns[node_id].in_flight_requests) else: return sum([len(conn.in_flight_requests) for conn in self._conns.values()]) def least_loaded_node(self): """Choose the node with fewest outstanding requests, with fallbacks. This method will prefer a node with an existing connection, but will potentially choose a node for which we don't yet have a connection if all existing connections are in use. This method will never choose a node that was disconnected within the reconnect backoff period. If all else fails, the method will attempt to bootstrap again using the bootstrap_servers list. Returns: node_id or None if no suitable node was found """ nodes = [broker.nodeId for broker in self.cluster.brokers()] random.shuffle(nodes) inflight = float('inf') found = None for node_id in nodes: conn = self._conns.get(node_id) connected = conn is not None and conn.connected() blacked_out = conn is not None and conn.blacked_out() curr_inflight = len(conn.in_flight_requests) if conn else 0 if connected and curr_inflight == 0: # if we find an established connection # with no in-flight requests, we can stop right away return node_id elif not blacked_out and curr_inflight < inflight: # otherwise if this is the best we have found so far, record that inflight = curr_inflight found = node_id if found is not None: return found # some broker versions return an empty list of broker metadata # if there are no topics created yet. the bootstrap process # should detect this and keep a 'bootstrap' node alive until # a non-bootstrap node is connected and non-empty broker # metadata is available elif 'bootstrap' in self._conns: return 'bootstrap' # Last option: try to bootstrap again # this should only happen if no prior bootstrap has been successful log.error('No nodes found in metadata -- retrying bootstrap') self._bootstrap(collect_hosts(self.config['bootstrap_servers'])) return None def set_topics(self, topics): """Set specific topics to track for metadata. Arguments: topics (list of str): topics to check for metadata Returns: Future: resolves after metadata request/response """ if set(topics).difference(self._topics): future = self.cluster.request_update() else: future = Future().success(set(topics)) self._topics = set(topics) return future def add_topic(self, topic): """Add a topic to the list of topics tracked via metadata. Arguments: topic (str): topic to track Returns: Future: resolves after metadata request/response """ if topic in self._topics: return Future().success(set(self._topics)) self._topics.add(topic) return self.cluster.request_update() # request metadata update on disconnect and timedout def _maybe_refresh_metadata(self): """Send a metadata request if needed. Returns: int: milliseconds until next refresh """ ttl = self.cluster.ttl() next_reconnect_ms = self._last_no_node_available_ms + self.cluster.refresh_backoff() next_reconnect_ms = max(next_reconnect_ms - time.time() * 1000, 0) wait_for_in_progress_ms = 9999999999 if self._metadata_refresh_in_progress else 0 timeout = max(ttl, next_reconnect_ms, wait_for_in_progress_ms) if timeout == 0: node_id = self.least_loaded_node() if node_id is None: log.debug("Give up sending metadata request since no node is available") # mark the timestamp for no node available to connect self._last_no_node_available_ms = time.time() * 1000 return timeout if self._can_send_request(node_id): topics = list(self._topics) if self.cluster.need_all_topic_metadata or not topics: topics = [] if self.config['api_version'] < (0, 10) else None api_version = 0 if self.config['api_version'] < (0, 10) else 1 request = MetadataRequest[api_version](topics) log.debug("Sending metadata request %s to node %s", request, node_id) future = self.send(node_id, request) future.add_callback(self.cluster.update_metadata) future.add_errback(self.cluster.failed_update) self._metadata_refresh_in_progress = True def refresh_done(val_or_error): self._metadata_refresh_in_progress = False future.add_callback(refresh_done) future.add_errback(refresh_done) elif self._can_connect(node_id): log.debug("Initializing connection to node %s for metadata request", node_id) self._maybe_connect(node_id) # If initiateConnect failed immediately, this node will be put into blackout and we # should allow immediately retrying in case there is another candidate node. If it # is still connecting, the worst case is that we end up setting a longer timeout # on the next round and then wait for the response. else: # connected, but can't send more OR connecting # In either case, we just need to wait for a network event to let us know the selected # connection might be usable again. self._last_no_node_available_ms = time.time() * 1000 return timeout def schedule(self, task, at): """Schedule a new task to be executed at the given time. This is "best-effort" scheduling and should only be used for coarse synchronization. A task cannot be scheduled for multiple times simultaneously; any previously scheduled instance of the same task will be cancelled. Arguments: task (callable): task to be scheduled at (float or int): epoch seconds when task should run Returns: Future: resolves to result of task call, or exception if raised """ return self._delayed_tasks.add(task, at) def unschedule(self, task): """Unschedule a task. This will remove all instances of the task from the task queue. This is a no-op if the task is not scheduled. Arguments: task (callable): task to be unscheduled """ self._delayed_tasks.remove(task) def check_version(self, node_id=None, timeout=2, strict=False): """Attempt to guess a broker version Note: it is possible that this method blocks longer than the specified timeout. This can happen if the entire cluster is down and the client enters a bootstrap backoff sleep. This is only possible if node_id is None. Returns: version tuple, i.e. (0, 10), (0, 9), (0, 8, 2), ... Raises: NodeNotReadyError (if node_id is provided) NoBrokersAvailable (if node_id is None) UnrecognizedBrokerVersion: please file bug if seen! AssertionError (if strict=True): please file bug if seen! """ end = time.time() + timeout while time.time() < end: # It is possible that least_loaded_node falls back to bootstrap, # which can block for an increasing backoff period try_node = node_id or self.least_loaded_node() if try_node is None: raise Errors.NoBrokersAvailable() self._maybe_connect(try_node) conn = self._conns[try_node] # We will intentionally cause socket failures # These should not trigger metadata refresh self._refresh_on_disconnects = False try: remaining = end - time.time() version = conn.check_version(timeout=remaining, strict=strict) return version except Errors.NodeNotReadyError: # Only raise to user if this is a node-specific request if node_id is not None: raise finally: self._refresh_on_disconnects = True # Timeout else: raise Errors.NoBrokersAvailable() def wakeup(self): with self._wake_lock: if self._wake_w.send(b'x') != 1: log.warning('Unable to send to wakeup socket!') def _clear_wake_fd(self): # reading from wake socket should only happen in a single thread while True: try: self._wake_r.recv(1024) except: break class DelayedTaskQueue(object): # see https://docs.python.org/2/library/heapq.html def __init__(self): self._tasks = [] # list of entries arranged in a heap self._task_map = {} # mapping of tasks to entries self._counter = itertools.count() # unique sequence count def add(self, task, at): """Add a task to run at a later time. Arguments: task: can be anything, but generally a callable at (float or int): epoch seconds to schedule task Returns: Future: a future that will be returned with the task when ready """ if task in self._task_map: self.remove(task) count = next(self._counter) future = Future() entry = [at, count, (task, future)] self._task_map[task] = entry heapq.heappush(self._tasks, entry) return future def remove(self, task): """Remove a previously scheduled task. Raises: KeyError: if task is not found """ entry = self._task_map.pop(task) task, future = entry[-1] future.failure(Errors.Cancelled) entry[-1] = 'REMOVED' def _drop_removed(self): while self._tasks and self._tasks[0][-1] is 'REMOVED': at, count, task = heapq.heappop(self._tasks) def _pop_next(self): self._drop_removed() if not self._tasks: raise KeyError('pop from an empty DelayedTaskQueue') _, _, maybe_task = heapq.heappop(self._tasks) if maybe_task is 'REMOVED': raise ValueError('popped a removed tasks from queue - bug') else: task, future = maybe_task del self._task_map[task] return (task, future) def next_at(self): """Number of seconds until next task is ready.""" self._drop_removed() if not self._tasks: return 9999999999 else: return max(self._tasks[0][0] - time.time(), 0) def pop_ready(self): """Pop and return a list of all ready (task, future) tuples""" ready_tasks = [] while self._tasks and self._tasks[0][0] < time.time(): try: task = self._pop_next() except KeyError: break ready_tasks.append(task) return ready_tasks class KafkaClientMetrics(object): def __init__(self, metrics, metric_group_prefix, conns): self.metrics = metrics self.metric_group_name = metric_group_prefix + '-metrics' self.connection_closed = metrics.sensor('connections-closed') self.connection_closed.add(metrics.metric_name( 'connection-close-rate', self.metric_group_name, 'Connections closed per second in the window.'), Rate()) self.connection_created = metrics.sensor('connections-created') self.connection_created.add(metrics.metric_name( 'connection-creation-rate', self.metric_group_name, 'New connections established per second in the window.'), Rate()) self.select_time = metrics.sensor('select-time') self.select_time.add(metrics.metric_name( 'select-rate', self.metric_group_name, 'Number of times the I/O layer checked for new I/O to perform per' ' second'), Rate(sampled_stat=Count())) self.select_time.add(metrics.metric_name( 'io-wait-time-ns-avg', self.metric_group_name, 'The average length of time the I/O thread spent waiting for a' ' socket ready for reads or writes in nanoseconds.'), Avg()) self.select_time.add(metrics.metric_name( 'io-wait-ratio', self.metric_group_name, 'The fraction of time the I/O thread spent waiting.'), Rate(time_unit=TimeUnit.NANOSECONDS)) self.io_time = metrics.sensor('io-time') self.io_time.add(metrics.metric_name( 'io-time-ns-avg', self.metric_group_name, 'The average length of time for I/O per select call in nanoseconds.'), Avg()) self.io_time.add(metrics.metric_name( 'io-ratio', self.metric_group_name, 'The fraction of time the I/O thread spent doing I/O'), Rate(time_unit=TimeUnit.NANOSECONDS)) metrics.add_metric(metrics.metric_name( 'connection-count', self.metric_group_name, 'The current number of active connections.'), AnonMeasurable( lambda config, now: len(conns))) kafka-1.3.2/kafka/cluster.py0000644001271300127130000003052213025302127015465 0ustar dpowers00000000000000from __future__ import absolute_import import collections import copy import logging import threading import time from kafka.vendor import six from . import errors as Errors from .future import Future from .structs import BrokerMetadata, PartitionMetadata, TopicPartition log = logging.getLogger(__name__) class ClusterMetadata(object): DEFAULT_CONFIG = { 'retry_backoff_ms': 100, 'metadata_max_age_ms': 300000, } def __init__(self, **configs): self._brokers = {} # node_id -> BrokerMetadata self._partitions = {} # topic -> partition -> PartitionMetadata self._broker_partitions = collections.defaultdict(set) # node_id -> {TopicPartition...} self._groups = {} # group_name -> node_id self._last_refresh_ms = 0 self._last_successful_refresh_ms = 0 self._need_update = False self._future = None self._listeners = set() self._lock = threading.Lock() self.need_all_topic_metadata = False self.unauthorized_topics = set() self.internal_topics = set() self.controller = None self.config = copy.copy(self.DEFAULT_CONFIG) for key in self.config: if key in configs: self.config[key] = configs[key] def brokers(self): """Get all BrokerMetadata Returns: set: {BrokerMetadata, ...} """ return set(self._brokers.values()) def broker_metadata(self, broker_id): """Get BrokerMetadata Arguments: broker_id (int): node_id for a broker to check Returns: BrokerMetadata or None if not found """ return self._brokers.get(broker_id) def partitions_for_topic(self, topic): """Return set of all partitions for topic (whether available or not) Arguments: topic (str): topic to check for partitions Returns: set: {partition (int), ...} """ if topic not in self._partitions: return None return set(self._partitions[topic].keys()) def available_partitions_for_topic(self, topic): """Return set of partitions with known leaders Arguments: topic (str): topic to check for partitions Returns: set: {partition (int), ...} """ if topic not in self._partitions: return None return set([partition for partition, metadata in six.iteritems(self._partitions[topic]) if metadata.leader != -1]) def leader_for_partition(self, partition): """Return node_id of leader, -1 unavailable, None if unknown.""" if partition.topic not in self._partitions: return None elif partition.partition not in self._partitions[partition.topic]: return None return self._partitions[partition.topic][partition.partition].leader def partitions_for_broker(self, broker_id): """Return TopicPartitions for which the broker is a leader. Arguments: broker_id (int): node id for a broker Returns: set: {TopicPartition, ...} """ return self._broker_partitions.get(broker_id) def coordinator_for_group(self, group): """Return node_id of group coordinator. Arguments: group (str): name of consumer group Returns: int: node_id for group coordinator """ return self._groups.get(group) def ttl(self): """Milliseconds until metadata should be refreshed""" now = time.time() * 1000 if self._need_update: ttl = 0 else: metadata_age = now - self._last_successful_refresh_ms ttl = self.config['metadata_max_age_ms'] - metadata_age retry_age = now - self._last_refresh_ms next_retry = self.config['retry_backoff_ms'] - retry_age return max(ttl, next_retry, 0) def refresh_backoff(self): """Return milliseconds to wait before attempting to retry after failure""" return self.config['retry_backoff_ms'] def request_update(self): """Flags metadata for update, return Future() Actual update must be handled separately. This method will only change the reported ttl() Returns: kafka.future.Future (value will be the cluster object after update) """ with self._lock: self._need_update = True if not self._future or self._future.is_done: self._future = Future() return self._future def topics(self, exclude_internal_topics=True): """Get set of known topics. Arguments: exclude_internal_topics (bool): Whether records from internal topics (such as offsets) should be exposed to the consumer. If set to True the only way to receive records from an internal topic is subscribing to it. Default True Returns: set: {topic (str), ...} """ topics = set(self._partitions.keys()) if exclude_internal_topics: return topics - self.internal_topics else: return topics def failed_update(self, exception): """Update cluster state given a failed MetadataRequest.""" f = None with self._lock: if self._future: f = self._future self._future = None if f: f.failure(exception) self._last_refresh_ms = time.time() * 1000 def update_metadata(self, metadata): """Update cluster state given a MetadataResponse. Arguments: metadata (MetadataResponse): broker response to a metadata request Returns: None """ # In the common case where we ask for a single topic and get back an # error, we should fail the future if len(metadata.topics) == 1 and metadata.topics[0][0] != 0: error_code, topic = metadata.topics[0][:2] error = Errors.for_code(error_code)(topic) return self.failed_update(error) if not metadata.brokers: log.warning("No broker metadata found in MetadataResponse") for broker in metadata.brokers: if metadata.API_VERSION == 0: node_id, host, port = broker rack = None else: node_id, host, port, rack = broker self._brokers.update({ node_id: BrokerMetadata(node_id, host, port, rack) }) if metadata.API_VERSION == 0: self.controller = None else: self.controller = self._brokers.get(metadata.controller_id) _new_partitions = {} _new_broker_partitions = collections.defaultdict(set) _new_unauthorized_topics = set() _new_internal_topics = set() for topic_data in metadata.topics: if metadata.API_VERSION == 0: error_code, topic, partitions = topic_data is_internal = False else: error_code, topic, is_internal, partitions = topic_data if is_internal: _new_internal_topics.add(topic) error_type = Errors.for_code(error_code) if error_type is Errors.NoError: _new_partitions[topic] = {} for p_error, partition, leader, replicas, isr in partitions: _new_partitions[topic][partition] = PartitionMetadata( topic=topic, partition=partition, leader=leader, replicas=replicas, isr=isr, error=p_error) if leader != -1: _new_broker_partitions[leader].add( TopicPartition(topic, partition)) elif error_type is Errors.LeaderNotAvailableError: log.warning("Topic %s is not available during auto-create" " initialization", topic) elif error_type is Errors.UnknownTopicOrPartitionError: log.error("Topic %s not found in cluster metadata", topic) elif error_type is Errors.TopicAuthorizationFailedError: log.error("Topic %s is not authorized for this client", topic) _new_unauthorized_topics.add(topic) elif error_type is Errors.InvalidTopicError: log.error("'%s' is not a valid topic name", topic) else: log.error("Error fetching metadata for topic %s: %s", topic, error_type) with self._lock: self._partitions = _new_partitions self._broker_partitions = _new_broker_partitions self.unauthorized_topics = _new_unauthorized_topics self.internal_topics = _new_internal_topics f = None if self._future: f = self._future self._future = None self._need_update = False now = time.time() * 1000 self._last_refresh_ms = now self._last_successful_refresh_ms = now if f: f.success(self) log.debug("Updated cluster metadata to %s", self) for listener in self._listeners: listener(self) def add_listener(self, listener): """Add a callback function to be called on each metadata update""" self._listeners.add(listener) def remove_listener(self, listener): """Remove a previously added listener callback""" self._listeners.remove(listener) def add_group_coordinator(self, group, response): """Update with metadata for a group coordinator Arguments: group (str): name of group from GroupCoordinatorRequest response (GroupCoordinatorResponse): broker response Returns: bool: True if metadata is updated, False on error """ log.debug("Updating coordinator for %s: %s", group, response) error_type = Errors.for_code(response.error_code) if error_type is not Errors.NoError: log.error("GroupCoordinatorResponse error: %s", error_type) self._groups[group] = -1 return False node_id = response.coordinator_id coordinator = BrokerMetadata( response.coordinator_id, response.host, response.port, None) # Assume that group coordinators are just brokers # (this is true now, but could diverge in future) if node_id not in self._brokers: self._brokers[node_id] = coordinator # If this happens, either brokers have moved without # changing IDs, or our assumption above is wrong else: node = self._brokers[node_id] if coordinator.host != node.host or coordinator.port != node.port: log.error("GroupCoordinator metadata conflicts with existing" " broker metadata. Coordinator: %s, Broker: %s", coordinator, node) self._groups[group] = node_id return False log.info("Group coordinator for %s is %s", group, coordinator) self._groups[group] = node_id return True def with_partitions(self, partitions_to_add): """Returns a copy of cluster metadata with partitions added""" new_metadata = ClusterMetadata(**self.config) new_metadata._brokers = copy.deepcopy(self._brokers) new_metadata._partitions = copy.deepcopy(self._partitions) new_metadata._broker_partitions = copy.deepcopy(self._broker_partitions) new_metadata._groups = copy.deepcopy(self._groups) new_metadata.internal_topics = copy.deepcopy(self.internal_topics) new_metadata.unauthorized_topics = copy.deepcopy(self.unauthorized_topics) for partition in partitions_to_add: new_metadata._partitions[partition.topic][partition.partition] = partition if partition.leader is not None and partition.leader != -1: new_metadata._broker_partitions[partition.leader].add( TopicPartition(partition.topic, partition.partition)) return new_metadata def __str__(self): return 'ClusterMetadata(brokers: %d, topics: %d, groups: %d)' % \ (len(self._brokers), len(self._partitions), len(self._groups)) kafka-1.3.2/kafka/codec.py0000644001271300127130000001662413025302127015070 0ustar dpowers00000000000000from __future__ import absolute_import import gzip import io import platform import struct from kafka.vendor import six from kafka.vendor.six.moves import xrange # pylint: disable=import-error _XERIAL_V1_HEADER = (-126, b'S', b'N', b'A', b'P', b'P', b'Y', 0, 1, 1) _XERIAL_V1_FORMAT = 'bccccccBii' try: import snappy except ImportError: snappy = None try: import lz4f import xxhash except ImportError: lz4f = None PYPY = bool(platform.python_implementation() == 'PyPy') def has_gzip(): return True def has_snappy(): return snappy is not None def has_lz4(): return lz4f is not None def gzip_encode(payload, compresslevel=None): if not compresslevel: compresslevel = 9 buf = io.BytesIO() # Gzip context manager introduced in python 2.7 # so old-fashioned way until we decide to not support 2.6 gzipper = gzip.GzipFile(fileobj=buf, mode="w", compresslevel=compresslevel) try: gzipper.write(payload) finally: gzipper.close() return buf.getvalue() def gzip_decode(payload): buf = io.BytesIO(payload) # Gzip context manager introduced in python 2.7 # so old-fashioned way until we decide to not support 2.6 gzipper = gzip.GzipFile(fileobj=buf, mode='r') try: return gzipper.read() finally: gzipper.close() def snappy_encode(payload, xerial_compatible=True, xerial_blocksize=32*1024): """Encodes the given data with snappy compression. If xerial_compatible is set then the stream is encoded in a fashion compatible with the xerial snappy library. The block size (xerial_blocksize) controls how frequent the blocking occurs 32k is the default in the xerial library. The format winds up being: +-------------+------------+--------------+------------+--------------+ | Header | Block1 len | Block1 data | Blockn len | Blockn data | +-------------+------------+--------------+------------+--------------+ | 16 bytes | BE int32 | snappy bytes | BE int32 | snappy bytes | +-------------+------------+--------------+------------+--------------+ It is important to note that the blocksize is the amount of uncompressed data presented to snappy at each block, whereas the blocklen is the number of bytes that will be present in the stream; so the length will always be <= blocksize. """ if not has_snappy(): raise NotImplementedError("Snappy codec is not available") if not xerial_compatible: return snappy.compress(payload) out = io.BytesIO() for fmt, dat in zip(_XERIAL_V1_FORMAT, _XERIAL_V1_HEADER): out.write(struct.pack('!' + fmt, dat)) # Chunk through buffers to avoid creating intermediate slice copies if PYPY: # on pypy, snappy.compress() on a sliced buffer consumes the entire # buffer... likely a python-snappy bug, so just use a slice copy chunker = lambda payload, i, size: payload[i:size+i] elif six.PY2: # Sliced buffer avoids additional copies # pylint: disable-msg=undefined-variable chunker = lambda payload, i, size: buffer(payload, i, size) else: # snappy.compress does not like raw memoryviews, so we have to convert # tobytes, which is a copy... oh well. it's the thought that counts. # pylint: disable-msg=undefined-variable chunker = lambda payload, i, size: memoryview(payload)[i:size+i].tobytes() for chunk in (chunker(payload, i, xerial_blocksize) for i in xrange(0, len(payload), xerial_blocksize)): block = snappy.compress(chunk) block_size = len(block) out.write(struct.pack('!i', block_size)) out.write(block) return out.getvalue() def _detect_xerial_stream(payload): """Detects if the data given might have been encoded with the blocking mode of the xerial snappy library. This mode writes a magic header of the format: +--------+--------------+------------+---------+--------+ | Marker | Magic String | Null / Pad | Version | Compat | +--------+--------------+------------+---------+--------+ | byte | c-string | byte | int32 | int32 | +--------+--------------+------------+---------+--------+ | -126 | 'SNAPPY' | \0 | | | +--------+--------------+------------+---------+--------+ The pad appears to be to ensure that SNAPPY is a valid cstring The version is the version of this format as written by xerial, in the wild this is currently 1 as such we only support v1. Compat is there to claim the miniumum supported version that can read a xerial block stream, presently in the wild this is 1. """ if len(payload) > 16: header = struct.unpack('!' + _XERIAL_V1_FORMAT, bytes(payload)[:16]) return header == _XERIAL_V1_HEADER return False def snappy_decode(payload): if not has_snappy(): raise NotImplementedError("Snappy codec is not available") if _detect_xerial_stream(payload): # TODO ? Should become a fileobj ? out = io.BytesIO() byt = payload[16:] length = len(byt) cursor = 0 while cursor < length: block_size = struct.unpack_from('!i', byt[cursor:])[0] # Skip the block size cursor += 4 end = cursor + block_size out.write(snappy.decompress(byt[cursor:end])) cursor = end out.seek(0) return out.read() else: return snappy.decompress(payload) def lz4_encode(payload): """Encode payload using interoperable LZ4 framing. Requires Kafka >= 0.10""" # pylint: disable-msg=no-member return lz4f.compressFrame(payload) def lz4_decode(payload): """Decode payload using interoperable LZ4 framing. Requires Kafka >= 0.10""" # pylint: disable-msg=no-member ctx = lz4f.createDecompContext() data = lz4f.decompressFrame(payload, ctx) # lz4f python module does not expose how much of the payload was # actually read if the decompression was only partial. if data['next'] != 0: raise RuntimeError('lz4f unable to decompress full payload') return data['decomp'] def lz4_encode_old_kafka(payload): """Encode payload for 0.8/0.9 brokers -- requires an incorrect header checksum.""" data = lz4_encode(payload) header_size = 7 if isinstance(data[4], int): flg = data[4] else: flg = ord(data[4]) content_size_bit = ((flg >> 3) & 1) if content_size_bit: header_size += 8 # This is the incorrect hc hc = xxhash.xxh32(data[0:header_size-1]).digest()[-2:-1] # pylint: disable-msg=no-member return b''.join([ data[0:header_size-1], hc, data[header_size:] ]) def lz4_decode_old_kafka(payload): # Kafka's LZ4 code has a bug in its header checksum implementation header_size = 7 if isinstance(payload[4], int): flg = payload[4] else: flg = ord(payload[4]) content_size_bit = ((flg >> 3) & 1) if content_size_bit: header_size += 8 # This should be the correct hc hc = xxhash.xxh32(payload[4:header_size-1]).digest()[-2:-1] # pylint: disable-msg=no-member munged_payload = b''.join([ payload[0:header_size-1], hc, payload[header_size:] ]) return lz4_decode(munged_payload) kafka-1.3.2/kafka/common.py0000644001271300127130000000013713025302127015273 0ustar dpowers00000000000000from __future__ import absolute_import from kafka.structs import * from kafka.errors import * kafka-1.3.2/kafka/conn.py0000644001271300127130000013351613031057471014756 0ustar dpowers00000000000000from __future__ import absolute_import import collections import copy import errno import logging import io from random import shuffle import socket import ssl import time from kafka.vendor import six import kafka.errors as Errors from kafka.future import Future from kafka.metrics.stats import Avg, Count, Max, Rate from kafka.protocol.api import RequestHeader from kafka.protocol.admin import SaslHandShakeRequest from kafka.protocol.commit import GroupCoordinatorResponse from kafka.protocol.types import Int32 from kafka.version import __version__ if six.PY2: ConnectionError = socket.error BlockingIOError = Exception log = logging.getLogger(__name__) DEFAULT_KAFKA_PORT = 9092 # support older ssl libraries try: ssl.SSLWantReadError ssl.SSLWantWriteError ssl.SSLZeroReturnError except: log.warning('old ssl module detected.' ' ssl error handling may not operate cleanly.' ' Consider upgrading to python 3.5 or 2.7') ssl.SSLWantReadError = ssl.SSLError ssl.SSLWantWriteError = ssl.SSLError ssl.SSLZeroReturnError = ssl.SSLError class ConnectionStates(object): DISCONNECTING = '' DISCONNECTED = '' CONNECTING = '' HANDSHAKE = '' CONNECTED = '' AUTHENTICATING = '' InFlightRequest = collections.namedtuple('InFlightRequest', ['request', 'response_type', 'correlation_id', 'future', 'timestamp']) class BrokerConnection(object): DEFAULT_CONFIG = { 'client_id': 'kafka-python-' + __version__, 'node_id': 0, 'request_timeout_ms': 40000, 'reconnect_backoff_ms': 50, 'max_in_flight_requests_per_connection': 5, 'receive_buffer_bytes': None, 'send_buffer_bytes': None, 'socket_options': [(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)], 'security_protocol': 'PLAINTEXT', 'ssl_context': None, 'ssl_check_hostname': True, 'ssl_cafile': None, 'ssl_certfile': None, 'ssl_keyfile': None, 'ssl_crlfile': None, 'ssl_password': None, 'api_version': (0, 8, 2), # default to most restrictive 'state_change_callback': lambda conn: True, 'metrics': None, 'metric_group_prefix': '', 'sasl_mechanism': 'PLAIN', 'sasl_plain_username': None, 'sasl_plain_password': None } SASL_MECHANISMS = ('PLAIN',) def __init__(self, host, port, afi, **configs): """Initialize a kafka broker connection Keyword Arguments: client_id (str): a name for this client. This string is passed in each request to servers and can be used to identify specific server-side log entries that correspond to this client. Also submitted to GroupCoordinator for logging with respect to consumer group administration. Default: 'kafka-python-{version}' reconnect_backoff_ms (int): The amount of time in milliseconds to wait before attempting to reconnect to a given host. Default: 50. request_timeout_ms (int): Client request timeout in milliseconds. Default: 40000. max_in_flight_requests_per_connection (int): Requests are pipelined to kafka brokers up to this number of maximum requests per broker connection. Default: 5. receive_buffer_bytes (int): The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. Default: None (relies on system defaults). Java client defaults to 32768. send_buffer_bytes (int): The size of the TCP send buffer (SO_SNDBUF) to use when sending data. Default: None (relies on system defaults). Java client defaults to 131072. socket_options (list): List of tuple-arguments to socket.setsockopt to apply to broker connection sockets. Default: [(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)] security_protocol (str): Protocol used to communicate with brokers. Valid values are: PLAINTEXT, SSL. Default: PLAINTEXT. ssl_context (ssl.SSLContext): pre-configured SSLContext for wrapping socket connections. If provided, all other ssl_* configurations will be ignored. Default: None. ssl_check_hostname (bool): flag to configure whether ssl handshake should verify that the certificate matches the brokers hostname. default: True. ssl_cafile (str): optional filename of ca file to use in certificate veriication. default: None. ssl_certfile (str): optional filename of file in pem format containing the client certificate, as well as any ca certificates needed to establish the certificate's authenticity. default: None. ssl_keyfile (str): optional filename containing the client private key. default: None. ssl_password (callable, str, bytes, bytearray): optional password or callable function that returns a password, for decrypting the client private key. Default: None. ssl_crlfile (str): optional filename containing the CRL to check for certificate expiration. By default, no CRL check is done. When providing a file, only the leaf certificate will be checked against this CRL. The CRL can only be checked with Python 3.4+ or 2.7.9+. default: None. api_version (tuple): specify which kafka API version to use. Accepted values are: (0, 8, 0), (0, 8, 1), (0, 8, 2), (0, 9), (0, 10) If None, KafkaClient will attempt to infer the broker version by probing various APIs. Default: None api_version_auto_timeout_ms (int): number of milliseconds to throw a timeout exception from the constructor when checking the broker api version. Only applies if api_version is None state_change_callback (callable): function to be called when the connection state changes from CONNECTING to CONNECTED etc. metrics (kafka.metrics.Metrics): Optionally provide a metrics instance for capturing network IO stats. Default: None. metric_group_prefix (str): Prefix for metric names. Default: '' sasl_mechanism (str): string picking sasl mechanism when security_protocol is SASL_PLAINTEXT or SASL_SSL. Currently only PLAIN is supported. Default: None sasl_plain_username (str): username for sasl PLAIN authentication. Default: None sasl_plain_password (str): password for sasl PLAIN authentication. Default: None """ self.host = host self.hostname = host self.port = port self.afi = afi self._init_host = host self._init_port = port self._init_afi = afi self.in_flight_requests = collections.deque() self.config = copy.copy(self.DEFAULT_CONFIG) for key in self.config: if key in configs: self.config[key] = configs[key] if self.config['receive_buffer_bytes'] is not None: self.config['socket_options'].append( (socket.SOL_SOCKET, socket.SO_RCVBUF, self.config['receive_buffer_bytes'])) if self.config['send_buffer_bytes'] is not None: self.config['socket_options'].append( (socket.SOL_SOCKET, socket.SO_SNDBUF, self.config['send_buffer_bytes'])) if self.config['security_protocol'] in ('SASL_PLAINTEXT', 'SASL_SSL'): assert self.config['sasl_mechanism'] in self.SASL_MECHANISMS, ( 'sasl_mechanism must be in ' + ', '.join(self.SASL_MECHANISMS)) if self.config['sasl_mechanism'] == 'PLAIN': assert self.config['sasl_plain_username'] is not None, 'sasl_plain_username required for PLAIN sasl' assert self.config['sasl_plain_password'] is not None, 'sasl_plain_password required for PLAIN sasl' self.state = ConnectionStates.DISCONNECTED self._sock = None self._ssl_context = None if self.config['ssl_context'] is not None: self._ssl_context = self.config['ssl_context'] self._sasl_auth_future = None self._rbuffer = io.BytesIO() self._receiving = False self._next_payload_bytes = 0 self.last_attempt = 0 self.last_failure = 0 self._processing = False self._correlation_id = 0 self._gai = None self._gai_index = 0 self._sensors = None if self.config['metrics']: self._sensors = BrokerConnectionMetrics(self.config['metrics'], self.config['metric_group_prefix'], self.config['node_id']) def connect(self): """Attempt to connect and return ConnectionState""" if self.state is ConnectionStates.DISCONNECTED: self.close() log.debug('%s: creating new socket', str(self)) # if self.afi is set to AF_UNSPEC, then we need to do a name # resolution and try all available address families if self._init_afi == socket.AF_UNSPEC: if self._gai is None: # XXX: all DNS functions in Python are blocking. If we really # want to be non-blocking here, we need to use a 3rd-party # library like python-adns, or move resolution onto its # own thread. This will be subject to the default libc # name resolution timeout (5s on most Linux boxes) try: self._gai = socket.getaddrinfo(self._init_host, self._init_port, socket.AF_UNSPEC, socket.SOCK_STREAM) except socket.gaierror as ex: raise socket.gaierror('getaddrinfo failed for {0}:{1}, ' 'exception was {2}. Is your advertised.listeners (called' 'advertised.host.name before Kafka 9) correct and resolvable?'.format( self._init_host, self._init_port, ex )) self._gai_index = 0 else: # if self._gai already exists, then we should try the next # name self._gai_index += 1 while True: if self._gai_index >= len(self._gai): log.error('Unable to connect to any of the names for {0}:{1}'.format( self._init_host, self._init_port )) self.close() return afi, _, __, ___, sockaddr = self._gai[self._gai_index] if afi not in (socket.AF_INET, socket.AF_INET6): self._gai_index += 1 continue break self.host, self.port = sockaddr[:2] self._sock = socket.socket(afi, socket.SOCK_STREAM) else: self._sock = socket.socket(self._init_afi, socket.SOCK_STREAM) for option in self.config['socket_options']: self._sock.setsockopt(*option) self._sock.setblocking(False) if self.config['security_protocol'] in ('SSL', 'SASL_SSL'): self._wrap_ssl() self.state = ConnectionStates.CONNECTING self.last_attempt = time.time() self.config['state_change_callback'](self) if self.state is ConnectionStates.CONNECTING: # in non-blocking mode, use repeated calls to socket.connect_ex # to check connection status request_timeout = self.config['request_timeout_ms'] / 1000.0 ret = None try: ret = self._sock.connect_ex((self.host, self.port)) # if we got here through a host lookup, we've found a host,port,af tuple # that works save it so we don't do a GAI lookup again if self._gai is not None: self.afi = self._sock.family self._gai = None except socket.error as err: ret = err.errno # Connection succeeded if not ret or ret == errno.EISCONN: log.debug('%s: established TCP connection', str(self)) if self.config['security_protocol'] in ('SSL', 'SASL_SSL'): log.debug('%s: initiating SSL handshake', str(self)) self.state = ConnectionStates.HANDSHAKE elif self.config['security_protocol'] == 'SASL_PLAINTEXT': self.state = ConnectionStates.AUTHENTICATING else: self.state = ConnectionStates.CONNECTED self.config['state_change_callback'](self) # Connection failed # WSAEINVAL == 10022, but errno.WSAEINVAL is not available on non-win systems elif ret not in (errno.EINPROGRESS, errno.EALREADY, errno.EWOULDBLOCK, 10022): log.error('Connect attempt to %s returned error %s.' ' Disconnecting.', self, ret) self.close() # Connection timed out elif time.time() > request_timeout + self.last_attempt: log.error('Connection attempt to %s timed out', self) self.close() # error=TimeoutError ? # Needs retry else: pass if self.state is ConnectionStates.HANDSHAKE: if self._try_handshake(): log.debug('%s: completed SSL handshake.', str(self)) if self.config['security_protocol'] == 'SASL_SSL': self.state = ConnectionStates.AUTHENTICATING else: self.state = ConnectionStates.CONNECTED self.config['state_change_callback'](self) if self.state is ConnectionStates.AUTHENTICATING: assert self.config['security_protocol'] in ('SASL_PLAINTEXT', 'SASL_SSL') if self._try_authenticate(): log.info('%s: Authenticated as %s', str(self), self.config['sasl_plain_username']) self.state = ConnectionStates.CONNECTED self.config['state_change_callback'](self) return self.state def _wrap_ssl(self): assert self.config['security_protocol'] in ('SSL', 'SASL_SSL') if self._ssl_context is None: log.debug('%s: configuring default SSL Context', str(self)) self._ssl_context = ssl.SSLContext(ssl.PROTOCOL_SSLv23) # pylint: disable=no-member self._ssl_context.options |= ssl.OP_NO_SSLv2 # pylint: disable=no-member self._ssl_context.options |= ssl.OP_NO_SSLv3 # pylint: disable=no-member self._ssl_context.verify_mode = ssl.CERT_OPTIONAL if self.config['ssl_check_hostname']: self._ssl_context.check_hostname = True if self.config['ssl_cafile']: log.info('%s: Loading SSL CA from %s', str(self), self.config['ssl_cafile']) self._ssl_context.load_verify_locations(self.config['ssl_cafile']) self._ssl_context.verify_mode = ssl.CERT_REQUIRED if self.config['ssl_certfile'] and self.config['ssl_keyfile']: log.info('%s: Loading SSL Cert from %s', str(self), self.config['ssl_certfile']) log.info('%s: Loading SSL Key from %s', str(self), self.config['ssl_keyfile']) self._ssl_context.load_cert_chain( certfile=self.config['ssl_certfile'], keyfile=self.config['ssl_keyfile'], password=self.config['ssl_password']) if self.config['ssl_crlfile']: if not hasattr(ssl, 'VERIFY_CRL_CHECK_LEAF'): log.error('%s: No CRL support with this version of Python.' ' Disconnecting.', self) self.close() return log.info('%s: Loading SSL CRL from %s', str(self), self.config['ssl_crlfile']) self._ssl_context.load_verify_locations(self.config['ssl_crlfile']) # pylint: disable=no-member self._ssl_context.verify_flags |= ssl.VERIFY_CRL_CHECK_LEAF log.debug('%s: wrapping socket in ssl context', str(self)) try: self._sock = self._ssl_context.wrap_socket( self._sock, server_hostname=self.hostname, do_handshake_on_connect=False) except ssl.SSLError: log.exception('%s: Failed to wrap socket in SSLContext!', str(self)) self.close() self.last_failure = time.time() def _try_handshake(self): assert self.config['security_protocol'] in ('SSL', 'SASL_SSL') try: self._sock.do_handshake() return True # old ssl in python2.6 will swallow all SSLErrors here... except (ssl.SSLWantReadError, ssl.SSLWantWriteError): pass except ssl.SSLZeroReturnError: log.warning('SSL connection closed by server during handshake.') self.close() # Other SSLErrors will be raised to user return False def _try_authenticate(self): assert self.config['api_version'] is None or self.config['api_version'] >= (0, 10) if self._sasl_auth_future is None: # Build a SaslHandShakeRequest message request = SaslHandShakeRequest[0](self.config['sasl_mechanism']) future = Future() sasl_response = self._send(request) sasl_response.add_callback(self._handle_sasl_handshake_response, future) sasl_response.add_errback(lambda f, e: f.failure(e), future) self._sasl_auth_future = future self._recv() if self._sasl_auth_future.failed(): raise self._sasl_auth_future.exception # pylint: disable-msg=raising-bad-type return self._sasl_auth_future.succeeded() def _handle_sasl_handshake_response(self, future, response): error_type = Errors.for_code(response.error_code) if error_type is not Errors.NoError: error = error_type(self) self.close(error=error) return future.failure(error_type(self)) if self.config['sasl_mechanism'] == 'PLAIN': return self._try_authenticate_plain(future) else: return future.failure( Errors.UnsupportedSaslMechanismError( 'kafka-python does not support SASL mechanism %s' % self.config['sasl_mechanism'])) def _try_authenticate_plain(self, future): if self.config['security_protocol'] == 'SASL_PLAINTEXT': log.warning('%s: Sending username and password in the clear', str(self)) data = b'' try: self._sock.setblocking(True) # Send PLAIN credentials per RFC-4616 msg = bytes('\0'.join([self.config['sasl_plain_username'], self.config['sasl_plain_username'], self.config['sasl_plain_password']]).encode('utf-8')) size = Int32.encode(len(msg)) self._sock.sendall(size + msg) # The server will send a zero sized message (that is Int32(0)) on success. # The connection is closed on failure while len(data) < 4: fragment = self._sock.recv(4 - len(data)) if not fragment: log.error('%s: Authentication failed for user %s', self, self.config['sasl_plain_username']) error = Errors.AuthenticationFailedError( 'Authentication failed for user {0}'.format( self.config['sasl_plain_username'])) future.failure(error) raise error data += fragment self._sock.setblocking(False) except (AssertionError, ConnectionError) as e: log.exception("%s: Error receiving reply from server", self) error = Errors.ConnectionError("%s: %s" % (str(self), e)) future.failure(error) self.close(error=error) if data != b'\x00\x00\x00\x00': return future.failure(Errors.AuthenticationFailedError()) return future.success(True) def blacked_out(self): """ Return true if we are disconnected from the given node and can't re-establish a connection yet """ if self.state is ConnectionStates.DISCONNECTED: backoff = self.config['reconnect_backoff_ms'] / 1000.0 if time.time() < self.last_attempt + backoff: return True return False def connected(self): """Return True iff socket is connected.""" return self.state is ConnectionStates.CONNECTED def connecting(self): """Returns True if still connecting (this may encompass several different states, such as SSL handshake, authorization, etc).""" return self.state in (ConnectionStates.CONNECTING, ConnectionStates.HANDSHAKE, ConnectionStates.AUTHENTICATING) def disconnected(self): """Return True iff socket is closed""" return self.state is ConnectionStates.DISCONNECTED def close(self, error=None): """Close socket and fail all in-flight-requests. Arguments: error (Exception, optional): pending in-flight-requests will be failed with this exception. Default: kafka.errors.ConnectionError. """ if self.state is not ConnectionStates.DISCONNECTED: self.state = ConnectionStates.DISCONNECTING self.config['state_change_callback'](self) if self._sock: self._sock.close() self._sock = None self.state = ConnectionStates.DISCONNECTED self.last_failure = time.time() self._receiving = False self._next_payload_bytes = 0 self._rbuffer.seek(0) self._rbuffer.truncate() if error is None: error = Errors.ConnectionError(str(self)) while self.in_flight_requests: ifr = self.in_flight_requests.popleft() ifr.future.failure(error) self.config['state_change_callback'](self) def send(self, request, expect_response=True): """send request, return Future() Can block on network if request is larger than send_buffer_bytes """ future = Future() if self.connecting(): return future.failure(Errors.NodeNotReadyError(str(self))) elif not self.connected(): return future.failure(Errors.ConnectionError(str(self))) elif not self.can_send_more(): return future.failure(Errors.TooManyInFlightRequests(str(self))) return self._send(request, expect_response=expect_response) def _send(self, request, expect_response=True): future = Future() correlation_id = self._next_correlation_id() header = RequestHeader(request, correlation_id=correlation_id, client_id=self.config['client_id']) message = b''.join([header.encode(), request.encode()]) size = Int32.encode(len(message)) data = size + message try: # In the future we might manage an internal write buffer # and send bytes asynchronously. For now, just block # sending each request payload self._sock.setblocking(True) total_sent = 0 while total_sent < len(data): sent_bytes = self._sock.send(data[total_sent:]) total_sent += sent_bytes assert total_sent == len(data) if self._sensors: self._sensors.bytes_sent.record(total_sent) self._sock.setblocking(False) except (AssertionError, ConnectionError) as e: log.exception("Error sending %s to %s", request, self) error = Errors.ConnectionError("%s: %s" % (str(self), e)) self.close(error=error) return future.failure(error) log.debug('%s Request %d: %s', self, correlation_id, request) if expect_response: ifr = InFlightRequest(request=request, correlation_id=correlation_id, response_type=request.RESPONSE_TYPE, future=future, timestamp=time.time()) self.in_flight_requests.append(ifr) else: future.success(None) return future def can_send_more(self): """Return True unless there are max_in_flight_requests_per_connection.""" max_ifrs = self.config['max_in_flight_requests_per_connection'] return len(self.in_flight_requests) < max_ifrs def recv(self): """Non-blocking network receive. Return response if available """ assert not self._processing, 'Recursion not supported' if not self.connected() and not self.state is ConnectionStates.AUTHENTICATING: log.warning('%s cannot recv: socket not connected', self) # If requests are pending, we should close the socket and # fail all the pending request futures if self.in_flight_requests: self.close() return None elif not self.in_flight_requests: log.warning('%s: No in-flight-requests to recv', self) return None response = self._recv() if not response and self.requests_timed_out(): log.warning('%s timed out after %s ms. Closing connection.', self, self.config['request_timeout_ms']) self.close(error=Errors.RequestTimedOutError( 'Request timed out after %s ms' % self.config['request_timeout_ms'])) return None return response def _recv(self): # Not receiving is the state of reading the payload header if not self._receiving: try: bytes_to_read = 4 - self._rbuffer.tell() data = self._sock.recv(bytes_to_read) # We expect socket.recv to raise an exception if there is not # enough data to read the full bytes_to_read # but if the socket is disconnected, we will get empty data # without an exception raised if not data: log.error('%s: socket disconnected', self) self.close(error=Errors.ConnectionError('socket disconnected')) return None self._rbuffer.write(data) except ssl.SSLWantReadError: return None except ConnectionError as e: if six.PY2 and e.errno == errno.EWOULDBLOCK: return None log.exception('%s: Error receiving 4-byte payload header -' ' closing socket', self) self.close(error=Errors.ConnectionError(e)) return None except BlockingIOError: if six.PY3: return None raise if self._rbuffer.tell() == 4: self._rbuffer.seek(0) self._next_payload_bytes = Int32.decode(self._rbuffer) # reset buffer and switch state to receiving payload bytes self._rbuffer.seek(0) self._rbuffer.truncate() self._receiving = True elif self._rbuffer.tell() > 4: raise Errors.KafkaError('this should not happen - are you threading?') if self._receiving: staged_bytes = self._rbuffer.tell() try: bytes_to_read = self._next_payload_bytes - staged_bytes data = self._sock.recv(bytes_to_read) # We expect socket.recv to raise an exception if there is not # enough data to read the full bytes_to_read # but if the socket is disconnected, we will get empty data # without an exception raised if bytes_to_read and not data: log.error('%s: socket disconnected', self) self.close(error=Errors.ConnectionError('socket disconnected')) return None self._rbuffer.write(data) except ssl.SSLWantReadError: return None except ConnectionError as e: # Extremely small chance that we have exactly 4 bytes for a # header, but nothing to read in the body yet if six.PY2 and e.errno == errno.EWOULDBLOCK: return None log.exception('%s: Error in recv', self) self.close(error=Errors.ConnectionError(e)) return None except BlockingIOError: if six.PY3: return None raise staged_bytes = self._rbuffer.tell() if staged_bytes > self._next_payload_bytes: self.close(error=Errors.KafkaError('Receive buffer has more bytes than expected?')) if staged_bytes != self._next_payload_bytes: return None self._receiving = False self._next_payload_bytes = 0 if self._sensors: self._sensors.bytes_received.record(4 + self._rbuffer.tell()) self._rbuffer.seek(0) response = self._process_response(self._rbuffer) self._rbuffer.seek(0) self._rbuffer.truncate() return response def _process_response(self, read_buffer): assert not self._processing, 'Recursion not supported' self._processing = True ifr = self.in_flight_requests.popleft() if self._sensors: self._sensors.request_time.record((time.time() - ifr.timestamp) * 1000) # verify send/recv correlation ids match recv_correlation_id = Int32.decode(read_buffer) # 0.8.2 quirk if (self.config['api_version'] == (0, 8, 2) and ifr.response_type is GroupCoordinatorResponse[0] and ifr.correlation_id != 0 and recv_correlation_id == 0): log.warning('Kafka 0.8.2 quirk -- GroupCoordinatorResponse' ' coorelation id does not match request. This' ' should go away once at least one topic has been' ' initialized on the broker') elif ifr.correlation_id != recv_correlation_id: error = Errors.CorrelationIdError( '%s: Correlation ids do not match: sent %d, recv %d' % (str(self), ifr.correlation_id, recv_correlation_id)) ifr.future.failure(error) self.close() self._processing = False return None # decode response try: response = ifr.response_type.decode(read_buffer) except ValueError: read_buffer.seek(0) buf = read_buffer.read() log.error('%s Response %d [ResponseType: %s Request: %s]:' ' Unable to decode %d-byte buffer: %r', self, ifr.correlation_id, ifr.response_type, ifr.request, len(buf), buf) ifr.future.failure(Errors.UnknownError('Unable to decode response')) self.close() self._processing = False return None log.debug('%s Response %d: %s', self, ifr.correlation_id, response) ifr.future.success(response) self._processing = False return response def requests_timed_out(self): if self.in_flight_requests: oldest_at = self.in_flight_requests[0].timestamp timeout = self.config['request_timeout_ms'] / 1000.0 if time.time() >= oldest_at + timeout: return True return False def _next_correlation_id(self): self._correlation_id = (self._correlation_id + 1) % 2**31 return self._correlation_id def check_version(self, timeout=2, strict=False): """Attempt to guess the broker version. Note: This is a blocking call. Returns: version tuple, i.e. (0, 10), (0, 9), (0, 8, 2), ... """ # Monkeypatch some connection configurations to avoid timeouts override_config = { 'request_timeout_ms': timeout * 1000, 'max_in_flight_requests_per_connection': 5 } stashed = {} for key in override_config: stashed[key] = self.config[key] self.config[key] = override_config[key] # kafka kills the connection when it doesnt recognize an API request # so we can send a test request and then follow immediately with a # vanilla MetadataRequest. If the server did not recognize the first # request, both will be failed with a ConnectionError that wraps # socket.error (32, 54, or 104) from .protocol.admin import ApiVersionRequest, ListGroupsRequest from .protocol.commit import OffsetFetchRequest, GroupCoordinatorRequest from .protocol.metadata import MetadataRequest # Socket errors are logged as exceptions and can alarm users. Mute them from logging import Filter class ConnFilter(Filter): def filter(self, record): if record.funcName == 'check_version': return True return False log_filter = ConnFilter() log.addFilter(log_filter) test_cases = [ ((0, 10), ApiVersionRequest[0]()), ((0, 9), ListGroupsRequest[0]()), ((0, 8, 2), GroupCoordinatorRequest[0]('kafka-python-default-group')), ((0, 8, 1), OffsetFetchRequest[0]('kafka-python-default-group', [])), ((0, 8, 0), MetadataRequest[0]([])), ] def connect(): self.connect() if self.connected(): return timeout_at = time.time() + timeout while time.time() < timeout_at and self.connecting(): if self.connect() is ConnectionStates.CONNECTED: return time.sleep(0.05) raise Errors.NodeNotReadyError() for version, request in test_cases: connect() f = self.send(request) # HACK: sleeping to wait for socket to send bytes time.sleep(0.1) # when broker receives an unrecognized request API # it abruptly closes our socket. # so we attempt to send a second request immediately # that we believe it will definitely recognize (metadata) # the attempt to write to a disconnected socket should # immediately fail and allow us to infer that the prior # request was unrecognized mr = self.send(MetadataRequest[0]([])) if self._sock: self._sock.setblocking(True) while not (f.is_done and mr.is_done): self.recv() if self._sock: self._sock.setblocking(False) if f.succeeded(): log.info('Broker version identifed as %s', '.'.join(map(str, version))) log.info('Set configuration api_version=%s to skip auto' ' check_version requests on startup', version) break # Only enable strict checking to verify that we understand failure # modes. For most users, the fact that the request failed should be # enough to rule out a particular broker version. if strict: # If the socket flush hack did not work (which should force the # connection to close and fail all pending requests), then we # get a basic Request Timeout. This is not ideal, but we'll deal if isinstance(f.exception, Errors.RequestTimedOutError): pass # 0.9 brokers do not close the socket on unrecognized api # requests (bug...). In this case we expect to see a correlation # id mismatch elif (isinstance(f.exception, Errors.CorrelationIdError) and version == (0, 10)): pass elif six.PY2: assert isinstance(f.exception.args[0], socket.error) assert f.exception.args[0].errno in (32, 54, 104) else: assert isinstance(f.exception.args[0], ConnectionError) log.info("Broker is not v%s -- it did not recognize %s", version, request.__class__.__name__) else: raise Errors.UnrecognizedBrokerVersion() log.removeFilter(log_filter) for key in stashed: self.config[key] = stashed[key] return version def __repr__(self): return "" % (self.hostname, self.host, self.port) class BrokerConnectionMetrics(object): def __init__(self, metrics, metric_group_prefix, node_id): self.metrics = metrics # Any broker may have registered summary metrics already # but if not, we need to create them so we can set as parents below all_conns_transferred = metrics.get_sensor('bytes-sent-received') if not all_conns_transferred: metric_group_name = metric_group_prefix + '-metrics' bytes_transferred = metrics.sensor('bytes-sent-received') bytes_transferred.add(metrics.metric_name( 'network-io-rate', metric_group_name, 'The average number of network operations (reads or writes) on all' ' connections per second.'), Rate(sampled_stat=Count())) bytes_sent = metrics.sensor('bytes-sent', parents=[bytes_transferred]) bytes_sent.add(metrics.metric_name( 'outgoing-byte-rate', metric_group_name, 'The average number of outgoing bytes sent per second to all' ' servers.'), Rate()) bytes_sent.add(metrics.metric_name( 'request-rate', metric_group_name, 'The average number of requests sent per second.'), Rate(sampled_stat=Count())) bytes_sent.add(metrics.metric_name( 'request-size-avg', metric_group_name, 'The average size of all requests in the window.'), Avg()) bytes_sent.add(metrics.metric_name( 'request-size-max', metric_group_name, 'The maximum size of any request sent in the window.'), Max()) bytes_received = metrics.sensor('bytes-received', parents=[bytes_transferred]) bytes_received.add(metrics.metric_name( 'incoming-byte-rate', metric_group_name, 'Bytes/second read off all sockets'), Rate()) bytes_received.add(metrics.metric_name( 'response-rate', metric_group_name, 'Responses received sent per second.'), Rate(sampled_stat=Count())) request_latency = metrics.sensor('request-latency') request_latency.add(metrics.metric_name( 'request-latency-avg', metric_group_name, 'The average request latency in ms.'), Avg()) request_latency.add(metrics.metric_name( 'request-latency-max', metric_group_name, 'The maximum request latency in ms.'), Max()) # if one sensor of the metrics has been registered for the connection, # then all other sensors should have been registered; and vice versa node_str = 'node-{0}'.format(node_id) node_sensor = metrics.get_sensor(node_str + '.bytes-sent') if not node_sensor: metric_group_name = metric_group_prefix + '-node-metrics.' + node_str bytes_sent = metrics.sensor( node_str + '.bytes-sent', parents=[metrics.get_sensor('bytes-sent')]) bytes_sent.add(metrics.metric_name( 'outgoing-byte-rate', metric_group_name, 'The average number of outgoing bytes sent per second.'), Rate()) bytes_sent.add(metrics.metric_name( 'request-rate', metric_group_name, 'The average number of requests sent per second.'), Rate(sampled_stat=Count())) bytes_sent.add(metrics.metric_name( 'request-size-avg', metric_group_name, 'The average size of all requests in the window.'), Avg()) bytes_sent.add(metrics.metric_name( 'request-size-max', metric_group_name, 'The maximum size of any request sent in the window.'), Max()) bytes_received = metrics.sensor( node_str + '.bytes-received', parents=[metrics.get_sensor('bytes-received')]) bytes_received.add(metrics.metric_name( 'incoming-byte-rate', metric_group_name, 'Bytes/second read off node-connection socket'), Rate()) bytes_received.add(metrics.metric_name( 'response-rate', metric_group_name, 'The average number of responses received per second.'), Rate(sampled_stat=Count())) request_time = metrics.sensor( node_str + '.latency', parents=[metrics.get_sensor('request-latency')]) request_time.add(metrics.metric_name( 'request-latency-avg', metric_group_name, 'The average request latency in ms.'), Avg()) request_time.add(metrics.metric_name( 'request-latency-max', metric_group_name, 'The maximum request latency in ms.'), Max()) self.bytes_sent = metrics.sensor(node_str + '.bytes-sent') self.bytes_received = metrics.sensor(node_str + '.bytes-received') self.request_time = metrics.sensor(node_str + '.latency') def _address_family(address): """ Attempt to determine the family of an address (or hostname) :return: either socket.AF_INET or socket.AF_INET6 or socket.AF_UNSPEC if the address family could not be determined """ if address.startswith('[') and address.endswith(']'): return socket.AF_INET6 for af in (socket.AF_INET, socket.AF_INET6): try: socket.inet_pton(af, address) return af except (ValueError, AttributeError, socket.error): continue return socket.AF_UNSPEC def get_ip_port_afi(host_and_port_str): """ Parse the IP and port from a string in the format of: * host_or_ip <- Can be either IPv4 address literal or hostname/fqdn * host_or_ipv4:port <- Can be either IPv4 address literal or hostname/fqdn * [host_or_ip] <- IPv6 address literal * [host_or_ip]:port. <- IPv6 address literal .. note:: IPv6 address literals with ports *must* be enclosed in brackets .. note:: If the port is not specified, default will be returned. :return: tuple (host, port, afi), afi will be socket.AF_INET or socket.AF_INET6 or socket.AF_UNSPEC """ host_and_port_str = host_and_port_str.strip() if host_and_port_str.startswith('['): af = socket.AF_INET6 host, rest = host_and_port_str[1:].split(']') if rest: port = int(rest[1:]) else: port = DEFAULT_KAFKA_PORT return host, port, af else: if ':' not in host_and_port_str: af = _address_family(host_and_port_str) return host_and_port_str, DEFAULT_KAFKA_PORT, af else: # now we have something with a colon in it and no square brackets. It could be # either an IPv6 address literal (e.g., "::1") or an IP:port pair or a host:port pair try: # if it decodes as an IPv6 address, use that socket.inet_pton(socket.AF_INET6, host_and_port_str) return host_and_port_str, DEFAULT_KAFKA_PORT, socket.AF_INET6 except AttributeError: log.warning('socket.inet_pton not available on this platform.' ' consider pip install win_inet_pton') pass except (ValueError, socket.error): # it's a host:port pair pass host, port = host_and_port_str.rsplit(':', 1) port = int(port) af = _address_family(host) return host, port, af def collect_hosts(hosts, randomize=True): """ Collects a comma-separated set of hosts (host:port) and optionally randomize the returned list. """ if isinstance(hosts, six.string_types): hosts = hosts.strip().split(',') result = [] afi = socket.AF_INET for host_port in hosts: host, port, afi = get_ip_port_afi(host_port) if port < 0: port = DEFAULT_KAFKA_PORT result.append((host, port, afi)) if randomize: shuffle(result) return result kafka-1.3.2/kafka/consumer/0000755001271300127130000000000013031057517015272 5ustar dpowers00000000000000kafka-1.3.2/kafka/consumer/__init__.py0000644001271300127130000000035013025302127017372 0ustar dpowers00000000000000from __future__ import absolute_import from .simple import SimpleConsumer from .multiprocess import MultiProcessConsumer from .group import KafkaConsumer __all__ = [ 'SimpleConsumer', 'MultiProcessConsumer', 'KafkaConsumer' ] kafka-1.3.2/kafka/consumer/base.py0000644001271300127130000001705512702214455016566 0ustar dpowers00000000000000from __future__ import absolute_import import atexit import logging import numbers from threading import Lock import warnings from kafka.errors import ( UnknownTopicOrPartitionError, check_error, KafkaError) from kafka.structs import ( OffsetRequestPayload, OffsetCommitRequestPayload, OffsetFetchRequestPayload) from kafka.util import ReentrantTimer log = logging.getLogger('kafka.consumer') AUTO_COMMIT_MSG_COUNT = 100 AUTO_COMMIT_INTERVAL = 5000 FETCH_DEFAULT_BLOCK_TIMEOUT = 1 FETCH_MAX_WAIT_TIME = 100 FETCH_MIN_BYTES = 4096 FETCH_BUFFER_SIZE_BYTES = 4096 MAX_FETCH_BUFFER_SIZE_BYTES = FETCH_BUFFER_SIZE_BYTES * 8 ITER_TIMEOUT_SECONDS = 60 NO_MESSAGES_WAIT_TIME_SECONDS = 0.1 FULL_QUEUE_WAIT_TIME_SECONDS = 0.1 MAX_BACKOFF_SECONDS = 60 class Consumer(object): """ Base class to be used by other consumers. Not to be used directly This base class provides logic for * initialization and fetching metadata of partitions * Auto-commit logic * APIs for fetching pending message count """ def __init__(self, client, group, topic, partitions=None, auto_commit=True, auto_commit_every_n=AUTO_COMMIT_MSG_COUNT, auto_commit_every_t=AUTO_COMMIT_INTERVAL): warnings.warn('deprecated -- this class will be removed in a future' ' release. Use KafkaConsumer instead.', DeprecationWarning) self.client = client self.topic = topic self.group = group self.client.load_metadata_for_topics(topic, ignore_leadernotavailable=True) self.offsets = {} if partitions is None: partitions = self.client.get_partition_ids_for_topic(topic) else: assert all(isinstance(x, numbers.Integral) for x in partitions) # Variables for handling offset commits self.commit_lock = Lock() self.commit_timer = None self.count_since_commit = 0 self.auto_commit = auto_commit self.auto_commit_every_n = auto_commit_every_n self.auto_commit_every_t = auto_commit_every_t # Set up the auto-commit timer if auto_commit is True and auto_commit_every_t is not None: self.commit_timer = ReentrantTimer(auto_commit_every_t, self.commit) self.commit_timer.start() # Set initial offsets if self.group is not None: self.fetch_last_known_offsets(partitions) else: for partition in partitions: self.offsets[partition] = 0 # Register a cleanup handler def cleanup(obj): obj.stop() self._cleanup_func = cleanup atexit.register(cleanup, self) self.partition_info = False # Do not return partition info in msgs def provide_partition_info(self): """ Indicates that partition info must be returned by the consumer """ self.partition_info = True def fetch_last_known_offsets(self, partitions=None): if self.group is None: raise ValueError('SimpleClient.group must not be None') if partitions is None: partitions = self.client.get_partition_ids_for_topic(self.topic) responses = self.client.send_offset_fetch_request( self.group, [OffsetFetchRequestPayload(self.topic, p) for p in partitions], fail_on_error=False ) for resp in responses: try: check_error(resp) # API spec says server wont set an error here # but 0.8.1.1 does actually... except UnknownTopicOrPartitionError: pass # -1 offset signals no commit is currently stored if resp.offset == -1: self.offsets[resp.partition] = 0 # Otherwise we committed the stored offset # and need to fetch the next one else: self.offsets[resp.partition] = resp.offset def commit(self, partitions=None): """Commit stored offsets to Kafka via OffsetCommitRequest (v0) Keyword Arguments: partitions (list): list of partitions to commit, default is to commit all of them Returns: True on success, False on failure """ # short circuit if nothing happened. This check is kept outside # to prevent un-necessarily acquiring a lock for checking the state if self.count_since_commit == 0: return with self.commit_lock: # Do this check again, just in case the state has changed # during the lock acquiring timeout if self.count_since_commit == 0: return reqs = [] if partitions is None: # commit all partitions partitions = list(self.offsets.keys()) log.debug('Committing new offsets for %s, partitions %s', self.topic, partitions) for partition in partitions: offset = self.offsets[partition] log.debug('Commit offset %d in SimpleConsumer: ' 'group=%s, topic=%s, partition=%s', offset, self.group, self.topic, partition) reqs.append(OffsetCommitRequestPayload(self.topic, partition, offset, None)) try: self.client.send_offset_commit_request(self.group, reqs) except KafkaError as e: log.error('%s saving offsets: %s', e.__class__.__name__, e) return False else: self.count_since_commit = 0 return True def _auto_commit(self): """ Check if we have to commit based on number of messages and commit """ # Check if we are supposed to do an auto-commit if not self.auto_commit or self.auto_commit_every_n is None: return if self.count_since_commit >= self.auto_commit_every_n: self.commit() def stop(self): if self.commit_timer is not None: self.commit_timer.stop() self.commit() if hasattr(self, '_cleanup_func'): # Remove cleanup handler now that we've stopped # py3 supports unregistering if hasattr(atexit, 'unregister'): atexit.unregister(self._cleanup_func) # pylint: disable=no-member # py2 requires removing from private attribute... else: # ValueError on list.remove() if the exithandler no longer # exists is fine here try: atexit._exithandlers.remove( # pylint: disable=no-member (self._cleanup_func, (self,), {})) except ValueError: pass del self._cleanup_func def pending(self, partitions=None): """ Gets the pending message count Keyword Arguments: partitions (list): list of partitions to check for, default is to check all """ if partitions is None: partitions = self.offsets.keys() total = 0 reqs = [] for partition in partitions: reqs.append(OffsetRequestPayload(self.topic, partition, -1, 1)) resps = self.client.send_offset_request(reqs) for resp in resps: partition = resp.partition pending = resp.offsets[0] offset = self.offsets[partition] total += pending - offset return total kafka-1.3.2/kafka/consumer/fetcher.py0000644001271300127130000011773413031057471017300 0ustar dpowers00000000000000from __future__ import absolute_import import collections import copy import logging import random import sys import time from kafka.vendor import six import kafka.errors as Errors from kafka.future import Future from kafka.metrics.stats import Avg, Count, Max, Rate from kafka.protocol.fetch import FetchRequest from kafka.protocol.message import PartialMessage from kafka.protocol.offset import OffsetRequest, OffsetResetStrategy from kafka.serializer import Deserializer from kafka.structs import TopicPartition log = logging.getLogger(__name__) ConsumerRecord = collections.namedtuple("ConsumerRecord", ["topic", "partition", "offset", "timestamp", "timestamp_type", "key", "value", "checksum", "serialized_key_size", "serialized_value_size"]) class NoOffsetForPartitionError(Errors.KafkaError): pass class RecordTooLargeError(Errors.KafkaError): pass class Fetcher(six.Iterator): DEFAULT_CONFIG = { 'key_deserializer': None, 'value_deserializer': None, 'fetch_min_bytes': 1, 'fetch_max_wait_ms': 500, 'max_partition_fetch_bytes': 1048576, 'max_poll_records': sys.maxsize, 'check_crcs': True, 'skip_double_compressed_messages': False, 'iterator_refetch_records': 1, # undocumented -- interface may change 'metric_group_prefix': 'consumer', 'api_version': (0, 8, 0), } def __init__(self, client, subscriptions, metrics, **configs): """Initialize a Kafka Message Fetcher. Keyword Arguments: key_deserializer (callable): Any callable that takes a raw message key and returns a deserialized key. value_deserializer (callable, optional): Any callable that takes a raw message value and returns a deserialized value. fetch_min_bytes (int): Minimum amount of data the server should return for a fetch request, otherwise wait up to fetch_max_wait_ms for more data to accumulate. Default: 1. fetch_max_wait_ms (int): The maximum amount of time in milliseconds the server will block before answering the fetch request if there isn't sufficient data to immediately satisfy the requirement given by fetch_min_bytes. Default: 500. max_partition_fetch_bytes (int): The maximum amount of data per-partition the server will return. The maximum total memory used for a request = #partitions * max_partition_fetch_bytes. This size must be at least as large as the maximum message size the server allows or else it is possible for the producer to send messages larger than the consumer can fetch. If that happens, the consumer can get stuck trying to fetch a large message on a certain partition. Default: 1048576. check_crcs (bool): Automatically check the CRC32 of the records consumed. This ensures no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so it may be disabled in cases seeking extreme performance. Default: True skip_double_compressed_messages (bool): A bug in KafkaProducer caused some messages to be corrupted via double-compression. By default, the fetcher will return the messages as a compressed blob of bytes with a single offset, i.e. how the message was actually published to the cluster. If you prefer to have the fetcher automatically detect corrupt messages and skip them, set this option to True. Default: False. """ self.config = copy.copy(self.DEFAULT_CONFIG) for key in self.config: if key in configs: self.config[key] = configs[key] self._client = client self._subscriptions = subscriptions self._records = collections.deque() # (offset, topic_partition, messages) self._unauthorized_topics = set() self._offset_out_of_range_partitions = dict() # {topic_partition: offset} self._record_too_large_partitions = dict() # {topic_partition: offset} self._iterator = None self._fetch_futures = collections.deque() self._sensors = FetchManagerMetrics(metrics, self.config['metric_group_prefix']) def send_fetches(self): """Send FetchRequests asynchronously for all assigned partitions. Note: noop if there are unconsumed records internal to the fetcher Returns: List of Futures: each future resolves to a FetchResponse """ futures = [] for node_id, request in six.iteritems(self._create_fetch_requests()): if self._client.ready(node_id): log.debug("Sending FetchRequest to node %s", node_id) future = self._client.send(node_id, request) future.add_callback(self._handle_fetch_response, request, time.time()) future.add_errback(log.error, 'Fetch to node %s failed: %s', node_id) futures.append(future) self._fetch_futures.extend(futures) self._clean_done_fetch_futures() return futures def _clean_done_fetch_futures(self): while True: if not self._fetch_futures: break if not self._fetch_futures[0].is_done: break self._fetch_futures.popleft() def in_flight_fetches(self): """Return True if there are any unprocessed FetchRequests in flight.""" self._clean_done_fetch_futures() return bool(self._fetch_futures) def update_fetch_positions(self, partitions): """Update the fetch positions for the provided partitions. Arguments: partitions (list of TopicPartitions): partitions to update Raises: NoOffsetForPartitionError: if no offset is stored for a given partition and no reset policy is available """ # reset the fetch position to the committed position for tp in partitions: if not self._subscriptions.is_assigned(tp): log.warning("partition %s is not assigned - skipping offset" " update", tp) continue elif self._subscriptions.is_fetchable(tp): log.warning("partition %s is still fetchable -- skipping offset" " update", tp) continue # TODO: If there are several offsets to reset, # we could submit offset requests in parallel # for now, each call to _reset_offset will block if self._subscriptions.is_offset_reset_needed(tp): self._reset_offset(tp) elif self._subscriptions.assignment[tp].committed is None: # there's no committed position, so we need to reset with the # default strategy self._subscriptions.need_offset_reset(tp) self._reset_offset(tp) else: committed = self._subscriptions.assignment[tp].committed log.debug("Resetting offset for partition %s to the committed" " offset %s", tp, committed) self._subscriptions.seek(tp, committed) def _reset_offset(self, partition): """Reset offsets for the given partition using the offset reset strategy. Arguments: partition (TopicPartition): the partition that needs reset offset Raises: NoOffsetForPartitionError: if no offset reset strategy is defined """ timestamp = self._subscriptions.assignment[partition].reset_strategy if timestamp is OffsetResetStrategy.EARLIEST: strategy = 'earliest' elif timestamp is OffsetResetStrategy.LATEST: strategy = 'latest' else: raise NoOffsetForPartitionError(partition) log.debug("Resetting offset for partition %s to %s offset.", partition, strategy) offset = self._offset(partition, timestamp) # we might lose the assignment while fetching the offset, # so check it is still active if self._subscriptions.is_assigned(partition): self._subscriptions.seek(partition, offset) def _offset(self, partition, timestamp): """Fetch a single offset before the given timestamp for the partition. Blocks until offset is obtained, or a non-retriable exception is raised Arguments: partition The partition that needs fetching offset. timestamp (int): timestamp for fetching offset. -1 for the latest available, -2 for the earliest available. Otherwise timestamp is treated as epoch seconds. Returns: int: message offset """ while True: future = self._send_offset_request(partition, timestamp) self._client.poll(future=future) if future.succeeded(): return future.value if not future.retriable(): raise future.exception # pylint: disable-msg=raising-bad-type if future.exception.invalid_metadata: refresh_future = self._client.cluster.request_update() self._client.poll(future=refresh_future, sleep=True) def _raise_if_offset_out_of_range(self): """Check FetchResponses for offset out of range. Raises: OffsetOutOfRangeError: if any partition from previous FetchResponse contains OffsetOutOfRangeError and the default_reset_policy is None """ if not self._offset_out_of_range_partitions: return current_out_of_range_partitions = {} # filter only the fetchable partitions for partition, offset in six.iteritems(self._offset_out_of_range_partitions): if not self._subscriptions.is_fetchable(partition): log.debug("Ignoring fetched records for %s since it is no" " longer fetchable", partition) continue position = self._subscriptions.assignment[partition].position # ignore partition if the current position != offset in FetchResponse # e.g. after seek() if position is not None and offset == position: current_out_of_range_partitions[partition] = position self._offset_out_of_range_partitions.clear() if current_out_of_range_partitions: raise Errors.OffsetOutOfRangeError(current_out_of_range_partitions) def _raise_if_unauthorized_topics(self): """Check FetchResponses for topic authorization failures. Raises: TopicAuthorizationFailedError """ if self._unauthorized_topics: topics = set(self._unauthorized_topics) self._unauthorized_topics.clear() raise Errors.TopicAuthorizationFailedError(topics) def _raise_if_record_too_large(self): """Check FetchResponses for messages larger than the max per partition. Raises: RecordTooLargeError: if there is a message larger than fetch size """ if not self._record_too_large_partitions: return copied_record_too_large_partitions = dict(self._record_too_large_partitions) self._record_too_large_partitions.clear() raise RecordTooLargeError( "There are some messages at [Partition=Offset]: %s " " whose size is larger than the fetch size %s" " and hence cannot be ever returned." " Increase the fetch size, or decrease the maximum message" " size the broker will allow.", copied_record_too_large_partitions, self.config['max_partition_fetch_bytes']) def fetched_records(self, max_records=None): """Returns previously fetched records and updates consumed offsets. Arguments: max_records (int): Maximum number of records returned. Defaults to max_poll_records configuration. Raises: OffsetOutOfRangeError: if no subscription offset_reset_strategy InvalidMessageError: if message crc validation fails (check_crcs must be set to True) RecordTooLargeError: if a message is larger than the currently configured max_partition_fetch_bytes TopicAuthorizationError: if consumer is not authorized to fetch messages from the topic Returns: (records (dict), partial (bool)) records: {TopicPartition: [messages]} partial: True if records returned did not fully drain any pending partition requests. This may be useful for choosing when to pipeline additional fetch requests. """ if max_records is None: max_records = self.config['max_poll_records'] assert max_records > 0 if self._subscriptions.needs_partition_assignment: return {}, False self._raise_if_offset_out_of_range() self._raise_if_unauthorized_topics() self._raise_if_record_too_large() drained = collections.defaultdict(list) partial = bool(self._records and max_records) while self._records and max_records > 0: part = self._records.popleft() max_records -= self._append(drained, part, max_records) if part.has_more(): self._records.appendleft(part) else: partial &= False return dict(drained), partial def _append(self, drained, part, max_records): tp = part.topic_partition fetch_offset = part.fetch_offset if not self._subscriptions.is_assigned(tp): # this can happen when a rebalance happened before # fetched records are returned to the consumer's poll call log.debug("Not returning fetched records for partition %s" " since it is no longer assigned", tp) else: # note that the position should always be available # as long as the partition is still assigned position = self._subscriptions.assignment[tp].position if not self._subscriptions.is_fetchable(tp): # this can happen when a partition is paused before # fetched records are returned to the consumer's poll call log.debug("Not returning fetched records for assigned partition" " %s since it is no longer fetchable", tp) elif fetch_offset == position: part_records = part.take(max_records) if not part_records: return 0 next_offset = part_records[-1].offset + 1 log.log(0, "Returning fetched records at offset %d for assigned" " partition %s and update position to %s", position, tp, next_offset) for record in part_records: # Fetched compressed messages may include additional records if record.offset < fetch_offset: log.debug("Skipping message offset: %s (expecting %s)", record.offset, fetch_offset) continue drained[tp].append(record) self._subscriptions.assignment[tp].position = next_offset return len(part_records) else: # these records aren't next in line based on the last consumed # position, ignore them they must be from an obsolete request log.debug("Ignoring fetched records for %s at offset %s since" " the current position is %d", tp, part.fetch_offset, position) part.discard() return 0 def _message_generator(self): """Iterate over fetched_records""" if self._subscriptions.needs_partition_assignment: raise StopIteration('Subscription needs partition assignment') while self._records: # Check on each iteration since this is a generator self._raise_if_offset_out_of_range() self._raise_if_unauthorized_topics() self._raise_if_record_too_large() # Send additional FetchRequests when the internal queue is low # this should enable moderate pipelining if len(self._records) <= self.config['iterator_refetch_records']: self.send_fetches() part = self._records.popleft() tp = part.topic_partition fetch_offset = part.fetch_offset if not self._subscriptions.is_assigned(tp): # this can happen when a rebalance happened before # fetched records are returned log.debug("Not returning fetched records for partition %s" " since it is no longer assigned", tp) continue # note that the position should always be available # as long as the partition is still assigned position = self._subscriptions.assignment[tp].position if not self._subscriptions.is_fetchable(tp): # this can happen when a partition is paused before # fetched records are returned log.debug("Not returning fetched records for assigned partition" " %s since it is no longer fetchable", tp) elif fetch_offset == position: log.log(0, "Returning fetched records at offset %d for assigned" " partition %s", position, tp) # We can ignore any prior signal to drop pending message sets # because we are starting from a fresh one where fetch_offset == position # i.e., the user seek()'d to this position self._subscriptions.assignment[tp].drop_pending_message_set = False for msg in part.messages: # Because we are in a generator, it is possible for # subscription state to change between yield calls # so we need to re-check on each loop # this should catch assignment changes, pauses # and resets via seek_to_beginning / seek_to_end if not self._subscriptions.is_fetchable(tp): log.debug("Not returning fetched records for partition %s" " since it is no longer fetchable", tp) break # If there is a seek during message iteration, # we should stop unpacking this message set and # wait for a new fetch response that aligns with the # new seek position elif self._subscriptions.assignment[tp].drop_pending_message_set: log.debug("Skipping remainder of message set for partition %s", tp) self._subscriptions.assignment[tp].drop_pending_message_set = False break # Compressed messagesets may include earlier messages elif msg.offset < self._subscriptions.assignment[tp].position: log.debug("Skipping message offset: %s (expecting %s)", msg.offset, self._subscriptions.assignment[tp].position) continue self._subscriptions.assignment[tp].position = msg.offset + 1 yield msg else: # these records aren't next in line based on the last consumed # position, ignore them they must be from an obsolete request log.debug("Ignoring fetched records for %s at offset %s since" " the current position is %d", tp, part.fetch_offset, position) def _unpack_message_set(self, tp, messages): try: for offset, size, msg in messages: if self.config['check_crcs'] and not msg.validate_crc(): raise Errors.InvalidMessageError(msg) elif msg.is_compressed(): # If relative offset is used, we need to decompress the entire message first to compute # the absolute offset. inner_mset = msg.decompress() # There should only ever be a single layer of compression if inner_mset[0][-1].is_compressed(): log.warning('MessageSet at %s offset %d appears ' ' double-compressed. This should not' ' happen -- check your producers!', tp, offset) if self.config['skip_double_compressed_messages']: log.warning('Skipping double-compressed message at' ' %s %d', tp, offset) continue if msg.magic > 0: last_offset, _, _ = inner_mset[-1] absolute_base_offset = offset - last_offset else: absolute_base_offset = -1 for inner_offset, inner_size, inner_msg in inner_mset: if msg.magic > 0: # When magic value is greater than 0, the timestamp # of a compressed message depends on the # typestamp type of the wrapper message: if msg.timestamp_type == 0: # CREATE_TIME (0) inner_timestamp = inner_msg.timestamp elif msg.timestamp_type == 1: # LOG_APPEND_TIME (1) inner_timestamp = msg.timestamp else: raise ValueError('Unknown timestamp type: {0}'.format(msg.timestamp_type)) else: inner_timestamp = msg.timestamp if absolute_base_offset >= 0: inner_offset += absolute_base_offset key = self._deserialize( self.config['key_deserializer'], tp.topic, inner_msg.key) value = self._deserialize( self.config['value_deserializer'], tp.topic, inner_msg.value) yield ConsumerRecord(tp.topic, tp.partition, inner_offset, inner_timestamp, msg.timestamp_type, key, value, inner_msg.crc, len(inner_msg.key) if inner_msg.key is not None else -1, len(inner_msg.value) if inner_msg.value is not None else -1) else: key = self._deserialize( self.config['key_deserializer'], tp.topic, msg.key) value = self._deserialize( self.config['value_deserializer'], tp.topic, msg.value) yield ConsumerRecord(tp.topic, tp.partition, offset, msg.timestamp, msg.timestamp_type, key, value, msg.crc, len(msg.key) if msg.key is not None else -1, len(msg.value) if msg.value is not None else -1) # If unpacking raises StopIteration, it is erroneously # caught by the generator. We want all exceptions to be raised # back to the user. See Issue 545 except StopIteration as e: log.exception('StopIteration raised unpacking messageset: %s', e) raise Exception('StopIteration raised unpacking messageset') def __iter__(self): # pylint: disable=non-iterator-returned return self def __next__(self): if not self._iterator: self._iterator = self._message_generator() try: return next(self._iterator) except StopIteration: self._iterator = None raise def _deserialize(self, f, topic, bytes_): if not f: return bytes_ if isinstance(f, Deserializer): return f.deserialize(topic, bytes_) return f(bytes_) def _send_offset_request(self, partition, timestamp): """Fetch a single offset before the given timestamp for the partition. Arguments: partition (TopicPartition): partition that needs fetching offset timestamp (int): timestamp for fetching offset Returns: Future: resolves to the corresponding offset """ node_id = self._client.cluster.leader_for_partition(partition) if node_id is None: log.debug("Partition %s is unknown for fetching offset," " wait for metadata refresh", partition) return Future().failure(Errors.StaleMetadata(partition)) elif node_id == -1: log.debug("Leader for partition %s unavailable for fetching offset," " wait for metadata refresh", partition) return Future().failure(Errors.LeaderNotAvailableError(partition)) request = OffsetRequest[0]( -1, [(partition.topic, [(partition.partition, timestamp, 1)])] ) # Client returns a future that only fails on network issues # so create a separate future and attach a callback to update it # based on response error codes future = Future() _f = self._client.send(node_id, request) _f.add_callback(self._handle_offset_response, partition, future) _f.add_errback(lambda e: future.failure(e)) return future def _handle_offset_response(self, partition, future, response): """Callback for the response of the list offset call above. Arguments: partition (TopicPartition): The partition that was fetched future (Future): the future to update based on response response (OffsetResponse): response from the server Raises: AssertionError: if response does not match partition """ topic, partition_info = response.topics[0] assert len(response.topics) == 1 and len(partition_info) == 1, ( 'OffsetResponse should only be for a single topic-partition') part, error_code, offsets = partition_info[0] assert topic == partition.topic and part == partition.partition, ( 'OffsetResponse partition does not match OffsetRequest partition') error_type = Errors.for_code(error_code) if error_type is Errors.NoError: assert len(offsets) == 1, 'Expected OffsetResponse with one offset' offset = offsets[0] log.debug("Fetched offset %d for partition %s", offset, partition) future.success(offset) elif error_type in (Errors.NotLeaderForPartitionError, Errors.UnknownTopicOrPartitionError): log.debug("Attempt to fetch offsets for partition %s failed due" " to obsolete leadership information, retrying.", partition) future.failure(error_type(partition)) else: log.warning("Attempt to fetch offsets for partition %s failed due to:" " %s", partition, error_type) future.failure(error_type(partition)) def _fetchable_partitions(self): fetchable = self._subscriptions.fetchable_partitions() pending = set([part.topic_partition for part in self._records]) return fetchable.difference(pending) def _create_fetch_requests(self): """Create fetch requests for all assigned partitions, grouped by node. FetchRequests skipped if no leader, or node has requests in flight Returns: dict: {node_id: FetchRequest, ...} (version depends on api_version) """ # create the fetch info as a dict of lists of partition info tuples # which can be passed to FetchRequest() via .items() fetchable = collections.defaultdict(lambda: collections.defaultdict(list)) for partition in self._fetchable_partitions(): node_id = self._client.cluster.leader_for_partition(partition) position = self._subscriptions.assignment[partition].position # fetch if there is a leader and no in-flight requests if node_id is None or node_id == -1: log.debug("No leader found for partition %s." " Requesting metadata update", partition) self._client.cluster.request_update() elif self._client.in_flight_request_count(node_id) == 0: partition_info = ( partition.partition, position, self.config['max_partition_fetch_bytes'] ) fetchable[node_id][partition.topic].append(partition_info) log.debug("Adding fetch request for partition %s at offset %d", partition, position) if self.config['api_version'] >= (0, 10): version = 2 elif self.config['api_version'] == (0, 9): version = 1 else: version = 0 requests = {} for node_id, partition_data in six.iteritems(fetchable): requests[node_id] = FetchRequest[version]( -1, # replica_id self.config['fetch_max_wait_ms'], self.config['fetch_min_bytes'], partition_data.items()) return requests def _handle_fetch_response(self, request, send_time, response): """The callback for fetch completion""" total_bytes = 0 total_count = 0 recv_time = time.time() fetch_offsets = {} for topic, partitions in request.topics: for partition, offset, _ in partitions: fetch_offsets[TopicPartition(topic, partition)] = offset # randomized ordering should improve balance for short-lived consumers random.shuffle(response.topics) for topic, partitions in response.topics: random.shuffle(partitions) for partition, error_code, highwater, messages in partitions: tp = TopicPartition(topic, partition) error_type = Errors.for_code(error_code) if not self._subscriptions.is_fetchable(tp): # this can happen when a rebalance happened or a partition # consumption paused while fetch is still in-flight log.debug("Ignoring fetched records for partition %s" " since it is no longer fetchable", tp) elif error_type is Errors.NoError: self._subscriptions.assignment[tp].highwater = highwater # we are interested in this fetch only if the beginning # offset (of the *request*) matches the current consumed position # Note that the *response* may return a messageset that starts # earlier (e.g., compressed messages) or later (e.g., compacted topic) fetch_offset = fetch_offsets[tp] position = self._subscriptions.assignment[tp].position if position is None or position != fetch_offset: log.debug("Discarding fetch response for partition %s" " since its offset %d does not match the" " expected offset %d", tp, fetch_offset, position) continue num_bytes = 0 partial = None if messages and isinstance(messages[-1][-1], PartialMessage): partial = messages.pop() if messages: log.debug("Adding fetched record for partition %s with" " offset %d to buffered record list", tp, position) unpacked = list(self._unpack_message_set(tp, messages)) self._records.append(self.PartitionRecords(fetch_offset, tp, unpacked)) last_offset, _, _ = messages[-1] self._sensors.records_fetch_lag.record(highwater - last_offset) num_bytes = sum(msg[1] for msg in messages) elif partial: # we did not read a single message from a non-empty # buffer because that message's size is larger than # fetch size, in this case record this exception self._record_too_large_partitions[tp] = fetch_offset self._sensors.record_topic_fetch_metrics(topic, num_bytes, len(messages)) total_bytes += num_bytes total_count += len(messages) elif error_type in (Errors.NotLeaderForPartitionError, Errors.UnknownTopicOrPartitionError): self._client.cluster.request_update() elif error_type is Errors.OffsetOutOfRangeError: fetch_offset = fetch_offsets[tp] log.info("Fetch offset %s is out of range for topic-partition %s", fetch_offset, tp) if self._subscriptions.has_default_offset_reset_policy(): self._subscriptions.need_offset_reset(tp) log.info("Resetting offset for topic-partition %s", tp) else: self._offset_out_of_range_partitions[tp] = fetch_offset elif error_type is Errors.TopicAuthorizationFailedError: log.warn("Not authorized to read from topic %s.", tp.topic) self._unauthorized_topics.add(tp.topic) elif error_type is Errors.UnknownError: log.warn("Unknown error fetching data for topic-partition %s", tp) else: raise error_type('Unexpected error while fetching data') # Because we are currently decompressing messages lazily, the sensors here # will get compressed bytes / message set stats when compression is enabled self._sensors.bytes_fetched.record(total_bytes) self._sensors.records_fetched.record(total_count) if response.API_VERSION >= 1: self._sensors.fetch_throttle_time_sensor.record(response.throttle_time_ms) self._sensors.fetch_latency.record((recv_time - send_time) * 1000) class PartitionRecords(six.Iterator): def __init__(self, fetch_offset, tp, messages): self.fetch_offset = fetch_offset self.topic_partition = tp self.messages = messages self.message_idx = 0 def discard(self): self.messages = None def take(self, n): if not self.has_more(): return [] next_idx = self.message_idx + n res = self.messages[self.message_idx:next_idx] self.message_idx = next_idx if self.has_more(): self.fetch_offset = self.messages[self.message_idx].offset return res def has_more(self): return self.messages and self.message_idx < len(self.messages) class FetchManagerMetrics(object): def __init__(self, metrics, prefix): self.metrics = metrics self.group_name = '%s-fetch-manager-metrics' % prefix self.bytes_fetched = metrics.sensor('bytes-fetched') self.bytes_fetched.add(metrics.metric_name('fetch-size-avg', self.group_name, 'The average number of bytes fetched per request'), Avg()) self.bytes_fetched.add(metrics.metric_name('fetch-size-max', self.group_name, 'The maximum number of bytes fetched per request'), Max()) self.bytes_fetched.add(metrics.metric_name('bytes-consumed-rate', self.group_name, 'The average number of bytes consumed per second'), Rate()) self.records_fetched = self.metrics.sensor('records-fetched') self.records_fetched.add(metrics.metric_name('records-per-request-avg', self.group_name, 'The average number of records in each request'), Avg()) self.records_fetched.add(metrics.metric_name('records-consumed-rate', self.group_name, 'The average number of records consumed per second'), Rate()) self.fetch_latency = metrics.sensor('fetch-latency') self.fetch_latency.add(metrics.metric_name('fetch-latency-avg', self.group_name, 'The average time taken for a fetch request.'), Avg()) self.fetch_latency.add(metrics.metric_name('fetch-latency-max', self.group_name, 'The max time taken for any fetch request.'), Max()) self.fetch_latency.add(metrics.metric_name('fetch-rate', self.group_name, 'The number of fetch requests per second.'), Rate(sampled_stat=Count())) self.records_fetch_lag = metrics.sensor('records-lag') self.records_fetch_lag.add(metrics.metric_name('records-lag-max', self.group_name, 'The maximum lag in terms of number of records for any partition in self window'), Max()) self.fetch_throttle_time_sensor = metrics.sensor('fetch-throttle-time') self.fetch_throttle_time_sensor.add(metrics.metric_name('fetch-throttle-time-avg', self.group_name, 'The average throttle time in ms'), Avg()) self.fetch_throttle_time_sensor.add(metrics.metric_name('fetch-throttle-time-max', self.group_name, 'The maximum throttle time in ms'), Max()) def record_topic_fetch_metrics(self, topic, num_bytes, num_records): # record bytes fetched name = '.'.join(['topic', topic, 'bytes-fetched']) bytes_fetched = self.metrics.get_sensor(name) if not bytes_fetched: metric_tags = {'topic': topic.replace('.', '_')} bytes_fetched = self.metrics.sensor(name) bytes_fetched.add(self.metrics.metric_name('fetch-size-avg', self.group_name, 'The average number of bytes fetched per request for topic %s' % topic, metric_tags), Avg()) bytes_fetched.add(self.metrics.metric_name('fetch-size-max', self.group_name, 'The maximum number of bytes fetched per request for topic %s' % topic, metric_tags), Max()) bytes_fetched.add(self.metrics.metric_name('bytes-consumed-rate', self.group_name, 'The average number of bytes consumed per second for topic %s' % topic, metric_tags), Rate()) bytes_fetched.record(num_bytes) # record records fetched name = '.'.join(['topic', topic, 'records-fetched']) records_fetched = self.metrics.get_sensor(name) if not records_fetched: metric_tags = {'topic': topic.replace('.', '_')} records_fetched = self.metrics.sensor(name) records_fetched.add(self.metrics.metric_name('records-per-request-avg', self.group_name, 'The average number of records in each request for topic %s' % topic, metric_tags), Avg()) records_fetched.add(self.metrics.metric_name('records-consumed-rate', self.group_name, 'The average number of records consumed per second for topic %s' % topic, metric_tags), Rate()) records_fetched.record(num_records) kafka-1.3.2/kafka/consumer/group.py0000644001271300127130000012767213031057471017016 0ustar dpowers00000000000000from __future__ import absolute_import import copy import logging import socket import sys import time from kafka.vendor import six from kafka.client_async import KafkaClient, selectors from kafka.consumer.fetcher import Fetcher from kafka.consumer.subscription_state import SubscriptionState from kafka.coordinator.consumer import ConsumerCoordinator from kafka.coordinator.assignors.range import RangePartitionAssignor from kafka.coordinator.assignors.roundrobin import RoundRobinPartitionAssignor from kafka.metrics import MetricConfig, Metrics from kafka.protocol.offset import OffsetResetStrategy from kafka.structs import TopicPartition from kafka.version import __version__ log = logging.getLogger(__name__) class KafkaConsumer(six.Iterator): """Consume records from a Kafka cluster. The consumer will transparently handle the failure of servers in the Kafka cluster, and adapt as topic-partitions are created or migrate between brokers. It also interacts with the assigned kafka Group Coordinator node to allow multiple consumers to load balance consumption of topics (requires kafka >= 0.9.0.0). Arguments: *topics (str): optional list of topics to subscribe to. If not set, call subscribe() or assign() before consuming records. Keyword Arguments: bootstrap_servers: 'host[:port]' string (or list of 'host[:port]' strings) that the consumer should contact to bootstrap initial cluster metadata. This does not have to be the full node list. It just needs to have at least one broker that will respond to a Metadata API Request. Default port is 9092. If no servers are specified, will default to localhost:9092. client_id (str): a name for this client. This string is passed in each request to servers and can be used to identify specific server-side log entries that correspond to this client. Also submitted to GroupCoordinator for logging with respect to consumer group administration. Default: 'kafka-python-{version}' group_id (str or None): name of the consumer group to join for dynamic partition assignment (if enabled), and to use for fetching and committing offsets. If None, auto-partition assignment (via group coordinator) and offset commits are disabled. Default: 'kafka-python-default-group' key_deserializer (callable): Any callable that takes a raw message key and returns a deserialized key. value_deserializer (callable): Any callable that takes a raw message value and returns a deserialized value. fetch_min_bytes (int): Minimum amount of data the server should return for a fetch request, otherwise wait up to fetch_max_wait_ms for more data to accumulate. Default: 1. fetch_max_wait_ms (int): The maximum amount of time in milliseconds the server will block before answering the fetch request if there isn't sufficient data to immediately satisfy the requirement given by fetch_min_bytes. Default: 500. max_partition_fetch_bytes (int): The maximum amount of data per-partition the server will return. The maximum total memory used for a request = #partitions * max_partition_fetch_bytes. This size must be at least as large as the maximum message size the server allows or else it is possible for the producer to send messages larger than the consumer can fetch. If that happens, the consumer can get stuck trying to fetch a large message on a certain partition. Default: 1048576. request_timeout_ms (int): Client request timeout in milliseconds. Default: 40000. retry_backoff_ms (int): Milliseconds to backoff when retrying on errors. Default: 100. reconnect_backoff_ms (int): The amount of time in milliseconds to wait before attempting to reconnect to a given host. Default: 50. max_in_flight_requests_per_connection (int): Requests are pipelined to kafka brokers up to this number of maximum requests per broker connection. Default: 5. auto_offset_reset (str): A policy for resetting offsets on OffsetOutOfRange errors: 'earliest' will move to the oldest available message, 'latest' will move to the most recent. Any other value will raise the exception. Default: 'latest'. enable_auto_commit (bool): If true the consumer's offset will be periodically committed in the background. Default: True. auto_commit_interval_ms (int): milliseconds between automatic offset commits, if enable_auto_commit is True. Default: 5000. default_offset_commit_callback (callable): called as callback(offsets, response) response will be either an Exception or a OffsetCommitResponse struct. This callback can be used to trigger custom actions when a commit request completes. check_crcs (bool): Automatically check the CRC32 of the records consumed. This ensures no on-the-wire or on-disk corruption to the messages occurred. This check adds some overhead, so it may be disabled in cases seeking extreme performance. Default: True metadata_max_age_ms (int): The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions. Default: 300000 partition_assignment_strategy (list): List of objects to use to distribute partition ownership amongst consumer instances when group management is used. Default: [RangePartitionAssignor, RoundRobinPartitionAssignor] heartbeat_interval_ms (int): The expected time in milliseconds between heartbeats to the consumer coordinator when using Kafka's group management feature. Heartbeats are used to ensure that the consumer's session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session_timeout_ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances. Default: 3000 session_timeout_ms (int): The timeout used to detect failures when using Kafka's group managementment facilities. Default: 30000 max_poll_records (int): The maximum number of records returned in a single call to poll(). receive_buffer_bytes (int): The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. Default: None (relies on system defaults). The java client defaults to 32768. send_buffer_bytes (int): The size of the TCP send buffer (SO_SNDBUF) to use when sending data. Default: None (relies on system defaults). The java client defaults to 131072. socket_options (list): List of tuple-arguments to socket.setsockopt to apply to broker connection sockets. Default: [(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)] consumer_timeout_ms (int): number of milliseconds to block during message iteration before raising StopIteration (i.e., ending the iterator). Default block forever [float('inf')]. skip_double_compressed_messages (bool): A bug in KafkaProducer <= 1.2.4 caused some messages to be corrupted via double-compression. By default, the fetcher will return these messages as a compressed blob of bytes with a single offset, i.e. how the message was actually published to the cluster. If you prefer to have the fetcher automatically detect corrupt messages and skip them, set this option to True. Default: False. security_protocol (str): Protocol used to communicate with brokers. Valid values are: PLAINTEXT, SSL. Default: PLAINTEXT. ssl_context (ssl.SSLContext): pre-configured SSLContext for wrapping socket connections. If provided, all other ssl_* configurations will be ignored. Default: None. ssl_check_hostname (bool): flag to configure whether ssl handshake should verify that the certificate matches the brokers hostname. default: true. ssl_cafile (str): optional filename of ca file to use in certificate verification. default: none. ssl_certfile (str): optional filename of file in pem format containing the client certificate, as well as any ca certificates needed to establish the certificate's authenticity. default: none. ssl_keyfile (str): optional filename containing the client private key. default: none. ssl_password (str): optional password to be used when loading the certificate chain. default: None. ssl_crlfile (str): optional filename containing the CRL to check for certificate expiration. By default, no CRL check is done. When providing a file, only the leaf certificate will be checked against this CRL. The CRL can only be checked with Python 3.4+ or 2.7.9+. default: none. api_version (tuple): specify which kafka API version to use. If set to None, the client will attempt to infer the broker version by probing various APIs. Default: None Examples: (0, 9) enables full group coordination features with automatic partition assignment and rebalancing, (0, 8, 2) enables kafka-storage offset commits with manual partition assignment only, (0, 8, 1) enables zookeeper-storage offset commits with manual partition assignment only, (0, 8, 0) enables basic functionality but requires manual partition assignment and offset management. For a full list of supported versions, see KafkaClient.API_VERSIONS api_version_auto_timeout_ms (int): number of milliseconds to throw a timeout exception from the constructor when checking the broker api version. Only applies if api_version set to 'auto' metric_reporters (list): A list of classes to use as metrics reporters. Implementing the AbstractMetricsReporter interface allows plugging in classes that will be notified of new metric creation. Default: [] metrics_num_samples (int): The number of samples maintained to compute metrics. Default: 2 metrics_sample_window_ms (int): The maximum age in milliseconds of samples used to compute metrics. Default: 30000 selector (selectors.BaseSelector): Provide a specific selector implementation to use for I/O multiplexing. Default: selectors.DefaultSelector exclude_internal_topics (bool): Whether records from internal topics (such as offsets) should be exposed to the consumer. If set to True the only way to receive records from an internal topic is subscribing to it. Requires 0.10+ Default: True sasl_mechanism (str): string picking sasl mechanism when security_protocol is SASL_PLAINTEXT or SASL_SSL. Currently only PLAIN is supported. Default: None sasl_plain_username (str): username for sasl PLAIN authentication. Default: None sasl_plain_password (str): password for sasl PLAIN authentication. Default: None Note: Configuration parameters are described in more detail at https://kafka.apache.org/0100/configuration.html#newconsumerconfigs """ DEFAULT_CONFIG = { 'bootstrap_servers': 'localhost', 'client_id': 'kafka-python-' + __version__, 'group_id': 'kafka-python-default-group', 'key_deserializer': None, 'value_deserializer': None, 'fetch_max_wait_ms': 500, 'fetch_min_bytes': 1, 'max_partition_fetch_bytes': 1 * 1024 * 1024, 'request_timeout_ms': 40 * 1000, 'retry_backoff_ms': 100, 'reconnect_backoff_ms': 50, 'max_in_flight_requests_per_connection': 5, 'auto_offset_reset': 'latest', 'enable_auto_commit': True, 'auto_commit_interval_ms': 5000, 'default_offset_commit_callback': lambda offsets, response: True, 'check_crcs': True, 'metadata_max_age_ms': 5 * 60 * 1000, 'partition_assignment_strategy': (RangePartitionAssignor, RoundRobinPartitionAssignor), 'heartbeat_interval_ms': 3000, 'session_timeout_ms': 30000, 'max_poll_records': sys.maxsize, 'receive_buffer_bytes': None, 'send_buffer_bytes': None, 'socket_options': [(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)], 'consumer_timeout_ms': float('inf'), 'skip_double_compressed_messages': False, 'security_protocol': 'PLAINTEXT', 'ssl_context': None, 'ssl_check_hostname': True, 'ssl_cafile': None, 'ssl_certfile': None, 'ssl_keyfile': None, 'ssl_crlfile': None, 'ssl_password': None, 'api_version': None, 'api_version_auto_timeout_ms': 2000, 'connections_max_idle_ms': 9 * 60 * 1000, # not implemented yet 'metric_reporters': [], 'metrics_num_samples': 2, 'metrics_sample_window_ms': 30000, 'metric_group_prefix': 'consumer', 'selector': selectors.DefaultSelector, 'exclude_internal_topics': True, 'sasl_mechanism': None, 'sasl_plain_username': None, 'sasl_plain_password': None, } def __init__(self, *topics, **configs): self.config = copy.copy(self.DEFAULT_CONFIG) for key in self.config: if key in configs: self.config[key] = configs.pop(key) # Only check for extra config keys in top-level class assert not configs, 'Unrecognized configs: %s' % configs deprecated = {'smallest': 'earliest', 'largest': 'latest'} if self.config['auto_offset_reset'] in deprecated: new_config = deprecated[self.config['auto_offset_reset']] log.warning('use auto_offset_reset=%s (%s is deprecated)', new_config, self.config['auto_offset_reset']) self.config['auto_offset_reset'] = new_config metrics_tags = {'client-id': self.config['client_id']} metric_config = MetricConfig(samples=self.config['metrics_num_samples'], time_window_ms=self.config['metrics_sample_window_ms'], tags=metrics_tags) reporters = [reporter() for reporter in self.config['metric_reporters']] self._metrics = Metrics(metric_config, reporters) # TODO _metrics likely needs to be passed to KafkaClient, etc. # api_version was previously a str. accept old format for now if isinstance(self.config['api_version'], str): str_version = self.config['api_version'] if str_version == 'auto': self.config['api_version'] = None else: self.config['api_version'] = tuple(map(int, str_version.split('.'))) log.warning('use api_version=%s [tuple] -- "%s" as str is deprecated', str(self.config['api_version']), str_version) self._client = KafkaClient(metrics=self._metrics, **self.config) # Get auto-discovered version from client if necessary if self.config['api_version'] is None: self.config['api_version'] = self._client.config['api_version'] self._subscription = SubscriptionState(self.config['auto_offset_reset']) self._fetcher = Fetcher( self._client, self._subscription, self._metrics, **self.config) self._coordinator = ConsumerCoordinator( self._client, self._subscription, self._metrics, assignors=self.config['partition_assignment_strategy'], **self.config) self._closed = False self._iterator = None self._consumer_timeout = float('inf') if topics: self._subscription.subscribe(topics=topics) self._client.set_topics(topics) def assign(self, partitions): """Manually assign a list of TopicPartitions to this consumer. Arguments: partitions (list of TopicPartition): assignment for this instance. Raises: IllegalStateError: if consumer has already called subscribe() Warning: It is not possible to use both manual partition assignment with assign() and group assignment with subscribe(). Note: This interface does not support incremental assignment and will replace the previous assignment (if there was one). Note: Manual topic assignment through this method does not use the consumer's group management functionality. As such, there will be no rebalance operation triggered when group membership or cluster and topic metadata change. """ self._subscription.assign_from_user(partitions) self._client.set_topics([tp.topic for tp in partitions]) def assignment(self): """Get the TopicPartitions currently assigned to this consumer. If partitions were directly assigned using assign(), then this will simply return the same partitions that were previously assigned. If topics were subscribed using subscribe(), then this will give the set of topic partitions currently assigned to the consumer (which may be none if the assignment hasn't happened yet, or if the partitions are in the process of being reassigned). Returns: set: {TopicPartition, ...} """ return self._subscription.assigned_partitions() def close(self): """Close the consumer, waiting indefinitely for any needed cleanup.""" if self._closed: return log.debug("Closing the KafkaConsumer.") self._closed = True self._coordinator.close() self._metrics.close() self._client.close() try: self.config['key_deserializer'].close() except AttributeError: pass try: self.config['value_deserializer'].close() except AttributeError: pass log.debug("The KafkaConsumer has closed.") def commit_async(self, offsets=None, callback=None): """Commit offsets to kafka asynchronously, optionally firing callback This commits offsets only to Kafka. The offsets committed using this API will be used on the first fetch after every rebalance and also on startup. As such, if you need to store offsets in anything other than Kafka, this API should not be used. To avoid re-processing the last message read if a consumer is restarted, the committed offset should be the next message your application should consume, i.e.: last_offset + 1. This is an asynchronous call and will not block. Any errors encountered are either passed to the callback (if provided) or discarded. Arguments: offsets (dict, optional): {TopicPartition: OffsetAndMetadata} dict to commit with the configured group_id. Defaults to current consumed offsets for all subscribed partitions. callback (callable, optional): called as callback(offsets, response) with response as either an Exception or a OffsetCommitResponse struct. This callback can be used to trigger custom actions when a commit request completes. Returns: kafka.future.Future """ assert self.config['api_version'] >= (0, 8, 1), 'Requires >= Kafka 0.8.1' assert self.config['group_id'] is not None, 'Requires group_id' if offsets is None: offsets = self._subscription.all_consumed_offsets() log.debug("Committing offsets: %s", offsets) future = self._coordinator.commit_offsets_async( offsets, callback=callback) return future def commit(self, offsets=None): """Commit offsets to kafka, blocking until success or error This commits offsets only to Kafka. The offsets committed using this API will be used on the first fetch after every rebalance and also on startup. As such, if you need to store offsets in anything other than Kafka, this API should not be used. To avoid re-processing the last message read if a consumer is restarted, the committed offset should be the next message your application should consume, i.e.: last_offset + 1. Blocks until either the commit succeeds or an unrecoverable error is encountered (in which case it is thrown to the caller). Currently only supports kafka-topic offset storage (not zookeeper) Arguments: offsets (dict, optional): {TopicPartition: OffsetAndMetadata} dict to commit with the configured group_id. Defaults to current consumed offsets for all subscribed partitions. """ assert self.config['api_version'] >= (0, 8, 1), 'Requires >= Kafka 0.8.1' assert self.config['group_id'] is not None, 'Requires group_id' if offsets is None: offsets = self._subscription.all_consumed_offsets() self._coordinator.commit_offsets_sync(offsets) def committed(self, partition): """Get the last committed offset for the given partition This offset will be used as the position for the consumer in the event of a failure. This call may block to do a remote call if the partition in question isn't assigned to this consumer or if the consumer hasn't yet initialized its cache of committed offsets. Arguments: partition (TopicPartition): the partition to check Returns: The last committed offset, or None if there was no prior commit. """ assert self.config['api_version'] >= (0, 8, 1), 'Requires >= Kafka 0.8.1' assert self.config['group_id'] is not None, 'Requires group_id' if not isinstance(partition, TopicPartition): raise TypeError('partition must be a TopicPartition namedtuple') if self._subscription.is_assigned(partition): committed = self._subscription.assignment[partition].committed if committed is None: self._coordinator.refresh_committed_offsets_if_needed() committed = self._subscription.assignment[partition].committed else: commit_map = self._coordinator.fetch_committed_offsets([partition]) if partition in commit_map: committed = commit_map[partition].offset else: committed = None return committed def topics(self): """Get all topics the user is authorized to view. Returns: set: topics """ cluster = self._client.cluster if self._client._metadata_refresh_in_progress and self._client._topics: future = cluster.request_update() self._client.poll(future=future) stash = cluster.need_all_topic_metadata cluster.need_all_topic_metadata = True future = cluster.request_update() self._client.poll(future=future) cluster.need_all_topic_metadata = stash return cluster.topics() def partitions_for_topic(self, topic): """Get metadata about the partitions for a given topic. Arguments: topic (str): topic to check Returns: set: partition ids """ return self._client.cluster.partitions_for_topic(topic) def poll(self, timeout_ms=0, max_records=None): """Fetch data from assigned topics / partitions. Records are fetched and returned in batches by topic-partition. On each poll, consumer will try to use the last consumed offset as the starting offset and fetch sequentially. The last consumed offset can be manually set through seek(partition, offset) or automatically set as the last committed offset for the subscribed list of partitions. Incompatible with iterator interface -- use one or the other, not both. Arguments: timeout_ms (int, optional): milliseconds spent waiting in poll if data is not available in the buffer. If 0, returns immediately with any records that are available currently in the buffer, else returns empty. Must not be negative. Default: 0 max_records (int, optional): The maximum number of records returned in a single call to :meth:`poll`. Default: Inherit value from max_poll_records. Returns: dict: topic to list of records since the last fetch for the subscribed list of topics and partitions """ assert timeout_ms >= 0, 'Timeout must not be negative' if max_records is None: max_records = self.config['max_poll_records'] # poll for new data until the timeout expires start = time.time() remaining = timeout_ms while True: records = self._poll_once(remaining, max_records) if records: return records elapsed_ms = (time.time() - start) * 1000 remaining = timeout_ms - elapsed_ms if remaining <= 0: return {} def _poll_once(self, timeout_ms, max_records): """ Do one round of polling. In addition to checking for new data, this does any needed heart-beating, auto-commits, and offset updates. Arguments: timeout_ms (int): The maximum time in milliseconds to block Returns: dict: map of topic to list of records (may be empty) """ if self._use_consumer_group(): self._coordinator.ensure_coordinator_known() self._coordinator.ensure_active_group() # 0.8.2 brokers support kafka-backed offset storage via group coordinator elif self.config['group_id'] is not None and self.config['api_version'] >= (0, 8, 2): self._coordinator.ensure_coordinator_known() # fetch positions if we have partitions we're subscribed to that we # don't know the offset for if not self._subscription.has_all_fetch_positions(): self._update_fetch_positions(self._subscription.missing_fetch_positions()) # if data is available already, e.g. from a previous network client # poll() call to commit, then just return it immediately records, partial = self._fetcher.fetched_records(max_records) if records: # before returning the fetched records, we can send off the # next round of fetches and avoid block waiting for their # responses to enable pipelining while the user is handling the # fetched records. if not partial: self._fetcher.send_fetches() return records # send any new fetches (won't resend pending fetches) self._fetcher.send_fetches() self._client.poll(timeout_ms=timeout_ms, sleep=True) records, _ = self._fetcher.fetched_records(max_records) return records def position(self, partition): """Get the offset of the next record that will be fetched Arguments: partition (TopicPartition): partition to check Returns: int: offset """ if not isinstance(partition, TopicPartition): raise TypeError('partition must be a TopicPartition namedtuple') assert self._subscription.is_assigned(partition), 'Partition is not assigned' offset = self._subscription.assignment[partition].position if offset is None: self._update_fetch_positions([partition]) offset = self._subscription.assignment[partition].position return offset def highwater(self, partition): """Last known highwater offset for a partition A highwater offset is the offset that will be assigned to the next message that is produced. It may be useful for calculating lag, by comparing with the reported position. Note that both position and highwater refer to the *next* offset -- i.e., highwater offset is one greater than the newest available message. Highwater offsets are returned in FetchResponse messages, so will not be available if no FetchRequests have been sent for this partition yet. Arguments: partition (TopicPartition): partition to check Returns: int or None: offset if available """ if not isinstance(partition, TopicPartition): raise TypeError('partition must be a TopicPartition namedtuple') assert self._subscription.is_assigned(partition), 'Partition is not assigned' return self._subscription.assignment[partition].highwater def pause(self, *partitions): """Suspend fetching from the requested partitions. Future calls to poll() will not return any records from these partitions until they have been resumed using resume(). Note that this method does not affect partition subscription. In particular, it does not cause a group rebalance when automatic assignment is used. Arguments: *partitions (TopicPartition): partitions to pause """ if not all([isinstance(p, TopicPartition) for p in partitions]): raise TypeError('partitions must be TopicPartition namedtuples') for partition in partitions: log.debug("Pausing partition %s", partition) self._subscription.pause(partition) def paused(self): """Get the partitions that were previously paused by a call to pause(). Returns: set: {partition (TopicPartition), ...} """ return self._subscription.paused_partitions() def resume(self, *partitions): """Resume fetching from the specified (paused) partitions. Arguments: *partitions (TopicPartition): partitions to resume """ if not all([isinstance(p, TopicPartition) for p in partitions]): raise TypeError('partitions must be TopicPartition namedtuples') for partition in partitions: log.debug("Resuming partition %s", partition) self._subscription.resume(partition) def seek(self, partition, offset): """Manually specify the fetch offset for a TopicPartition. Overrides the fetch offsets that the consumer will use on the next poll(). If this API is invoked for the same partition more than once, the latest offset will be used on the next poll(). Note that you may lose data if this API is arbitrarily used in the middle of consumption, to reset the fetch offsets. Arguments: partition (TopicPartition): partition for seek operation offset (int): message offset in partition Raises: AssertionError: if offset is not an int >= 0; or if partition is not currently assigned. """ if not isinstance(partition, TopicPartition): raise TypeError('partition must be a TopicPartition namedtuple') assert isinstance(offset, int) and offset >= 0, 'Offset must be >= 0' assert partition in self._subscription.assigned_partitions(), 'Unassigned partition' log.debug("Seeking to offset %s for partition %s", offset, partition) self._subscription.assignment[partition].seek(offset) def seek_to_beginning(self, *partitions): """Seek to the oldest available offset for partitions. Arguments: *partitions: optionally provide specific TopicPartitions, otherwise default to all assigned partitions Raises: AssertionError: if any partition is not currently assigned, or if no partitions are assigned """ if not all([isinstance(p, TopicPartition) for p in partitions]): raise TypeError('partitions must be TopicPartition namedtuples') if not partitions: partitions = self._subscription.assigned_partitions() assert partitions, 'No partitions are currently assigned' else: for p in partitions: assert p in self._subscription.assigned_partitions(), 'Unassigned partition' for tp in partitions: log.debug("Seeking to beginning of partition %s", tp) self._subscription.need_offset_reset(tp, OffsetResetStrategy.EARLIEST) def seek_to_end(self, *partitions): """Seek to the most recent available offset for partitions. Arguments: *partitions: optionally provide specific TopicPartitions, otherwise default to all assigned partitions Raises: AssertionError: if any partition is not currently assigned, or if no partitions are assigned """ if not all([isinstance(p, TopicPartition) for p in partitions]): raise TypeError('partitions must be TopicPartition namedtuples') if not partitions: partitions = self._subscription.assigned_partitions() assert partitions, 'No partitions are currently assigned' else: for p in partitions: assert p in self._subscription.assigned_partitions(), 'Unassigned partition' for tp in partitions: log.debug("Seeking to end of partition %s", tp) self._subscription.need_offset_reset(tp, OffsetResetStrategy.LATEST) def subscribe(self, topics=(), pattern=None, listener=None): """Subscribe to a list of topics, or a topic regex pattern Partitions will be dynamically assigned via a group coordinator. Topic subscriptions are not incremental: this list will replace the current assignment (if there is one). This method is incompatible with assign() Arguments: topics (list): List of topics for subscription. pattern (str): Pattern to match available topics. You must provide either topics or pattern, but not both. listener (ConsumerRebalanceListener): Optionally include listener callback, which will be called before and after each rebalance operation. As part of group management, the consumer will keep track of the list of consumers that belong to a particular group and will trigger a rebalance operation if one of the following events trigger: * Number of partitions change for any of the subscribed topics * Topic is created or deleted * An existing member of the consumer group dies * A new member is added to the consumer group When any of these events are triggered, the provided listener will be invoked first to indicate that the consumer's assignment has been revoked, and then again when the new assignment has been received. Note that this listener will immediately override any listener set in a previous call to subscribe. It is guaranteed, however, that the partitions revoked/assigned through this interface are from topics subscribed in this call. Raises: IllegalStateError: if called after previously calling assign() AssertionError: if neither topics or pattern is provided TypeError: if listener is not a ConsumerRebalanceListener """ # SubscriptionState handles error checking self._subscription.subscribe(topics=topics, pattern=pattern, listener=listener) # regex will need all topic metadata if pattern is not None: self._client.cluster.need_all_topic_metadata = True self._client.set_topics([]) self._client.cluster.request_update() log.debug("Subscribed to topic pattern: %s", pattern) else: self._client.cluster.need_all_topic_metadata = False self._client.set_topics(self._subscription.group_subscription()) log.debug("Subscribed to topic(s): %s", topics) def subscription(self): """Get the current topic subscription. Returns: set: {topic, ...} """ return self._subscription.subscription def unsubscribe(self): """Unsubscribe from all topics and clear all assigned partitions.""" self._subscription.unsubscribe() self._coordinator.close() self._client.cluster.need_all_topic_metadata = False self._client.set_topics([]) log.debug("Unsubscribed all topics or patterns and assigned partitions") def metrics(self, raw=False): """Warning: this is an unstable interface. It may change in future releases without warning""" if raw: return self._metrics.metrics metrics = {} for k, v in self._metrics.metrics.items(): if k.group not in metrics: metrics[k.group] = {} if k.name not in metrics[k.group]: metrics[k.group][k.name] = {} metrics[k.group][k.name] = v.value() return metrics def _use_consumer_group(self): """Return True iff this consumer can/should join a broker-coordinated group.""" if self.config['api_version'] < (0, 9): return False elif self.config['group_id'] is None: return False elif not self._subscription.partitions_auto_assigned(): return False return True def _update_fetch_positions(self, partitions): """ Set the fetch position to the committed position (if there is one) or reset it using the offset reset policy the user has configured. Arguments: partitions (List[TopicPartition]): The partitions that need updating fetch positions Raises: NoOffsetForPartitionError: If no offset is stored for a given partition and no offset reset policy is defined """ if (self.config['api_version'] >= (0, 8, 1) and self.config['group_id'] is not None): # refresh commits for all assigned partitions self._coordinator.refresh_committed_offsets_if_needed() # then do any offset lookups in case some positions are not known self._fetcher.update_fetch_positions(partitions) def _message_generator(self): assert self.assignment() or self.subscription() is not None, 'No topic subscription or manual partition assignment' while time.time() < self._consumer_timeout: if self._use_consumer_group(): self._coordinator.ensure_coordinator_known() self._coordinator.ensure_active_group() # 0.8.2 brokers support kafka-backed offset storage via group coordinator elif self.config['group_id'] is not None and self.config['api_version'] >= (0, 8, 2): self._coordinator.ensure_coordinator_known() # fetch offsets for any subscribed partitions that we arent tracking yet if not self._subscription.has_all_fetch_positions(): partitions = self._subscription.missing_fetch_positions() self._update_fetch_positions(partitions) poll_ms = 1000 * (self._consumer_timeout - time.time()) if not self._fetcher.in_flight_fetches(): poll_ms = 0 self._client.poll(timeout_ms=poll_ms, sleep=True) # We need to make sure we at least keep up with scheduled tasks, # like heartbeats, auto-commits, and metadata refreshes timeout_at = self._next_timeout() # Because the consumer client poll does not sleep unless blocking on # network IO, we need to explicitly sleep when we know we are idle # because we haven't been assigned any partitions to fetch / consume if self._use_consumer_group() and not self.assignment(): sleep_time = max(timeout_at - time.time(), 0) if sleep_time > 0 and not self._client.in_flight_request_count(): log.debug('No partitions assigned; sleeping for %s', sleep_time) time.sleep(sleep_time) continue # Short-circuit the fetch iterator if we are already timed out # to avoid any unintentional interaction with fetcher setup if time.time() > timeout_at: continue for msg in self._fetcher: yield msg if time.time() > timeout_at: log.debug("internal iterator timeout - breaking for poll") break # an else block on a for loop only executes if there was no break # so this should only be called on a StopIteration from the fetcher # and we assume that it is safe to init_fetches when fetcher is done # i.e., there are no more records stored internally else: self._fetcher.send_fetches() def _next_timeout(self): timeout = min(self._consumer_timeout, self._client._delayed_tasks.next_at() + time.time(), self._client.cluster.ttl() / 1000.0 + time.time()) # Although the delayed_tasks timeout above should cover processing # HeartbeatRequests, it is still possible that HeartbeatResponses # are left unprocessed during a long _fetcher iteration without # an intermediate poll(). And because tasks are responsible for # rescheduling themselves, an unprocessed response will prevent # the next heartbeat from being sent. This check should help # avoid that. if self._use_consumer_group(): heartbeat = time.time() + self._coordinator.heartbeat.ttl() timeout = min(timeout, heartbeat) return timeout def __iter__(self): # pylint: disable=non-iterator-returned return self def __next__(self): if not self._iterator: self._iterator = self._message_generator() self._set_consumer_timeout() try: return next(self._iterator) except StopIteration: self._iterator = None raise def _set_consumer_timeout(self): # consumer_timeout_ms can be used to stop iteration early if self.config['consumer_timeout_ms'] >= 0: self._consumer_timeout = time.time() + ( self.config['consumer_timeout_ms'] / 1000.0) # old KafkaConsumer methods are deprecated def configure(self, **configs): raise NotImplementedError( 'deprecated -- initialize a new consumer') def set_topic_partitions(self, *topics): raise NotImplementedError( 'deprecated -- use subscribe() or assign()') def fetch_messages(self): raise NotImplementedError( 'deprecated -- use poll() or iterator interface') def get_partition_offsets(self, topic, partition, request_time_ms, max_num_offsets): raise NotImplementedError( 'deprecated -- send an OffsetRequest with KafkaClient') def offsets(self, group=None): raise NotImplementedError('deprecated -- use committed(partition)') def task_done(self, message): raise NotImplementedError( 'deprecated -- commit offsets manually if needed') kafka-1.3.2/kafka/consumer/multiprocess.py0000644001271300127130000002674013025302127020377 0ustar dpowers00000000000000from __future__ import absolute_import from collections import namedtuple import logging from multiprocessing import Process, Manager as MPManager import time import warnings from kafka.vendor.six.moves import queue # pylint: disable=import-error from ..common import KafkaError from .base import ( Consumer, AUTO_COMMIT_MSG_COUNT, AUTO_COMMIT_INTERVAL, NO_MESSAGES_WAIT_TIME_SECONDS, FULL_QUEUE_WAIT_TIME_SECONDS, MAX_BACKOFF_SECONDS, ) from .simple import SimpleConsumer log = logging.getLogger(__name__) Events = namedtuple("Events", ["start", "pause", "exit"]) def _mp_consume(client, group, topic, message_queue, size, events, **consumer_options): """ A child process worker which consumes messages based on the notifications given by the controller process NOTE: Ideally, this should have been a method inside the Consumer class. However, multiprocessing module has issues in windows. The functionality breaks unless this function is kept outside of a class """ # Initial interval for retries in seconds. interval = 1 while not events.exit.is_set(): try: # Make the child processes open separate socket connections client.reinit() # We will start consumers without auto-commit. Auto-commit will be # done by the master controller process. consumer = SimpleConsumer(client, group, topic, auto_commit=False, auto_commit_every_n=None, auto_commit_every_t=None, **consumer_options) # Ensure that the consumer provides the partition information consumer.provide_partition_info() while True: # Wait till the controller indicates us to start consumption events.start.wait() # If we are asked to quit, do so if events.exit.is_set(): break # Consume messages and add them to the queue. If the controller # indicates a specific number of messages, follow that advice count = 0 message = consumer.get_message() if message: while True: try: message_queue.put(message, timeout=FULL_QUEUE_WAIT_TIME_SECONDS) break except queue.Full: if events.exit.is_set(): break count += 1 # We have reached the required size. The controller might have # more than what he needs. Wait for a while. # Without this logic, it is possible that we run into a big # loop consuming all available messages before the controller # can reset the 'start' event if count == size.value: events.pause.wait() else: # In case we did not receive any message, give up the CPU for # a while before we try again time.sleep(NO_MESSAGES_WAIT_TIME_SECONDS) consumer.stop() except KafkaError as e: # Retry with exponential backoff log.error("Problem communicating with Kafka (%s), retrying in %d seconds..." % (e, interval)) time.sleep(interval) interval = interval*2 if interval*2 < MAX_BACKOFF_SECONDS else MAX_BACKOFF_SECONDS class MultiProcessConsumer(Consumer): """ A consumer implementation that consumes partitions for a topic in parallel using multiple processes Arguments: client: a connected SimpleClient group: a name for this consumer, used for offset storage and must be unique If you are connecting to a server that does not support offset commit/fetch (any prior to 0.8.1.1), then you *must* set this to None topic: the topic to consume Keyword Arguments: partitions: An optional list of partitions to consume the data from auto_commit: default True. Whether or not to auto commit the offsets auto_commit_every_n: default 100. How many messages to consume before a commit auto_commit_every_t: default 5000. How much time (in milliseconds) to wait before commit num_procs: Number of processes to start for consuming messages. The available partitions will be divided among these processes partitions_per_proc: Number of partitions to be allocated per process (overrides num_procs) Auto commit details: If both auto_commit_every_n and auto_commit_every_t are set, they will reset one another when one is triggered. These triggers simply call the commit method on this class. A manual call to commit will also reset these triggers """ def __init__(self, client, group, topic, partitions=None, auto_commit=True, auto_commit_every_n=AUTO_COMMIT_MSG_COUNT, auto_commit_every_t=AUTO_COMMIT_INTERVAL, num_procs=1, partitions_per_proc=0, **simple_consumer_options): warnings.warn('This class has been deprecated and will be removed in a' ' future release. Use KafkaConsumer instead', DeprecationWarning) # Initiate the base consumer class super(MultiProcessConsumer, self).__init__( client, group, topic, partitions=partitions, auto_commit=auto_commit, auto_commit_every_n=auto_commit_every_n, auto_commit_every_t=auto_commit_every_t) # Variables for managing and controlling the data flow from # consumer child process to master manager = MPManager() self.queue = manager.Queue(1024) # Child consumers dump messages into this self.events = Events( start = manager.Event(), # Indicates the consumers to start fetch exit = manager.Event(), # Requests the consumers to shutdown pause = manager.Event()) # Requests the consumers to pause fetch self.size = manager.Value('i', 0) # Indicator of number of messages to fetch # dict.keys() returns a view in py3 + it's not a thread-safe operation # http://blog.labix.org/2008/06/27/watch-out-for-listdictkeys-in-python-3 # It's safer to copy dict as it only runs during the init. partitions = list(self.offsets.copy().keys()) # By default, start one consumer process for all partitions # The logic below ensures that # * we do not cross the num_procs limit # * we have an even distribution of partitions among processes if partitions_per_proc: num_procs = len(partitions) / partitions_per_proc if num_procs * partitions_per_proc < len(partitions): num_procs += 1 # The final set of chunks chunks = [partitions[proc::num_procs] for proc in range(num_procs)] self.procs = [] for chunk in chunks: options = {'partitions': list(chunk)} if simple_consumer_options: simple_consumer_options.pop('partitions', None) options.update(simple_consumer_options) args = (client.copy(), self.group, self.topic, self.queue, self.size, self.events) proc = Process(target=_mp_consume, args=args, kwargs=options) proc.daemon = True proc.start() self.procs.append(proc) def __repr__(self): return '' % \ (self.group, self.topic, len(self.procs)) def stop(self): # Set exit and start off all waiting consumers self.events.exit.set() self.events.pause.set() self.events.start.set() for proc in self.procs: proc.join() proc.terminate() super(MultiProcessConsumer, self).stop() def __iter__(self): """ Iterator to consume the messages available on this consumer """ # Trigger the consumer procs to start off. # We will iterate till there are no more messages available self.size.value = 0 self.events.pause.set() while True: self.events.start.set() try: # We will block for a small while so that the consumers get # a chance to run and put some messages in the queue # TODO: This is a hack and will make the consumer block for # at least one second. Need to find a better way of doing this partition, message = self.queue.get(block=True, timeout=1) except queue.Empty: break # Count, check and commit messages if necessary self.offsets[partition] = message.offset + 1 self.events.start.clear() self.count_since_commit += 1 self._auto_commit() yield message self.events.start.clear() def get_messages(self, count=1, block=True, timeout=10): """ Fetch the specified number of messages Keyword Arguments: count: Indicates the maximum number of messages to be fetched block: If True, the API will block till all messages are fetched. If block is a positive integer the API will block until that many messages are fetched. timeout: When blocking is requested the function will block for the specified time (in seconds) until count messages is fetched. If None, it will block forever. """ messages = [] # Give a size hint to the consumers. Each consumer process will fetch # a maximum of "count" messages. This will fetch more messages than # necessary, but these will not be committed to kafka. Also, the extra # messages can be provided in subsequent runs self.size.value = count self.events.pause.clear() if timeout is not None: max_time = time.time() + timeout new_offsets = {} while count > 0 and (timeout is None or timeout > 0): # Trigger consumption only if the queue is empty # By doing this, we will ensure that consumers do not # go into overdrive and keep consuming thousands of # messages when the user might need only a few if self.queue.empty(): self.events.start.set() block_next_call = block is True or block > len(messages) try: partition, message = self.queue.get(block_next_call, timeout) except queue.Empty: break _msg = (partition, message) if self.partition_info else message messages.append(_msg) new_offsets[partition] = message.offset + 1 count -= 1 if timeout is not None: timeout = max_time - time.time() self.size.value = 0 self.events.start.clear() self.events.pause.set() # Update and commit offsets if necessary self.offsets.update(new_offsets) self.count_since_commit += len(messages) self._auto_commit() return messages kafka-1.3.2/kafka/consumer/simple.py0000644001271300127130000004432313025302127017134 0ustar dpowers00000000000000from __future__ import absolute_import try: from itertools import zip_longest as izip_longest, repeat # pylint: disable=E0611 except ImportError: from itertools import izip_longest as izip_longest, repeat # pylint: disable=E0611 import logging import sys import time import warnings from kafka.vendor import six from kafka.vendor.six.moves import queue # pylint: disable=import-error from .base import ( Consumer, FETCH_DEFAULT_BLOCK_TIMEOUT, AUTO_COMMIT_MSG_COUNT, AUTO_COMMIT_INTERVAL, FETCH_MIN_BYTES, FETCH_BUFFER_SIZE_BYTES, MAX_FETCH_BUFFER_SIZE_BYTES, FETCH_MAX_WAIT_TIME, ITER_TIMEOUT_SECONDS, NO_MESSAGES_WAIT_TIME_SECONDS ) from ..common import ( FetchRequestPayload, KafkaError, OffsetRequestPayload, ConsumerFetchSizeTooSmall, UnknownTopicOrPartitionError, NotLeaderForPartitionError, OffsetOutOfRangeError, FailedPayloadsError, check_error ) from kafka.protocol.message import PartialMessage log = logging.getLogger(__name__) class FetchContext(object): """ Class for managing the state of a consumer during fetch """ def __init__(self, consumer, block, timeout): warnings.warn('deprecated - this class will be removed in a future' ' release', DeprecationWarning) self.consumer = consumer self.block = block if block: if not timeout: timeout = FETCH_DEFAULT_BLOCK_TIMEOUT self.timeout = timeout * 1000 def __enter__(self): """Set fetch values based on blocking status""" self.orig_fetch_max_wait_time = self.consumer.fetch_max_wait_time self.orig_fetch_min_bytes = self.consumer.fetch_min_bytes if self.block: self.consumer.fetch_max_wait_time = self.timeout self.consumer.fetch_min_bytes = 1 else: self.consumer.fetch_min_bytes = 0 def __exit__(self, type, value, traceback): """Reset values""" self.consumer.fetch_max_wait_time = self.orig_fetch_max_wait_time self.consumer.fetch_min_bytes = self.orig_fetch_min_bytes class SimpleConsumer(Consumer): """ A simple consumer implementation that consumes all/specified partitions for a topic Arguments: client: a connected SimpleClient group: a name for this consumer, used for offset storage and must be unique If you are connecting to a server that does not support offset commit/fetch (any prior to 0.8.1.1), then you *must* set this to None topic: the topic to consume Keyword Arguments: partitions: An optional list of partitions to consume the data from auto_commit: default True. Whether or not to auto commit the offsets auto_commit_every_n: default 100. How many messages to consume before a commit auto_commit_every_t: default 5000. How much time (in milliseconds) to wait before commit fetch_size_bytes: number of bytes to request in a FetchRequest buffer_size: default 4K. Initial number of bytes to tell kafka we have available. This will double as needed. max_buffer_size: default 16K. Max number of bytes to tell kafka we have available. None means no limit. iter_timeout: default None. How much time (in seconds) to wait for a message in the iterator before exiting. None means no timeout, so it will wait forever. auto_offset_reset: default largest. Reset partition offsets upon OffsetOutOfRangeError. Valid values are largest and smallest. Otherwise, do not reset the offsets and raise OffsetOutOfRangeError. Auto commit details: If both auto_commit_every_n and auto_commit_every_t are set, they will reset one another when one is triggered. These triggers simply call the commit method on this class. A manual call to commit will also reset these triggers """ def __init__(self, client, group, topic, auto_commit=True, partitions=None, auto_commit_every_n=AUTO_COMMIT_MSG_COUNT, auto_commit_every_t=AUTO_COMMIT_INTERVAL, fetch_size_bytes=FETCH_MIN_BYTES, buffer_size=FETCH_BUFFER_SIZE_BYTES, max_buffer_size=MAX_FETCH_BUFFER_SIZE_BYTES, iter_timeout=None, auto_offset_reset='largest'): warnings.warn('deprecated - this class will be removed in a future' ' release. Use KafkaConsumer instead.', DeprecationWarning) super(SimpleConsumer, self).__init__( client, group, topic, partitions=partitions, auto_commit=auto_commit, auto_commit_every_n=auto_commit_every_n, auto_commit_every_t=auto_commit_every_t) if max_buffer_size is not None and buffer_size > max_buffer_size: raise ValueError('buffer_size (%d) is greater than ' 'max_buffer_size (%d)' % (buffer_size, max_buffer_size)) self.buffer_size = buffer_size self.max_buffer_size = max_buffer_size self.fetch_max_wait_time = FETCH_MAX_WAIT_TIME self.fetch_min_bytes = fetch_size_bytes self.fetch_offsets = self.offsets.copy() self.iter_timeout = iter_timeout self.auto_offset_reset = auto_offset_reset self.queue = queue.Queue() def __repr__(self): return '' % \ (self.group, self.topic, str(self.offsets.keys())) def reset_partition_offset(self, partition): """Update offsets using auto_offset_reset policy (smallest|largest) Arguments: partition (int): the partition for which offsets should be updated Returns: Updated offset on success, None on failure """ LATEST = -1 EARLIEST = -2 if self.auto_offset_reset == 'largest': reqs = [OffsetRequestPayload(self.topic, partition, LATEST, 1)] elif self.auto_offset_reset == 'smallest': reqs = [OffsetRequestPayload(self.topic, partition, EARLIEST, 1)] else: # Let's raise an reasonable exception type if user calls # outside of an exception context if sys.exc_info() == (None, None, None): raise OffsetOutOfRangeError('Cannot reset partition offsets without a ' 'valid auto_offset_reset setting ' '(largest|smallest)') # Otherwise we should re-raise the upstream exception # b/c it typically includes additional data about # the request that triggered it, and we do not want to drop that raise # pylint: disable=E0704 # send_offset_request log.info('Resetting topic-partition offset to %s for %s:%d', self.auto_offset_reset, self.topic, partition) try: (resp, ) = self.client.send_offset_request(reqs) except KafkaError as e: log.error('%s sending offset request for %s:%d', e.__class__.__name__, self.topic, partition) else: self.offsets[partition] = resp.offsets[0] self.fetch_offsets[partition] = resp.offsets[0] return resp.offsets[0] def seek(self, offset, whence=None, partition=None): """ Alter the current offset in the consumer, similar to fseek Arguments: offset: how much to modify the offset whence: where to modify it from, default is None * None is an absolute offset * 0 is relative to the earliest available offset (head) * 1 is relative to the current offset * 2 is relative to the latest known offset (tail) partition: modify which partition, default is None. If partition is None, would modify all partitions. """ if whence is None: # set an absolute offset if partition is None: for tmp_partition in self.offsets: self.offsets[tmp_partition] = offset else: self.offsets[partition] = offset elif whence == 1: # relative to current position if partition is None: for tmp_partition, _offset in self.offsets.items(): self.offsets[tmp_partition] = _offset + offset else: self.offsets[partition] += offset elif whence in (0, 2): # relative to beginning or end reqs = [] deltas = {} if partition is None: # divide the request offset by number of partitions, # distribute the remained evenly (delta, rem) = divmod(offset, len(self.offsets)) for tmp_partition, r in izip_longest(self.offsets.keys(), repeat(1, rem), fillvalue=0): deltas[tmp_partition] = delta + r for tmp_partition in self.offsets.keys(): if whence == 0: reqs.append(OffsetRequestPayload(self.topic, tmp_partition, -2, 1)) elif whence == 2: reqs.append(OffsetRequestPayload(self.topic, tmp_partition, -1, 1)) else: pass else: deltas[partition] = offset if whence == 0: reqs.append(OffsetRequestPayload(self.topic, partition, -2, 1)) elif whence == 2: reqs.append(OffsetRequestPayload(self.topic, partition, -1, 1)) else: pass resps = self.client.send_offset_request(reqs) for resp in resps: self.offsets[resp.partition] = \ resp.offsets[0] + deltas[resp.partition] else: raise ValueError('Unexpected value for `whence`, %d' % whence) # Reset queue and fetch offsets since they are invalid self.fetch_offsets = self.offsets.copy() self.count_since_commit += 1 if self.auto_commit: self.commit() self.queue = queue.Queue() def get_messages(self, count=1, block=True, timeout=0.1): """ Fetch the specified number of messages Keyword Arguments: count: Indicates the maximum number of messages to be fetched block: If True, the API will block till all messages are fetched. If block is a positive integer the API will block until that many messages are fetched. timeout: When blocking is requested the function will block for the specified time (in seconds) until count messages is fetched. If None, it will block forever. """ messages = [] if timeout is not None: timeout += time.time() new_offsets = {} log.debug('getting %d messages', count) while len(messages) < count: block_time = timeout - time.time() log.debug('calling _get_message block=%s timeout=%s', block, block_time) block_next_call = block is True or block > len(messages) result = self._get_message(block_next_call, block_time, get_partition_info=True, update_offset=False) log.debug('got %s from _get_messages', result) if not result: if block_next_call and (timeout is None or time.time() <= timeout): continue break partition, message = result _msg = (partition, message) if self.partition_info else message messages.append(_msg) new_offsets[partition] = message.offset + 1 # Update and commit offsets if necessary self.offsets.update(new_offsets) self.count_since_commit += len(messages) self._auto_commit() log.debug('got %d messages: %s', len(messages), messages) return messages def get_message(self, block=True, timeout=0.1, get_partition_info=None): return self._get_message(block, timeout, get_partition_info) def _get_message(self, block=True, timeout=0.1, get_partition_info=None, update_offset=True): """ If no messages can be fetched, returns None. If get_partition_info is None, it defaults to self.partition_info If get_partition_info is True, returns (partition, message) If get_partition_info is False, returns message """ start_at = time.time() while self.queue.empty(): # We're out of messages, go grab some more. log.debug('internal queue empty, fetching more messages') with FetchContext(self, block, timeout): self._fetch() if not block or time.time() > (start_at + timeout): break try: partition, message = self.queue.get_nowait() if update_offset: # Update partition offset self.offsets[partition] = message.offset + 1 # Count, check and commit messages if necessary self.count_since_commit += 1 self._auto_commit() if get_partition_info is None: get_partition_info = self.partition_info if get_partition_info: return partition, message else: return message except queue.Empty: log.debug('internal queue empty after fetch - returning None') return None def __iter__(self): if self.iter_timeout is None: timeout = ITER_TIMEOUT_SECONDS else: timeout = self.iter_timeout while True: message = self.get_message(True, timeout) if message: yield message elif self.iter_timeout is None: # We did not receive any message yet but we don't have a # timeout, so give up the CPU for a while before trying again time.sleep(NO_MESSAGES_WAIT_TIME_SECONDS) else: # Timed out waiting for a message break def _fetch(self): # Create fetch request payloads for all the partitions partitions = dict((p, self.buffer_size) for p in self.fetch_offsets.keys()) while partitions: requests = [] for partition, buffer_size in six.iteritems(partitions): requests.append(FetchRequestPayload(self.topic, partition, self.fetch_offsets[partition], buffer_size)) # Send request responses = self.client.send_fetch_request( requests, max_wait_time=int(self.fetch_max_wait_time), min_bytes=self.fetch_min_bytes, fail_on_error=False ) retry_partitions = {} for resp in responses: try: check_error(resp) except UnknownTopicOrPartitionError: log.error('UnknownTopicOrPartitionError for %s:%d', resp.topic, resp.partition) self.client.reset_topic_metadata(resp.topic) raise except NotLeaderForPartitionError: log.error('NotLeaderForPartitionError for %s:%d', resp.topic, resp.partition) self.client.reset_topic_metadata(resp.topic) continue except OffsetOutOfRangeError: log.warning('OffsetOutOfRangeError for %s:%d. ' 'Resetting partition offset...', resp.topic, resp.partition) self.reset_partition_offset(resp.partition) # Retry this partition retry_partitions[resp.partition] = partitions[resp.partition] continue except FailedPayloadsError as e: log.warning('FailedPayloadsError for %s:%d', e.payload.topic, e.payload.partition) # Retry this partition retry_partitions[e.payload.partition] = partitions[e.payload.partition] continue partition = resp.partition buffer_size = partitions[partition] # Check for partial message if resp.messages and isinstance(resp.messages[-1].message, PartialMessage): # If buffer is at max and all we got was a partial message # raise ConsumerFetchSizeTooSmall if (self.max_buffer_size is not None and buffer_size == self.max_buffer_size and len(resp.messages) == 1): log.error('Max fetch size %d too small', self.max_buffer_size) raise ConsumerFetchSizeTooSmall() if self.max_buffer_size is None: buffer_size *= 2 else: buffer_size = min(buffer_size * 2, self.max_buffer_size) log.warning('Fetch size too small, increase to %d (2x) ' 'and retry', buffer_size) retry_partitions[partition] = buffer_size resp.messages.pop() for message in resp.messages: if message.offset < self.fetch_offsets[partition]: log.debug('Skipping message %s because its offset is less than the consumer offset', message) continue # Put the message in our queue self.queue.put((partition, message)) self.fetch_offsets[partition] = message.offset + 1 partitions = retry_partitions kafka-1.3.2/kafka/consumer/subscription_state.py0000644001271300127130000004663113031057471021601 0ustar dpowers00000000000000from __future__ import absolute_import import abc import logging import re from kafka.vendor import six from kafka.errors import IllegalStateError from kafka.protocol.offset import OffsetResetStrategy from kafka.structs import OffsetAndMetadata log = logging.getLogger(__name__) class SubscriptionState(object): """ A class for tracking the topics, partitions, and offsets for the consumer. A partition is "assigned" either directly with assign_from_user() (manual assignment) or with assign_from_subscribed() (automatic assignment from subscription). Once assigned, the partition is not considered "fetchable" until its initial position has been set with seek(). Fetchable partitions track a fetch position which is used to set the offset of the next fetch, and a consumed position which is the last offset that has been returned to the user. You can suspend fetching from a partition through pause() without affecting the fetched/consumed offsets. The partition will remain unfetchable until the resume() is used. You can also query the pause state independently with is_paused(). Note that pause state as well as fetch/consumed positions are not preserved when partition assignment is changed whether directly by the user or through a group rebalance. This class also maintains a cache of the latest commit position for each of the assigned partitions. This is updated through committed() and can be used to set the initial fetch position (e.g. Fetcher._reset_offset() ). """ _SUBSCRIPTION_EXCEPTION_MESSAGE = ( "You must choose only one way to configure your consumer:" " (1) subscribe to specific topics by name," " (2) subscribe to topics matching a regex pattern," " (3) assign itself specific topic-partitions.") def __init__(self, offset_reset_strategy='earliest'): """Initialize a SubscriptionState instance Keyword Arguments: offset_reset_strategy: 'earliest' or 'latest', otherwise exception will be raised when fetching an offset that is no longer available. Default: 'earliest' """ try: offset_reset_strategy = getattr(OffsetResetStrategy, offset_reset_strategy.upper()) except AttributeError: log.warning('Unrecognized offset_reset_strategy, using NONE') offset_reset_strategy = OffsetResetStrategy.NONE self._default_offset_reset_strategy = offset_reset_strategy self.subscription = None # set() or None self.subscribed_pattern = None # regex str or None self._group_subscription = set() self._user_assignment = set() self.assignment = dict() self.needs_partition_assignment = False self.listener = None # initialize to true for the consumers to fetch offset upon starting up self.needs_fetch_committed_offsets = True def subscribe(self, topics=(), pattern=None, listener=None): """Subscribe to a list of topics, or a topic regex pattern. Partitions will be dynamically assigned via a group coordinator. Topic subscriptions are not incremental: this list will replace the current assignment (if there is one). This method is incompatible with assign_from_user() Arguments: topics (list): List of topics for subscription. pattern (str): Pattern to match available topics. You must provide either topics or pattern, but not both. listener (ConsumerRebalanceListener): Optionally include listener callback, which will be called before and after each rebalance operation. As part of group management, the consumer will keep track of the list of consumers that belong to a particular group and will trigger a rebalance operation if one of the following events trigger: * Number of partitions change for any of the subscribed topics * Topic is created or deleted * An existing member of the consumer group dies * A new member is added to the consumer group When any of these events are triggered, the provided listener will be invoked first to indicate that the consumer's assignment has been revoked, and then again when the new assignment has been received. Note that this listener will immediately override any listener set in a previous call to subscribe. It is guaranteed, however, that the partitions revoked/assigned through this interface are from topics subscribed in this call. """ if self._user_assignment or (topics and pattern): raise IllegalStateError(self._SUBSCRIPTION_EXCEPTION_MESSAGE) assert topics or pattern, 'Must provide topics or pattern' if pattern: log.info('Subscribing to pattern: /%s/', pattern) self.subscription = set() self.subscribed_pattern = re.compile(pattern) else: self.change_subscription(topics) if listener and not isinstance(listener, ConsumerRebalanceListener): raise TypeError('listener must be a ConsumerRebalanceListener') self.listener = listener def change_subscription(self, topics): """Change the topic subscription. Arguments: topics (list of str): topics for subscription Raises: IllegalStateErrror: if assign_from_user has been used already TypeError: if a non-str topic is given """ if self._user_assignment: raise IllegalStateError(self._SUBSCRIPTION_EXCEPTION_MESSAGE) if isinstance(topics, six.string_types): topics = [topics] if self.subscription == set(topics): log.warning("subscription unchanged by change_subscription(%s)", topics) return if any(not isinstance(t, six.string_types) for t in topics): raise TypeError('All topics must be strings') log.info('Updating subscribed topics to: %s', topics) self.subscription = set(topics) self._group_subscription.update(topics) self.needs_partition_assignment = True # Remove any assigned partitions which are no longer subscribed to for tp in set(self.assignment.keys()): if tp.topic not in self.subscription: del self.assignment[tp] def group_subscribe(self, topics): """Add topics to the current group subscription. This is used by the group leader to ensure that it receives metadata updates for all topics that any member of the group is subscribed to. Arguments: topics (list of str): topics to add to the group subscription """ if self._user_assignment: raise IllegalStateError(self._SUBSCRIPTION_EXCEPTION_MESSAGE) self._group_subscription.update(topics) def mark_for_reassignment(self): if self._user_assignment: raise IllegalStateError(self._SUBSCRIPTION_EXCEPTION_MESSAGE) assert self.subscription is not None, 'Subscription required' self._group_subscription.intersection_update(self.subscription) self.needs_partition_assignment = True def assign_from_user(self, partitions): """Manually assign a list of TopicPartitions to this consumer. This interface does not allow for incremental assignment and will replace the previous assignment (if there was one). Manual topic assignment through this method does not use the consumer's group management functionality. As such, there will be no rebalance operation triggered when group membership or cluster and topic metadata change. Note that it is not possible to use both manual partition assignment with assign() and group assignment with subscribe(). Arguments: partitions (list of TopicPartition): assignment for this instance. Raises: IllegalStateError: if consumer has already called subscribe() """ if self.subscription is not None: raise IllegalStateError(self._SUBSCRIPTION_EXCEPTION_MESSAGE) self._user_assignment.clear() self._user_assignment.update(partitions) for partition in partitions: if partition not in self.assignment: self._add_assigned_partition(partition) for tp in set(self.assignment.keys()) - self._user_assignment: del self.assignment[tp] self.needs_partition_assignment = False self.needs_fetch_committed_offsets = True def assign_from_subscribed(self, assignments): """Update the assignment to the specified partitions This method is called by the coordinator to dynamically assign partitions based on the consumer's topic subscription. This is different from assign_from_user() which directly sets the assignment from a user-supplied TopicPartition list. Arguments: assignments (list of TopicPartition): partitions to assign to this consumer instance. """ if self.subscription is None: raise IllegalStateError(self._SUBSCRIPTION_EXCEPTION_MESSAGE) for tp in assignments: if tp.topic not in self.subscription: raise ValueError("Assigned partition %s for non-subscribed topic." % tp) self.assignment.clear() for tp in assignments: self._add_assigned_partition(tp) self.needs_partition_assignment = False log.info("Updated partition assignment: %s", assignments) def unsubscribe(self): """Clear all topic subscriptions and partition assignments""" self.subscription = None self._user_assignment.clear() self.assignment.clear() self.needs_partition_assignment = True self.subscribed_pattern = None def group_subscription(self): """Get the topic subscription for the group. For the leader, this will include the union of all member subscriptions. For followers, it is the member's subscription only. This is used when querying topic metadata to detect metadata changes that would require rebalancing (the leader fetches metadata for all topics in the group so that it can do partition assignment). Returns: set: topics """ return self._group_subscription def seek(self, partition, offset): """Manually specify the fetch offset for a TopicPartition. Overrides the fetch offsets that the consumer will use on the next poll(). If this API is invoked for the same partition more than once, the latest offset will be used on the next poll(). Note that you may lose data if this API is arbitrarily used in the middle of consumption, to reset the fetch offsets. Arguments: partition (TopicPartition): partition for seek operation offset (int): message offset in partition """ self.assignment[partition].seek(offset) def assigned_partitions(self): """Return set of TopicPartitions in current assignment.""" return set(self.assignment.keys()) def paused_partitions(self): """Return current set of paused TopicPartitions.""" return set(partition for partition in self.assignment if self.is_paused(partition)) def fetchable_partitions(self): """Return set of TopicPartitions that should be Fetched.""" fetchable = set() for partition, state in six.iteritems(self.assignment): if state.is_fetchable(): fetchable.add(partition) return fetchable def partitions_auto_assigned(self): """Return True unless user supplied partitions manually.""" return self.subscription is not None def all_consumed_offsets(self): """Returns consumed offsets as {TopicPartition: OffsetAndMetadata}""" all_consumed = {} for partition, state in six.iteritems(self.assignment): if state.has_valid_position: all_consumed[partition] = OffsetAndMetadata(state.position, '') return all_consumed def need_offset_reset(self, partition, offset_reset_strategy=None): """Mark partition for offset reset using specified or default strategy. Arguments: partition (TopicPartition): partition to mark offset_reset_strategy (OffsetResetStrategy, optional) """ if offset_reset_strategy is None: offset_reset_strategy = self._default_offset_reset_strategy self.assignment[partition].await_reset(offset_reset_strategy) def has_default_offset_reset_policy(self): """Return True if default offset reset policy is Earliest or Latest""" return self._default_offset_reset_strategy != OffsetResetStrategy.NONE def is_offset_reset_needed(self, partition): return self.assignment[partition].awaiting_reset def has_all_fetch_positions(self): for state in self.assignment.values(): if not state.has_valid_position: return False return True def missing_fetch_positions(self): missing = set() for partition, state in six.iteritems(self.assignment): if not state.has_valid_position: missing.add(partition) return missing def is_assigned(self, partition): return partition in self.assignment def is_paused(self, partition): return partition in self.assignment and self.assignment[partition].paused def is_fetchable(self, partition): return partition in self.assignment and self.assignment[partition].is_fetchable() def pause(self, partition): self.assignment[partition].pause() def resume(self, partition): self.assignment[partition].resume() def _add_assigned_partition(self, partition): self.assignment[partition] = TopicPartitionState() class TopicPartitionState(object): def __init__(self): self.committed = None # last committed position self.has_valid_position = False # whether we have valid position self.paused = False # whether this partition has been paused by the user self.awaiting_reset = False # whether we are awaiting reset self.reset_strategy = None # the reset strategy if awaitingReset is set self._position = None # offset exposed to the user self.highwater = None self.drop_pending_message_set = False def _set_position(self, offset): assert self.has_valid_position, 'Valid position required' self._position = offset def _get_position(self): return self._position position = property(_get_position, _set_position, None, "last position") def await_reset(self, strategy): self.awaiting_reset = True self.reset_strategy = strategy self._position = None self.has_valid_position = False def seek(self, offset): self._position = offset self.awaiting_reset = False self.reset_strategy = None self.has_valid_position = True self.drop_pending_message_set = True def pause(self): self.paused = True def resume(self): self.paused = False def is_fetchable(self): return not self.paused and self.has_valid_position class ConsumerRebalanceListener(object): """ A callback interface that the user can implement to trigger custom actions when the set of partitions assigned to the consumer changes. This is applicable when the consumer is having Kafka auto-manage group membership. If the consumer's directly assign partitions, those partitions will never be reassigned and this callback is not applicable. When Kafka is managing the group membership, a partition re-assignment will be triggered any time the members of the group changes or the subscription of the members changes. This can occur when processes die, new process instances are added or old instances come back to life after failure. Rebalances can also be triggered by changes affecting the subscribed topics (e.g. when then number of partitions is administratively adjusted). There are many uses for this functionality. One common use is saving offsets in a custom store. By saving offsets in the on_partitions_revoked(), call we can ensure that any time partition assignment changes the offset gets saved. Another use is flushing out any kind of cache of intermediate results the consumer may be keeping. For example, consider a case where the consumer is subscribed to a topic containing user page views, and the goal is to count the number of page views per users for each five minute window. Let's say the topic is partitioned by the user id so that all events for a particular user will go to a single consumer instance. The consumer can keep in memory a running tally of actions per user and only flush these out to a remote data store when its cache gets too big. However if a partition is reassigned it may want to automatically trigger a flush of this cache, before the new owner takes over consumption. This callback will execute in the user thread as part of the Consumer.poll() whenever partition assignment changes. It is guaranteed that all consumer processes will invoke on_partitions_revoked() prior to any process invoking on_partitions_assigned(). So if offsets or other state is saved in the on_partitions_revoked() call, it should be saved by the time the process taking over that partition has their on_partitions_assigned() callback called to load the state. """ __metaclass__ = abc.ABCMeta @abc.abstractmethod def on_partitions_revoked(self, revoked): """ A callback method the user can implement to provide handling of offset commits to a customized store on the start of a rebalance operation. This method will be called before a rebalance operation starts and after the consumer stops fetching data. It is recommended that offsets should be committed in this callback to either Kafka or a custom offset store to prevent duplicate data. NOTE: This method is only called before rebalances. It is not called prior to KafkaConsumer.close() Arguments: revoked (list of TopicPartition): the partitions that were assigned to the consumer on the last rebalance """ pass @abc.abstractmethod def on_partitions_assigned(self, assigned): """ A callback method the user can implement to provide handling of customized offsets on completion of a successful partition re-assignment. This method will be called after an offset re-assignment completes and before the consumer starts fetching data. It is guaranteed that all the processes in a consumer group will execute their on_partitions_revoked() callback before any instance executes its on_partitions_assigned() callback. Arguments: assigned (list of TopicPartition): the partitions assigned to the consumer (may include partitions that were previously assigned) """ pass kafka-1.3.2/kafka/context.py0000644001271300127130000001355313025302127015475 0ustar dpowers00000000000000""" Context manager to commit/rollback consumer offsets. """ from __future__ import absolute_import from logging import getLogger from kafka.errors import check_error, OffsetOutOfRangeError from kafka.structs import OffsetCommitRequestPayload class OffsetCommitContext(object): """ Provides commit/rollback semantics around a `SimpleConsumer`. Usage assumes that `auto_commit` is disabled, that messages are consumed in batches, and that the consuming process will record its own successful processing of each message. Both the commit and rollback operations respect a "high-water mark" to ensure that last unsuccessfully processed message will be retried. Example: .. code:: python consumer = SimpleConsumer(client, group, topic, auto_commit=False) consumer.provide_partition_info() consumer.fetch_last_known_offsets() while some_condition: with OffsetCommitContext(consumer) as context: messages = consumer.get_messages(count, block=False) for partition, message in messages: if can_process(message): context.mark(partition, message.offset) else: break if not context: sleep(delay) These semantics allow for deferred message processing (e.g. if `can_process` compares message time to clock time) and for repeated processing of the last unsuccessful message (until some external error is resolved). """ def __init__(self, consumer): """ :param consumer: an instance of `SimpleConsumer` """ self.consumer = consumer self.initial_offsets = None self.high_water_mark = None self.logger = getLogger("kafka.context") def mark(self, partition, offset): """ Set the high-water mark in the current context. In order to know the current partition, it is helpful to initialize the consumer to provide partition info via: .. code:: python consumer.provide_partition_info() """ max_offset = max(offset + 1, self.high_water_mark.get(partition, 0)) self.logger.debug("Setting high-water mark to: %s", {partition: max_offset}) self.high_water_mark[partition] = max_offset def __nonzero__(self): """ Return whether any operations were marked in the context. """ return bool(self.high_water_mark) def __enter__(self): """ Start a new context: - Record the initial offsets for rollback - Reset the high-water mark """ self.initial_offsets = dict(self.consumer.offsets) self.high_water_mark = dict() self.logger.debug("Starting context at: %s", self.initial_offsets) return self def __exit__(self, exc_type, exc_value, traceback): """ End a context. - If there was no exception, commit up to the current high-water mark. - If there was an offset of range error, attempt to find the correct initial offset. - If there was any other error, roll back to the initial offsets. """ if exc_type is None: self.commit() elif isinstance(exc_value, OffsetOutOfRangeError): self.handle_out_of_range() return True else: self.rollback() def commit(self): """ Commit this context's offsets: - If the high-water mark has moved, commit up to and position the consumer at the high-water mark. - Otherwise, reset to the consumer to the initial offsets. """ if self.high_water_mark: self.logger.info("Committing offsets: %s", self.high_water_mark) self.commit_partition_offsets(self.high_water_mark) self.update_consumer_offsets(self.high_water_mark) else: self.update_consumer_offsets(self.initial_offsets) def rollback(self): """ Rollback this context: - Position the consumer at the initial offsets. """ self.logger.info("Rolling back context: %s", self.initial_offsets) self.update_consumer_offsets(self.initial_offsets) def commit_partition_offsets(self, partition_offsets): """ Commit explicit partition/offset pairs. """ self.logger.debug("Committing partition offsets: %s", partition_offsets) commit_requests = [ OffsetCommitRequestPayload(self.consumer.topic, partition, offset, None) for partition, offset in partition_offsets.items() ] commit_responses = self.consumer.client.send_offset_commit_request( self.consumer.group, commit_requests, ) for commit_response in commit_responses: check_error(commit_response) def update_consumer_offsets(self, partition_offsets): """ Update consumer offsets to explicit positions. """ self.logger.debug("Updating consumer offsets to: %s", partition_offsets) for partition, offset in partition_offsets.items(): self.consumer.offsets[partition] = offset # consumer keeps other offset states beyond its `offsets` dictionary, # a relative seek with zero delta forces the consumer to reset to the # current value of the `offsets` dictionary self.consumer.seek(0, 1) def handle_out_of_range(self): """ Handle out of range condition by seeking to the beginning of valid ranges. This assumes that an out of range doesn't happen by seeking past the end of valid ranges -- which is far less likely. """ self.logger.info("Seeking beginning of partition on out of range error") self.consumer.seek(0, 0) kafka-1.3.2/kafka/coordinator/0000755001271300127130000000000013031057517015762 5ustar dpowers00000000000000kafka-1.3.2/kafka/coordinator/__init__.py0000644001271300127130000000000012661137006020061 0ustar dpowers00000000000000kafka-1.3.2/kafka/coordinator/assignors/0000755001271300127130000000000013031057517017772 5ustar dpowers00000000000000kafka-1.3.2/kafka/coordinator/assignors/__init__.py0000644001271300127130000000000012661137006022071 0ustar dpowers00000000000000kafka-1.3.2/kafka/coordinator/assignors/abstract.py0000644001271300127130000000274313025302127022146 0ustar dpowers00000000000000from __future__ import absolute_import import abc import logging log = logging.getLogger(__name__) class AbstractPartitionAssignor(object): """ Abstract assignor implementation which does some common grunt work (in particular collecting partition counts which are always needed in assignors). """ @abc.abstractproperty def name(self): """.name should be a string identifying the assignor""" pass @abc.abstractmethod def assign(self, cluster, members): """Perform group assignment given cluster metadata and member subscriptions Arguments: cluster (ClusterMetadata): metadata for use in assignment members (dict of {member_id: MemberMetadata}): decoded metadata for each member in the group. Returns: dict: {member_id: MemberAssignment} """ pass @abc.abstractmethod def metadata(self, topics): """Generate ProtocolMetadata to be submitted via JoinGroupRequest. Arguments: topics (set): a member's subscribed topics Returns: MemberMetadata struct """ pass @abc.abstractmethod def on_assignment(self, assignment): """Callback that runs on each assignment. This method can be used to update internal state, if any, of the partition assignor. Arguments: assignment (MemberAssignment): the member's assignment """ pass kafka-1.3.2/kafka/coordinator/assignors/range.py0000644001271300127130000000562713025302127021443 0ustar dpowers00000000000000from __future__ import absolute_import import collections import logging from kafka.vendor import six from .abstract import AbstractPartitionAssignor from ..protocol import ConsumerProtocolMemberMetadata, ConsumerProtocolMemberAssignment log = logging.getLogger(__name__) class RangePartitionAssignor(AbstractPartitionAssignor): """ The range assignor works on a per-topic basis. For each topic, we lay out the available partitions in numeric order and the consumers in lexicographic order. We then divide the number of partitions by the total number of consumers to determine the number of partitions to assign to each consumer. If it does not evenly divide, then the first few consumers will have one extra partition. For example, suppose there are two consumers C0 and C1, two topics t0 and t1, and each topic has 3 partitions, resulting in partitions t0p0, t0p1, t0p2, t1p0, t1p1, and t1p2. The assignment will be: C0: [t0p0, t0p1, t1p0, t1p1] C1: [t0p2, t1p2] """ name = 'range' version = 0 @classmethod def assign(cls, cluster, member_metadata): consumers_per_topic = collections.defaultdict(list) for member, metadata in six.iteritems(member_metadata): for topic in metadata.subscription: consumers_per_topic[topic].append(member) # construct {member_id: {topic: [partition, ...]}} assignment = collections.defaultdict(dict) for topic, consumers_for_topic in six.iteritems(consumers_per_topic): partitions = cluster.partitions_for_topic(topic) if partitions is None: log.warning('No partition metadata for topic %s', topic) continue partitions = sorted(list(partitions)) partitions_for_topic = len(partitions) consumers_for_topic.sort() partitions_per_consumer = len(partitions) // len(consumers_for_topic) consumers_with_extra = len(partitions) % len(consumers_for_topic) for i in range(len(consumers_for_topic)): start = partitions_per_consumer * i start += min(i, consumers_with_extra) length = partitions_per_consumer if not i + 1 > consumers_with_extra: length += 1 member = consumers_for_topic[i] assignment[member][topic] = partitions[start:start+length] protocol_assignment = {} for member_id in member_metadata: protocol_assignment[member_id] = ConsumerProtocolMemberAssignment( cls.version, sorted(assignment[member_id].items()), b'') return protocol_assignment @classmethod def metadata(cls, topics): return ConsumerProtocolMemberMetadata(cls.version, list(topics), b'') @classmethod def on_assignment(cls, assignment): pass kafka-1.3.2/kafka/coordinator/assignors/roundrobin.py0000644001271300127130000000722113031057471022526 0ustar dpowers00000000000000from __future__ import absolute_import import collections import itertools import logging from kafka.vendor import six from .abstract import AbstractPartitionAssignor from ...common import TopicPartition from ..protocol import ConsumerProtocolMemberMetadata, ConsumerProtocolMemberAssignment log = logging.getLogger(__name__) class RoundRobinPartitionAssignor(AbstractPartitionAssignor): """ The roundrobin assignor lays out all the available partitions and all the available consumers. It then proceeds to do a roundrobin assignment from partition to consumer. If the subscriptions of all consumer instances are identical, then the partitions will be uniformly distributed. (i.e., the partition ownership counts will be within a delta of exactly one across all consumers.) For example, suppose there are two consumers C0 and C1, two topics t0 and t1, and each topic has 3 partitions, resulting in partitions t0p0, t0p1, t0p2, t1p0, t1p1, and t1p2. The assignment will be: C0: [t0p0, t0p2, t1p1] C1: [t0p1, t1p0, t1p2] When subscriptions differ across consumer instances, the assignment process still considers each consumer instance in round robin fashion but skips over an instance if it is not subscribed to the topic. Unlike the case when subscriptions are identical, this can result in imbalanced assignments. For example, suppose we have three consumers C0, C1, C2, and three topics t0, t1, t2, with unbalanced partitions t0p0, t1p0, t1p1, t2p0, t2p1, t2p2, where C0 is subscribed to t0; C1 is subscribed to t0, t1; and C2 is subscribed to t0, t1, t2. The assignment will be: C0: [t0p0] C1: [t1p0] C2: [t1p1, t2p0, t2p1, t2p2] """ name = 'roundrobin' version = 0 @classmethod def assign(cls, cluster, member_metadata): all_topics = set() for metadata in six.itervalues(member_metadata): all_topics.update(metadata.subscription) all_topic_partitions = [] for topic in all_topics: partitions = cluster.partitions_for_topic(topic) if partitions is None: log.warning('No partition metadata for topic %s', topic) continue for partition in partitions: all_topic_partitions.append(TopicPartition(topic, partition)) all_topic_partitions.sort() # construct {member_id: {topic: [partition, ...]}} assignment = collections.defaultdict(lambda: collections.defaultdict(list)) member_iter = itertools.cycle(sorted(member_metadata.keys())) for partition in all_topic_partitions: member_id = next(member_iter) # Because we constructed all_topic_partitions from the set of # member subscribed topics, we should be safe assuming that # each topic in all_topic_partitions is in at least one member # subscription; otherwise this could yield an infinite loop while partition.topic not in member_metadata[member_id].subscription: member_id = next(member_iter) assignment[member_id][partition.topic].append(partition.partition) protocol_assignment = {} for member_id in member_metadata: protocol_assignment[member_id] = ConsumerProtocolMemberAssignment( cls.version, sorted(assignment[member_id].items()), b'') return protocol_assignment @classmethod def metadata(cls, topics): return ConsumerProtocolMemberMetadata(cls.version, list(topics), b'') @classmethod def on_assignment(cls, assignment): pass kafka-1.3.2/kafka/coordinator/base.py0000644001271300127130000007573313031057471017264 0ustar dpowers00000000000000from __future__ import absolute_import, division import abc import copy import logging import time import weakref from kafka.vendor import six from .heartbeat import Heartbeat from .. import errors as Errors from ..future import Future from ..metrics import AnonMeasurable from ..metrics.stats import Avg, Count, Max, Rate from ..protocol.commit import GroupCoordinatorRequest, OffsetCommitRequest from ..protocol.group import (HeartbeatRequest, JoinGroupRequest, LeaveGroupRequest, SyncGroupRequest) log = logging.getLogger('kafka.coordinator') class BaseCoordinator(object): """ BaseCoordinator implements group management for a single group member by interacting with a designated Kafka broker (the coordinator). Group semantics are provided by extending this class. See ConsumerCoordinator for example usage. From a high level, Kafka's group management protocol consists of the following sequence of actions: 1. Group Registration: Group members register with the coordinator providing their own metadata (such as the set of topics they are interested in). 2. Group/Leader Selection: The coordinator select the members of the group and chooses one member as the leader. 3. State Assignment: The leader collects the metadata from all the members of the group and assigns state. 4. Group Stabilization: Each member receives the state assigned by the leader and begins processing. To leverage this protocol, an implementation must define the format of metadata provided by each member for group registration in group_protocols() and the format of the state assignment provided by the leader in _perform_assignment() and which becomes available to members in _on_join_complete(). """ DEFAULT_CONFIG = { 'group_id': 'kafka-python-default-group', 'session_timeout_ms': 30000, 'heartbeat_interval_ms': 3000, 'retry_backoff_ms': 100, 'api_version': (0, 9), 'metric_group_prefix': '', } def __init__(self, client, metrics, **configs): """ Keyword Arguments: group_id (str): name of the consumer group to join for dynamic partition assignment (if enabled), and to use for fetching and committing offsets. Default: 'kafka-python-default-group' session_timeout_ms (int): The timeout used to detect failures when using Kafka's group managementment facilities. Default: 30000 heartbeat_interval_ms (int): The expected time in milliseconds between heartbeats to the consumer coordinator when using Kafka's group management feature. Heartbeats are used to ensure that the consumer's session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session_timeout_ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances. Default: 3000 retry_backoff_ms (int): Milliseconds to backoff when retrying on errors. Default: 100. """ self.config = copy.copy(self.DEFAULT_CONFIG) for key in self.config: if key in configs: self.config[key] = configs[key] self._client = client self.generation = OffsetCommitRequest[2].DEFAULT_GENERATION_ID self.member_id = JoinGroupRequest[0].UNKNOWN_MEMBER_ID self.group_id = self.config['group_id'] self.coordinator_id = None self.rejoin_needed = True self.rejoining = False self.heartbeat = Heartbeat(**self.config) self.heartbeat_task = HeartbeatTask(weakref.proxy(self)) self.sensors = GroupCoordinatorMetrics(self.heartbeat, metrics, self.config['metric_group_prefix']) def __del__(self): if hasattr(self, 'heartbeat_task') and self.heartbeat_task: self.heartbeat_task.disable() @abc.abstractmethod def protocol_type(self): """ Unique identifier for the class of protocols implements (e.g. "consumer" or "connect"). Returns: str: protocol type name """ pass @abc.abstractmethod def group_protocols(self): """Return the list of supported group protocols and metadata. This list is submitted by each group member via a JoinGroupRequest. The order of the protocols in the list indicates the preference of the protocol (the first entry is the most preferred). The coordinator takes this preference into account when selecting the generation protocol (generally more preferred protocols will be selected as long as all members support them and there is no disagreement on the preference). Note: metadata must be type bytes or support an encode() method Returns: list: [(protocol, metadata), ...] """ pass @abc.abstractmethod def _on_join_prepare(self, generation, member_id): """Invoked prior to each group join or rejoin. This is typically used to perform any cleanup from the previous generation (such as committing offsets for the consumer) Arguments: generation (int): The previous generation or -1 if there was none member_id (str): The identifier of this member in the previous group or '' if there was none """ pass @abc.abstractmethod def _perform_assignment(self, leader_id, protocol, members): """Perform assignment for the group. This is used by the leader to push state to all the members of the group (e.g. to push partition assignments in the case of the new consumer) Arguments: leader_id (str): The id of the leader (which is this member) protocol (str): the chosen group protocol (assignment strategy) members (list): [(member_id, metadata_bytes)] from JoinGroupResponse. metadata_bytes are associated with the chosen group protocol, and the Coordinator subclass is responsible for decoding metadata_bytes based on that protocol. Returns: dict: {member_id: assignment}; assignment must either be bytes or have an encode() method to convert to bytes """ pass @abc.abstractmethod def _on_join_complete(self, generation, member_id, protocol, member_assignment_bytes): """Invoked when a group member has successfully joined a group. Arguments: generation (int): the generation that was joined member_id (str): the identifier for the local member in the group protocol (str): the protocol selected by the coordinator member_assignment_bytes (bytes): the protocol-encoded assignment propagated from the group leader. The Coordinator instance is responsible for decoding based on the chosen protocol. """ pass def coordinator_unknown(self): """Check if we know who the coordinator is and have an active connection Side-effect: reset coordinator_id to None if connection failed Returns: bool: True if the coordinator is unknown """ if self.coordinator_id is None: return True if self._client.is_disconnected(self.coordinator_id): self.coordinator_dead('Node Disconnected') return True return False def ensure_coordinator_known(self): """Block until the coordinator for this group is known (and we have an active connection -- java client uses unsent queue). """ while self.coordinator_unknown(): # Prior to 0.8.2 there was no group coordinator # so we will just pick a node at random and treat # it as the "coordinator" if self.config['api_version'] < (0, 8, 2): self.coordinator_id = self._client.least_loaded_node() self._client.ready(self.coordinator_id) continue future = self._send_group_coordinator_request() self._client.poll(future=future) if future.failed(): if isinstance(future.exception, Errors.GroupCoordinatorNotAvailableError): continue elif future.retriable(): metadata_update = self._client.cluster.request_update() self._client.poll(future=metadata_update) else: raise future.exception # pylint: disable-msg=raising-bad-type def need_rejoin(self): """Check whether the group should be rejoined (e.g. if metadata changes) Returns: bool: True if it should, False otherwise """ return self.rejoin_needed def ensure_active_group(self): """Ensure that the group is active (i.e. joined and synced)""" if not self.need_rejoin(): return if not self.rejoining: self._on_join_prepare(self.generation, self.member_id) self.rejoining = True while self.need_rejoin(): self.ensure_coordinator_known() # ensure that there are no pending requests to the coordinator. # This is important in particular to avoid resending a pending # JoinGroup request. if self._client.in_flight_request_count(self.coordinator_id): while not self.coordinator_unknown(): self._client.poll(delayed_tasks=False) if not self._client.in_flight_request_count(self.coordinator_id): break else: continue future = self._send_join_group_request() self._client.poll(future=future) if future.succeeded(): member_assignment_bytes = future.value self._on_join_complete(self.generation, self.member_id, self.protocol, member_assignment_bytes) self.rejoining = False self.heartbeat_task.reset() else: assert future.failed() exception = future.exception if isinstance(exception, (Errors.UnknownMemberIdError, Errors.RebalanceInProgressError, Errors.IllegalGenerationError)): continue elif not future.retriable(): raise exception # pylint: disable-msg=raising-bad-type time.sleep(self.config['retry_backoff_ms'] / 1000) def _send_join_group_request(self): """Join the group and return the assignment for the next generation. This function handles both JoinGroup and SyncGroup, delegating to _perform_assignment() if elected leader by the coordinator. Returns: Future: resolves to the encoded-bytes assignment returned from the group leader """ if self.coordinator_unknown(): e = Errors.GroupCoordinatorNotAvailableError(self.coordinator_id) return Future().failure(e) # send a join group request to the coordinator log.info("(Re-)joining group %s", self.group_id) request = JoinGroupRequest[0]( self.group_id, self.config['session_timeout_ms'], self.member_id, self.protocol_type(), [(protocol, metadata if isinstance(metadata, bytes) else metadata.encode()) for protocol, metadata in self.group_protocols()]) # create the request for the coordinator log.debug("Sending JoinGroup (%s) to coordinator %s", request, self.coordinator_id) future = Future() _f = self._client.send(self.coordinator_id, request) _f.add_callback(self._handle_join_group_response, future, time.time()) _f.add_errback(self._failed_request, self.coordinator_id, request, future) return future def _failed_request(self, node_id, request, future, error): log.error('Error sending %s to node %s [%s]', request.__class__.__name__, node_id, error) # Marking coordinator dead # unless the error is caused by internal client pipelining if not isinstance(error, (Errors.NodeNotReadyError, Errors.TooManyInFlightRequests)): self.coordinator_dead(error) future.failure(error) def _handle_join_group_response(self, future, send_time, response): error_type = Errors.for_code(response.error_code) if error_type is Errors.NoError: log.debug("Received successful JoinGroup response for group %s: %s", self.group_id, response) self.member_id = response.member_id self.generation = response.generation_id self.rejoin_needed = False self.protocol = response.group_protocol log.info("Joined group '%s' (generation %s) with member_id %s", self.group_id, self.generation, self.member_id) self.sensors.join_latency.record((time.time() - send_time) * 1000) if response.leader_id == response.member_id: log.info("Elected group leader -- performing partition" " assignments using %s", self.protocol) self._on_join_leader(response).chain(future) else: self._on_join_follower().chain(future) elif error_type is Errors.GroupLoadInProgressError: log.debug("Attempt to join group %s rejected since coordinator %s" " is loading the group.", self.group_id, self.coordinator_id) # backoff and retry future.failure(error_type(response)) elif error_type is Errors.UnknownMemberIdError: # reset the member id and retry immediately error = error_type(self.member_id) self.member_id = JoinGroupRequest[0].UNKNOWN_MEMBER_ID log.debug("Attempt to join group %s failed due to unknown member id", self.group_id) future.failure(error) elif error_type in (Errors.GroupCoordinatorNotAvailableError, Errors.NotCoordinatorForGroupError): # re-discover the coordinator and retry with backoff self.coordinator_dead(error_type()) log.debug("Attempt to join group %s failed due to obsolete " "coordinator information: %s", self.group_id, error_type.__name__) future.failure(error_type()) elif error_type in (Errors.InconsistentGroupProtocolError, Errors.InvalidSessionTimeoutError, Errors.InvalidGroupIdError): # log the error and re-throw the exception error = error_type(response) log.error("Attempt to join group %s failed due to fatal error: %s", self.group_id, error) future.failure(error) elif error_type is Errors.GroupAuthorizationFailedError: future.failure(error_type(self.group_id)) else: # unexpected error, throw the exception error = error_type() log.error("Unexpected error in join group response: %s", error) future.failure(error) def _on_join_follower(self): # send follower's sync group with an empty assignment request = SyncGroupRequest[0]( self.group_id, self.generation, self.member_id, {}) log.debug("Sending follower SyncGroup for group %s to coordinator %s: %s", self.group_id, self.coordinator_id, request) return self._send_sync_group_request(request) def _on_join_leader(self, response): """ Perform leader synchronization and send back the assignment for the group via SyncGroupRequest Arguments: response (JoinResponse): broker response to parse Returns: Future: resolves to member assignment encoded-bytes """ try: group_assignment = self._perform_assignment(response.leader_id, response.group_protocol, response.members) except Exception as e: return Future().failure(e) request = SyncGroupRequest[0]( self.group_id, self.generation, self.member_id, [(member_id, assignment if isinstance(assignment, bytes) else assignment.encode()) for member_id, assignment in six.iteritems(group_assignment)]) log.debug("Sending leader SyncGroup for group %s to coordinator %s: %s", self.group_id, self.coordinator_id, request) return self._send_sync_group_request(request) def _send_sync_group_request(self, request): if self.coordinator_unknown(): e = Errors.GroupCoordinatorNotAvailableError(self.coordinator_id) return Future().failure(e) future = Future() _f = self._client.send(self.coordinator_id, request) _f.add_callback(self._handle_sync_group_response, future, time.time()) _f.add_errback(self._failed_request, self.coordinator_id, request, future) return future def _handle_sync_group_response(self, future, send_time, response): error_type = Errors.for_code(response.error_code) if error_type is Errors.NoError: log.info("Successfully joined group %s with generation %s", self.group_id, self.generation) self.sensors.sync_latency.record((time.time() - send_time) * 1000) future.success(response.member_assignment) return # Always rejoin on error self.rejoin_needed = True if error_type is Errors.GroupAuthorizationFailedError: future.failure(error_type(self.group_id)) elif error_type is Errors.RebalanceInProgressError: log.debug("SyncGroup for group %s failed due to coordinator" " rebalance", self.group_id) future.failure(error_type(self.group_id)) elif error_type in (Errors.UnknownMemberIdError, Errors.IllegalGenerationError): error = error_type() log.debug("SyncGroup for group %s failed due to %s", self.group_id, error) self.member_id = JoinGroupRequest[0].UNKNOWN_MEMBER_ID future.failure(error) elif error_type in (Errors.GroupCoordinatorNotAvailableError, Errors.NotCoordinatorForGroupError): error = error_type() log.debug("SyncGroup for group %s failed due to %s", self.group_id, error) self.coordinator_dead(error) future.failure(error) else: error = error_type() log.error("Unexpected error from SyncGroup: %s", error) future.failure(error) def _send_group_coordinator_request(self): """Discover the current coordinator for the group. Returns: Future: resolves to the node id of the coordinator """ node_id = self._client.least_loaded_node() if node_id is None: return Future().failure(Errors.NoBrokersAvailable()) log.debug("Sending group coordinator request for group %s to broker %s", self.group_id, node_id) request = GroupCoordinatorRequest[0](self.group_id) future = Future() _f = self._client.send(node_id, request) _f.add_callback(self._handle_group_coordinator_response, future) _f.add_errback(self._failed_request, node_id, request, future) return future def _handle_group_coordinator_response(self, future, response): log.debug("Received group coordinator response %s", response) if not self.coordinator_unknown(): # We already found the coordinator, so ignore the request log.debug("Coordinator already known -- ignoring metadata response") future.success(self.coordinator_id) return error_type = Errors.for_code(response.error_code) if error_type is Errors.NoError: ok = self._client.cluster.add_group_coordinator(self.group_id, response) if not ok: # This could happen if coordinator metadata is different # than broker metadata future.failure(Errors.IllegalStateError()) return self.coordinator_id = response.coordinator_id log.info("Discovered coordinator %s for group %s", self.coordinator_id, self.group_id) self._client.ready(self.coordinator_id) # start sending heartbeats only if we have a valid generation if self.generation > 0: self.heartbeat_task.reset() future.success(self.coordinator_id) elif error_type is Errors.GroupCoordinatorNotAvailableError: log.debug("Group Coordinator Not Available; retry") future.failure(error_type()) elif error_type is Errors.GroupAuthorizationFailedError: error = error_type(self.group_id) log.error("Group Coordinator Request failed: %s", error) future.failure(error) else: error = error_type() log.error("Unrecognized failure in Group Coordinator Request: %s", error) future.failure(error) def coordinator_dead(self, error): """Mark the current coordinator as dead.""" if self.coordinator_id is not None: log.warning("Marking the coordinator dead (node %s) for group %s: %s.", self.coordinator_id, self.group_id, error) self.coordinator_id = None def close(self): """Close the coordinator, leave the current group and reset local generation/memberId.""" try: self._client.unschedule(self.heartbeat_task) except KeyError: pass if not self.coordinator_unknown() and self.generation > 0: # this is a minimal effort attempt to leave the group. we do not # attempt any resending if the request fails or times out. request = LeaveGroupRequest[0](self.group_id, self.member_id) future = self._client.send(self.coordinator_id, request) future.add_callback(self._handle_leave_group_response) future.add_errback(log.error, "LeaveGroup request failed: %s") self._client.poll(future=future) self.generation = OffsetCommitRequest[2].DEFAULT_GENERATION_ID self.member_id = JoinGroupRequest[0].UNKNOWN_MEMBER_ID self.rejoin_needed = True def _handle_leave_group_response(self, response): error_type = Errors.for_code(response.error_code) if error_type is Errors.NoError: log.info("LeaveGroup request succeeded") else: log.error("LeaveGroup request failed: %s", error_type()) def _send_heartbeat_request(self): """Send a heartbeat request""" request = HeartbeatRequest[0](self.group_id, self.generation, self.member_id) log.debug("Heartbeat: %s[%s] %s", request.group, request.generation_id, request.member_id) #pylint: disable-msg=no-member future = Future() _f = self._client.send(self.coordinator_id, request) _f.add_callback(self._handle_heartbeat_response, future, time.time()) _f.add_errback(self._failed_request, self.coordinator_id, request, future) return future def _handle_heartbeat_response(self, future, send_time, response): self.sensors.heartbeat_latency.record((time.time() - send_time) * 1000) error_type = Errors.for_code(response.error_code) if error_type is Errors.NoError: log.debug("Received successful heartbeat response for group %s", self.group_id) future.success(None) elif error_type in (Errors.GroupCoordinatorNotAvailableError, Errors.NotCoordinatorForGroupError): log.warning("Heartbeat failed for group %s: coordinator (node %s)" " is either not started or not valid", self.group_id, self.coordinator_id) self.coordinator_dead(error_type()) future.failure(error_type()) elif error_type is Errors.RebalanceInProgressError: log.warning("Heartbeat failed for group %s because it is" " rebalancing", self.group_id) self.rejoin_needed = True future.failure(error_type()) elif error_type is Errors.IllegalGenerationError: log.warning("Heartbeat failed for group %s: generation id is not " " current.", self.group_id) self.rejoin_needed = True future.failure(error_type()) elif error_type is Errors.UnknownMemberIdError: log.warning("Heartbeat: local member_id was not recognized;" " this consumer needs to re-join") self.member_id = JoinGroupRequest[0].UNKNOWN_MEMBER_ID self.rejoin_needed = True future.failure(error_type) elif error_type is Errors.GroupAuthorizationFailedError: error = error_type(self.group_id) log.error("Heartbeat failed: authorization error: %s", error) future.failure(error) else: error = error_type() log.error("Heartbeat failed: Unhandled error: %s", error) future.failure(error) class HeartbeatTask(object): def __init__(self, coordinator): self._coordinator = coordinator self._heartbeat = coordinator.heartbeat self._client = coordinator._client self._request_in_flight = False def disable(self): try: self._client.unschedule(self) except KeyError: pass def reset(self): # start or restart the heartbeat task to be executed at the next chance self._heartbeat.reset_session_timeout() try: self._client.unschedule(self) except KeyError: pass if not self._request_in_flight: self._client.schedule(self, time.time()) def __call__(self): if (self._coordinator.generation < 0 or self._coordinator.need_rejoin()): # no need to send the heartbeat we're not using auto-assignment # or if we are awaiting a rebalance log.info("Skipping heartbeat: no auto-assignment" " or waiting on rebalance") return if self._coordinator.coordinator_unknown(): log.warning("Coordinator unknown during heartbeat -- will retry") self._handle_heartbeat_failure(Errors.GroupCoordinatorNotAvailableError()) return if self._heartbeat.session_expired(): # we haven't received a successful heartbeat in one session interval # so mark the coordinator dead log.error("Heartbeat session expired - marking coordinator dead") self._coordinator.coordinator_dead('Heartbeat session expired') return if not self._heartbeat.should_heartbeat(): # we don't need to heartbeat now, so reschedule for when we do ttl = self._heartbeat.ttl() log.debug("Heartbeat task unneeded now, retrying in %s", ttl) self._client.schedule(self, time.time() + ttl) else: self._heartbeat.sent_heartbeat() self._request_in_flight = True future = self._coordinator._send_heartbeat_request() future.add_callback(self._handle_heartbeat_success) future.add_errback(self._handle_heartbeat_failure) def _handle_heartbeat_success(self, v): log.debug("Received successful heartbeat") self._request_in_flight = False self._heartbeat.received_heartbeat() ttl = self._heartbeat.ttl() self._client.schedule(self, time.time() + ttl) def _handle_heartbeat_failure(self, e): log.warning("Heartbeat failed (%s); retrying", e) self._request_in_flight = False etd = time.time() + self._coordinator.config['retry_backoff_ms'] / 1000 self._client.schedule(self, etd) class GroupCoordinatorMetrics(object): def __init__(self, heartbeat, metrics, prefix, tags=None): self.heartbeat = heartbeat self.metrics = metrics self.metric_group_name = prefix + "-coordinator-metrics" self.heartbeat_latency = metrics.sensor('heartbeat-latency') self.heartbeat_latency.add(metrics.metric_name( 'heartbeat-response-time-max', self.metric_group_name, 'The max time taken to receive a response to a heartbeat request', tags), Max()) self.heartbeat_latency.add(metrics.metric_name( 'heartbeat-rate', self.metric_group_name, 'The average number of heartbeats per second', tags), Rate(sampled_stat=Count())) self.join_latency = metrics.sensor('join-latency') self.join_latency.add(metrics.metric_name( 'join-time-avg', self.metric_group_name, 'The average time taken for a group rejoin', tags), Avg()) self.join_latency.add(metrics.metric_name( 'join-time-max', self.metric_group_name, 'The max time taken for a group rejoin', tags), Avg()) self.join_latency.add(metrics.metric_name( 'join-rate', self.metric_group_name, 'The number of group joins per second', tags), Rate(sampled_stat=Count())) self.sync_latency = metrics.sensor('sync-latency') self.sync_latency.add(metrics.metric_name( 'sync-time-avg', self.metric_group_name, 'The average time taken for a group sync', tags), Avg()) self.sync_latency.add(metrics.metric_name( 'sync-time-max', self.metric_group_name, 'The max time taken for a group sync', tags), Avg()) self.sync_latency.add(metrics.metric_name( 'sync-rate', self.metric_group_name, 'The number of group syncs per second', tags), Rate(sampled_stat=Count())) metrics.add_metric(metrics.metric_name( 'last-heartbeat-seconds-ago', self.metric_group_name, 'The number of seconds since the last controller heartbeat', tags), AnonMeasurable( lambda _, now: (now / 1000) - self.heartbeat.last_send)) kafka-1.3.2/kafka/coordinator/consumer.py0000644001271300127130000007761413031057471020205 0ustar dpowers00000000000000from __future__ import absolute_import import copy import collections import logging import time import weakref from kafka.vendor import six from .base import BaseCoordinator from .assignors.range import RangePartitionAssignor from .assignors.roundrobin import RoundRobinPartitionAssignor from .protocol import ConsumerProtocol from .. import errors as Errors from ..future import Future from ..metrics import AnonMeasurable from ..metrics.stats import Avg, Count, Max, Rate from ..protocol.commit import OffsetCommitRequest, OffsetFetchRequest from ..structs import OffsetAndMetadata, TopicPartition from ..util import WeakMethod log = logging.getLogger(__name__) class ConsumerCoordinator(BaseCoordinator): """This class manages the coordination process with the consumer coordinator.""" DEFAULT_CONFIG = { 'group_id': 'kafka-python-default-group', 'enable_auto_commit': True, 'auto_commit_interval_ms': 5000, 'default_offset_commit_callback': lambda offsets, response: True, 'assignors': (RangePartitionAssignor, RoundRobinPartitionAssignor), 'session_timeout_ms': 30000, 'heartbeat_interval_ms': 3000, 'retry_backoff_ms': 100, 'api_version': (0, 9), 'exclude_internal_topics': True, 'metric_group_prefix': 'consumer' } def __init__(self, client, subscription, metrics, **configs): """Initialize the coordination manager. Keyword Arguments: group_id (str): name of the consumer group to join for dynamic partition assignment (if enabled), and to use for fetching and committing offsets. Default: 'kafka-python-default-group' enable_auto_commit (bool): If true the consumer's offset will be periodically committed in the background. Default: True. auto_commit_interval_ms (int): milliseconds between automatic offset commits, if enable_auto_commit is True. Default: 5000. default_offset_commit_callback (callable): called as callback(offsets, response) response will be either an Exception or a OffsetCommitResponse struct. This callback can be used to trigger custom actions when a commit request completes. assignors (list): List of objects to use to distribute partition ownership amongst consumer instances when group management is used. Default: [RangePartitionAssignor, RoundRobinPartitionAssignor] heartbeat_interval_ms (int): The expected time in milliseconds between heartbeats to the consumer coordinator when using Kafka's group management feature. Heartbeats are used to ensure that the consumer's session stays active and to facilitate rebalancing when new consumers join or leave the group. The value must be set lower than session_timeout_ms, but typically should be set no higher than 1/3 of that value. It can be adjusted even lower to control the expected time for normal rebalances. Default: 3000 session_timeout_ms (int): The timeout used to detect failures when using Kafka's group managementment facilities. Default: 30000 retry_backoff_ms (int): Milliseconds to backoff when retrying on errors. Default: 100. exclude_internal_topics (bool): Whether records from internal topics (such as offsets) should be exposed to the consumer. If set to True the only way to receive records from an internal topic is subscribing to it. Requires 0.10+. Default: True """ super(ConsumerCoordinator, self).__init__(client, metrics, **configs) self.config = copy.copy(self.DEFAULT_CONFIG) for key in self.config: if key in configs: self.config[key] = configs[key] if self.config['api_version'] >= (0, 9) and self.config['group_id'] is not None: assert self.config['assignors'], 'Coordinator requires assignors' self._subscription = subscription self._metadata_snapshot = {} self._assignment_snapshot = None self._cluster = client.cluster self._cluster.request_update() self._cluster.add_listener(WeakMethod(self._handle_metadata_update)) self._auto_commit_task = None if self.config['enable_auto_commit']: if self.config['api_version'] < (0, 8, 1): log.warning('Broker version (%s) does not support offset' ' commits; disabling auto-commit.', self.config['api_version']) self.config['enable_auto_commit'] = False elif self.config['group_id'] is None: log.warning('group_id is None: disabling auto-commit.') self.config['enable_auto_commit'] = False else: interval = self.config['auto_commit_interval_ms'] / 1000.0 self._auto_commit_task = AutoCommitTask(weakref.proxy(self), interval) self._auto_commit_task.reschedule() self.consumer_sensors = ConsumerCoordinatorMetrics( metrics, self.config['metric_group_prefix'], self._subscription) def __del__(self): if hasattr(self, '_cluster') and self._cluster: self._cluster.remove_listener(WeakMethod(self._handle_metadata_update)) def protocol_type(self): return ConsumerProtocol.PROTOCOL_TYPE def group_protocols(self): """Returns list of preferred (protocols, metadata)""" topics = self._subscription.subscription assert topics is not None, 'Consumer has not subscribed to topics' metadata_list = [] for assignor in self.config['assignors']: metadata = assignor.metadata(topics) group_protocol = (assignor.name, metadata) metadata_list.append(group_protocol) return metadata_list def _handle_metadata_update(self, cluster): # if we encounter any unauthorized topics, raise an exception if cluster.unauthorized_topics: raise Errors.TopicAuthorizationFailedError(cluster.unauthorized_topics) if self._subscription.subscribed_pattern: topics = [] for topic in cluster.topics(self.config['exclude_internal_topics']): if self._subscription.subscribed_pattern.match(topic): topics.append(topic) self._subscription.change_subscription(topics) self._client.set_topics(self._subscription.group_subscription()) # check if there are any changes to the metadata which should trigger # a rebalance if self._subscription_metadata_changed(cluster): if (self.config['api_version'] >= (0, 9) and self.config['group_id'] is not None): self._subscription.mark_for_reassignment() # If we haven't got group coordinator support, # just assign all partitions locally else: self._subscription.assign_from_subscribed([ TopicPartition(topic, partition) for topic in self._subscription.subscription for partition in self._metadata_snapshot[topic] ]) def _subscription_metadata_changed(self, cluster): if not self._subscription.partitions_auto_assigned(): return False metadata_snapshot = {} for topic in self._subscription.group_subscription(): partitions = cluster.partitions_for_topic(topic) or [] metadata_snapshot[topic] = set(partitions) if self._metadata_snapshot != metadata_snapshot: self._metadata_snapshot = metadata_snapshot return True return False def _lookup_assignor(self, name): for assignor in self.config['assignors']: if assignor.name == name: return assignor return None def _on_join_complete(self, generation, member_id, protocol, member_assignment_bytes): # if we were the assignor, then we need to make sure that there have # been no metadata updates since the rebalance begin. Otherwise, we # won't rebalance again until the next metadata change if self._assignment_snapshot and self._assignment_snapshot != self._metadata_snapshot: self._subscription.mark_for_reassignment() return assignor = self._lookup_assignor(protocol) assert assignor, 'Coordinator selected invalid assignment protocol: %s' % protocol assignment = ConsumerProtocol.ASSIGNMENT.decode(member_assignment_bytes) # set the flag to refresh last committed offsets self._subscription.needs_fetch_committed_offsets = True # update partition assignment self._subscription.assign_from_subscribed(assignment.partitions()) # give the assignor a chance to update internal state # based on the received assignment assignor.on_assignment(assignment) # reschedule the auto commit starting from now if self._auto_commit_task: self._auto_commit_task.reschedule() assigned = set(self._subscription.assigned_partitions()) log.info("Setting newly assigned partitions %s for group %s", assigned, self.group_id) # execute the user's callback after rebalance if self._subscription.listener: try: self._subscription.listener.on_partitions_assigned(assigned) except Exception: log.exception("User provided listener %s for group %s" " failed on partition assignment: %s", self._subscription.listener, self.group_id, assigned) def _perform_assignment(self, leader_id, assignment_strategy, members): assignor = self._lookup_assignor(assignment_strategy) assert assignor, 'Invalid assignment protocol: %s' % assignment_strategy member_metadata = {} all_subscribed_topics = set() for member_id, metadata_bytes in members: metadata = ConsumerProtocol.METADATA.decode(metadata_bytes) member_metadata[member_id] = metadata all_subscribed_topics.update(metadata.subscription) # pylint: disable-msg=no-member # the leader will begin watching for changes to any of the topics # the group is interested in, which ensures that all metadata changes # will eventually be seen # Because assignment typically happens within response callbacks, # we cannot block on metadata updates here (no recursion into poll()) self._subscription.group_subscribe(all_subscribed_topics) self._client.set_topics(self._subscription.group_subscription()) # keep track of the metadata used for assignment so that we can check # after rebalance completion whether anything has changed self._cluster.request_update() self._assignment_snapshot = self._metadata_snapshot log.debug("Performing assignment for group %s using strategy %s" " with subscriptions %s", self.group_id, assignor.name, member_metadata) assignments = assignor.assign(self._cluster, member_metadata) log.debug("Finished assignment for group %s: %s", self.group_id, assignments) group_assignment = {} for member_id, assignment in six.iteritems(assignments): group_assignment[member_id] = assignment return group_assignment def _on_join_prepare(self, generation, member_id): # commit offsets prior to rebalance if auto-commit enabled self._maybe_auto_commit_offsets_sync() # execute the user's callback before rebalance log.info("Revoking previously assigned partitions %s for group %s", self._subscription.assigned_partitions(), self.group_id) if self._subscription.listener: try: revoked = set(self._subscription.assigned_partitions()) self._subscription.listener.on_partitions_revoked(revoked) except Exception: log.exception("User provided subscription listener %s" " for group %s failed on_partitions_revoked", self._subscription.listener, self.group_id) self._assignment_snapshot = None self._subscription.mark_for_reassignment() def need_rejoin(self): """Check whether the group should be rejoined Returns: bool: True if consumer should rejoin group, False otherwise """ return (self._subscription.partitions_auto_assigned() and (super(ConsumerCoordinator, self).need_rejoin() or self._subscription.needs_partition_assignment)) def refresh_committed_offsets_if_needed(self): """Fetch committed offsets for assigned partitions.""" if self._subscription.needs_fetch_committed_offsets: offsets = self.fetch_committed_offsets(self._subscription.assigned_partitions()) for partition, offset in six.iteritems(offsets): # verify assignment is still active if self._subscription.is_assigned(partition): self._subscription.assignment[partition].committed = offset.offset self._subscription.needs_fetch_committed_offsets = False def fetch_committed_offsets(self, partitions): """Fetch the current committed offsets for specified partitions Arguments: partitions (list of TopicPartition): partitions to fetch Returns: dict: {TopicPartition: OffsetAndMetadata} """ if not partitions: return {} while True: self.ensure_coordinator_known() # contact coordinator to fetch committed offsets future = self._send_offset_fetch_request(partitions) self._client.poll(future=future) if future.succeeded(): return future.value if not future.retriable(): raise future.exception # pylint: disable-msg=raising-bad-type time.sleep(self.config['retry_backoff_ms'] / 1000.0) def close(self): try: self._maybe_auto_commit_offsets_sync() finally: super(ConsumerCoordinator, self).close() def commit_offsets_async(self, offsets, callback=None): """Commit specific offsets asynchronously. Arguments: offsets (dict {TopicPartition: OffsetAndMetadata}): what to commit callback (callable, optional): called as callback(offsets, response) response will be either an Exception or a OffsetCommitResponse struct. This callback can be used to trigger custom actions when a commit request completes. Returns: Future: indicating whether the commit was successful or not """ assert self.config['api_version'] >= (0, 8, 1), 'Unsupported Broker API' assert all(map(lambda k: isinstance(k, TopicPartition), offsets)) assert all(map(lambda v: isinstance(v, OffsetAndMetadata), offsets.values())) if callback is None: callback = self.config['default_offset_commit_callback'] self._subscription.needs_fetch_committed_offsets = True future = self._send_offset_commit_request(offsets) future.add_both(callback, offsets) return future def commit_offsets_sync(self, offsets): """Commit specific offsets synchronously. This method will retry until the commit completes successfully or an unrecoverable error is encountered. Arguments: offsets (dict {TopicPartition: OffsetAndMetadata}): what to commit Raises error on failure """ assert self.config['api_version'] >= (0, 8, 1), 'Unsupported Broker API' assert all(map(lambda k: isinstance(k, TopicPartition), offsets)) assert all(map(lambda v: isinstance(v, OffsetAndMetadata), offsets.values())) if not offsets: return while True: self.ensure_coordinator_known() future = self._send_offset_commit_request(offsets) self._client.poll(future=future) if future.succeeded(): return future.value if not future.retriable(): raise future.exception # pylint: disable-msg=raising-bad-type time.sleep(self.config['retry_backoff_ms'] / 1000.0) def _maybe_auto_commit_offsets_sync(self): if self._auto_commit_task is None: return try: self.commit_offsets_sync(self._subscription.all_consumed_offsets()) # The three main group membership errors are known and should not # require a stacktrace -- just a warning except (Errors.UnknownMemberIdError, Errors.IllegalGenerationError, Errors.RebalanceInProgressError): log.warning("Offset commit failed: group membership out of date" " This is likely to cause duplicate message" " delivery.") except Exception: log.exception("Offset commit failed: This is likely to cause" " duplicate message delivery") def _send_offset_commit_request(self, offsets): """Commit offsets for the specified list of topics and partitions. This is a non-blocking call which returns a request future that can be polled in the case of a synchronous commit or ignored in the asynchronous case. Arguments: offsets (dict of {TopicPartition: OffsetAndMetadata}): what should be committed Returns: Future: indicating whether the commit was successful or not """ assert self.config['api_version'] >= (0, 8, 1), 'Unsupported Broker API' assert all(map(lambda k: isinstance(k, TopicPartition), offsets)) assert all(map(lambda v: isinstance(v, OffsetAndMetadata), offsets.values())) if not offsets: log.debug('No offsets to commit') return Future().success(True) elif self.coordinator_unknown(): return Future().failure(Errors.GroupCoordinatorNotAvailableError) node_id = self.coordinator_id # create the offset commit request offset_data = collections.defaultdict(dict) for tp, offset in six.iteritems(offsets): offset_data[tp.topic][tp.partition] = offset if self.config['api_version'] >= (0, 9): request = OffsetCommitRequest[2]( self.group_id, self.generation, self.member_id, OffsetCommitRequest[2].DEFAULT_RETENTION_TIME, [( topic, [( partition, offset.offset, offset.metadata ) for partition, offset in six.iteritems(partitions)] ) for topic, partitions in six.iteritems(offset_data)] ) elif self.config['api_version'] >= (0, 8, 2): request = OffsetCommitRequest[1]( self.group_id, -1, '', [( topic, [( partition, offset.offset, -1, offset.metadata ) for partition, offset in six.iteritems(partitions)] ) for topic, partitions in six.iteritems(offset_data)] ) elif self.config['api_version'] >= (0, 8, 1): request = OffsetCommitRequest[0]( self.group_id, [( topic, [( partition, offset.offset, offset.metadata ) for partition, offset in six.iteritems(partitions)] ) for topic, partitions in six.iteritems(offset_data)] ) log.debug("Sending offset-commit request with %s for group %s to %s", offsets, self.group_id, node_id) future = Future() _f = self._client.send(node_id, request) _f.add_callback(self._handle_offset_commit_response, offsets, future, time.time()) _f.add_errback(self._failed_request, node_id, request, future) return future def _handle_offset_commit_response(self, offsets, future, send_time, response): # TODO look at adding request_latency_ms to response (like java kafka) self.consumer_sensors.commit_latency.record((time.time() - send_time) * 1000) unauthorized_topics = set() for topic, partitions in response.topics: for partition, error_code in partitions: tp = TopicPartition(topic, partition) offset = offsets[tp] error_type = Errors.for_code(error_code) if error_type is Errors.NoError: log.debug("Group %s committed offset %s for partition %s", self.group_id, offset, tp) if self._subscription.is_assigned(tp): self._subscription.assignment[tp].committed = offset.offset elif error_type is Errors.GroupAuthorizationFailedError: log.error("Not authorized to commit offsets for group %s", self.group_id) future.failure(error_type(self.group_id)) return elif error_type is Errors.TopicAuthorizationFailedError: unauthorized_topics.add(topic) elif error_type in (Errors.OffsetMetadataTooLargeError, Errors.InvalidCommitOffsetSizeError): # raise the error to the user log.debug("OffsetCommit for group %s failed on partition %s" " %s", self.group_id, tp, error_type.__name__) future.failure(error_type()) return elif error_type is Errors.GroupLoadInProgressError: # just retry log.debug("OffsetCommit for group %s failed: %s", self.group_id, error_type.__name__) future.failure(error_type(self.group_id)) return elif error_type in (Errors.GroupCoordinatorNotAvailableError, Errors.NotCoordinatorForGroupError, Errors.RequestTimedOutError): log.debug("OffsetCommit for group %s failed: %s", self.group_id, error_type.__name__) self.coordinator_dead(error_type()) future.failure(error_type(self.group_id)) return elif error_type in (Errors.UnknownMemberIdError, Errors.IllegalGenerationError, Errors.RebalanceInProgressError): # need to re-join group error = error_type(self.group_id) log.debug("OffsetCommit for group %s failed: %s", self.group_id, error) self._subscription.mark_for_reassignment() future.failure(Errors.CommitFailedError( "Commit cannot be completed since the group has" " already rebalanced and assigned the partitions to" " another member. This means that the time between" " subsequent calls to poll() was longer than the" " configured session.timeout.ms, which typically" " implies that the poll loop is spending too much time" " message processing. You can address this either by" " increasing the session timeout or by reducing the" " maximum size of batches returned in poll() with" " max.poll.records.")) return else: log.error("Group %s failed to commit partition %s at offset" " %s: %s", self.group_id, tp, offset, error_type.__name__) future.failure(error_type()) return if unauthorized_topics: log.error("Not authorized to commit to topics %s for group %s", unauthorized_topics, self.group_id) future.failure(Errors.TopicAuthorizationFailedError(unauthorized_topics)) else: future.success(True) def _send_offset_fetch_request(self, partitions): """Fetch the committed offsets for a set of partitions. This is a non-blocking call. The returned future can be polled to get the actual offsets returned from the broker. Arguments: partitions (list of TopicPartition): the partitions to fetch Returns: Future: resolves to dict of offsets: {TopicPartition: int} """ assert self.config['api_version'] >= (0, 8, 1), 'Unsupported Broker API' assert all(map(lambda k: isinstance(k, TopicPartition), partitions)) if not partitions: return Future().success({}) elif self.coordinator_unknown(): return Future().failure(Errors.GroupCoordinatorNotAvailableError) node_id = self.coordinator_id # Verify node is ready if not self._client.ready(node_id): log.debug("Node %s not ready -- failing offset fetch request", node_id) return Future().failure(Errors.NodeNotReadyError) log.debug("Group %s fetching committed offsets for partitions: %s", self.group_id, partitions) # construct the request topic_partitions = collections.defaultdict(set) for tp in partitions: topic_partitions[tp.topic].add(tp.partition) if self.config['api_version'] >= (0, 8, 2): request = OffsetFetchRequest[1]( self.group_id, list(topic_partitions.items()) ) else: request = OffsetFetchRequest[0]( self.group_id, list(topic_partitions.items()) ) # send the request with a callback future = Future() _f = self._client.send(node_id, request) _f.add_callback(self._handle_offset_fetch_response, future) _f.add_errback(self._failed_request, node_id, request, future) return future def _handle_offset_fetch_response(self, future, response): offsets = {} for topic, partitions in response.topics: for partition, offset, metadata, error_code in partitions: tp = TopicPartition(topic, partition) error_type = Errors.for_code(error_code) if error_type is not Errors.NoError: error = error_type() log.debug("Group %s failed to fetch offset for partition" " %s: %s", self.group_id, tp, error) if error_type is Errors.GroupLoadInProgressError: # just retry future.failure(error) elif error_type is Errors.NotCoordinatorForGroupError: # re-discover the coordinator and retry self.coordinator_dead(error_type()) future.failure(error) elif error_type in (Errors.UnknownMemberIdError, Errors.IllegalGenerationError): # need to re-join group self._subscription.mark_for_reassignment() future.failure(error) elif error_type is Errors.UnknownTopicOrPartitionError: log.warning("OffsetFetchRequest -- unknown topic %s" " (have you committed any offsets yet?)", topic) continue else: log.error("Unknown error fetching offsets for %s: %s", tp, error) future.failure(error) return elif offset >= 0: # record the position with the offset # (-1 indicates no committed offset to fetch) offsets[tp] = OffsetAndMetadata(offset, metadata) else: log.debug("Group %s has no committed offset for partition" " %s", self.group_id, tp) future.success(offsets) class AutoCommitTask(object): def __init__(self, coordinator, interval): self._coordinator = coordinator self._client = coordinator._client self._interval = interval def reschedule(self, at=None): if at is None: at = time.time() + self._interval self._client.schedule(self, at) def __call__(self): if self._coordinator.coordinator_unknown(): log.debug("Cannot auto-commit offsets for group %s because the" " coordinator is unknown", self._coordinator.group_id) backoff = self._coordinator.config['retry_backoff_ms'] / 1000.0 self.reschedule(time.time() + backoff) return self._coordinator.commit_offsets_async( self._coordinator._subscription.all_consumed_offsets(), self._handle_commit_response) def _handle_commit_response(self, offsets, result): if result is True: log.debug("Successfully auto-committed offsets for group %s", self._coordinator.group_id) next_at = time.time() + self._interval elif not isinstance(result, BaseException): raise Errors.IllegalStateError( 'Unrecognized result in _handle_commit_response: %s' % result) elif hasattr(result, 'retriable') and result.retriable: log.debug("Failed to auto-commit offsets for group %s: %s," " will retry immediately", self._coordinator.group_id, result) next_at = time.time() else: log.warning("Auto offset commit failed for group %s: %s", self._coordinator.group_id, result) next_at = time.time() + self._interval self.reschedule(next_at) class ConsumerCoordinatorMetrics(object): def __init__(self, metrics, metric_group_prefix, subscription): self.metrics = metrics self.metric_group_name = '%s-coordinator-metrics' % metric_group_prefix self.commit_latency = metrics.sensor('commit-latency') self.commit_latency.add(metrics.metric_name( 'commit-latency-avg', self.metric_group_name, 'The average time taken for a commit request'), Avg()) self.commit_latency.add(metrics.metric_name( 'commit-latency-max', self.metric_group_name, 'The max time taken for a commit request'), Max()) self.commit_latency.add(metrics.metric_name( 'commit-rate', self.metric_group_name, 'The number of commit calls per second'), Rate(sampled_stat=Count())) num_parts = AnonMeasurable(lambda config, now: len(subscription.assigned_partitions())) metrics.add_metric(metrics.metric_name( 'assigned-partitions', self.metric_group_name, 'The number of partitions currently assigned to this consumer'), num_parts) kafka-1.3.2/kafka/coordinator/heartbeat.py0000644001271300127130000000261313025302127020266 0ustar dpowers00000000000000from __future__ import absolute_import import copy import time class Heartbeat(object): DEFAULT_CONFIG = { 'heartbeat_interval_ms': 3000, 'session_timeout_ms': 30000, } def __init__(self, **configs): self.config = copy.copy(self.DEFAULT_CONFIG) for key in self.config: if key in configs: self.config[key] = configs[key] assert (self.config['heartbeat_interval_ms'] <= self.config['session_timeout_ms']), ( 'Heartbeat interval must be lower than the session timeout') self.interval = self.config['heartbeat_interval_ms'] / 1000.0 self.timeout = self.config['session_timeout_ms'] / 1000.0 self.last_send = -1 * float('inf') self.last_receive = -1 * float('inf') self.last_reset = time.time() def sent_heartbeat(self): self.last_send = time.time() def received_heartbeat(self): self.last_receive = time.time() def ttl(self): last_beat = max(self.last_send, self.last_reset) return max(0, last_beat + self.interval - time.time()) def should_heartbeat(self): return self.ttl() == 0 def session_expired(self): last_recv = max(self.last_receive, self.last_reset) return (time.time() - last_recv) > self.timeout def reset_session_timeout(self): self.last_reset = time.time() kafka-1.3.2/kafka/coordinator/protocol.py0000644001271300127130000000202112702214455020170 0ustar dpowers00000000000000from __future__ import absolute_import from kafka.protocol.struct import Struct from kafka.protocol.types import Array, Bytes, Int16, Int32, Schema, String from kafka.structs import TopicPartition class ConsumerProtocolMemberMetadata(Struct): SCHEMA = Schema( ('version', Int16), ('subscription', Array(String('utf-8'))), ('user_data', Bytes)) class ConsumerProtocolMemberAssignment(Struct): SCHEMA = Schema( ('version', Int16), ('assignment', Array( ('topic', String('utf-8')), ('partitions', Array(Int32)))), ('user_data', Bytes)) def partitions(self): return [TopicPartition(topic, partition) for topic, partitions in self.assignment # pylint: disable-msg=no-member for partition in partitions] class ConsumerProtocol(object): PROTOCOL_TYPE = 'consumer' ASSIGNMENT_STRATEGIES = ('range', 'roundrobin') METADATA = ConsumerProtocolMemberMetadata ASSIGNMENT = ConsumerProtocolMemberAssignment kafka-1.3.2/kafka/errors.py0000644001271300127130000003146513031057471015335 0ustar dpowers00000000000000from __future__ import absolute_import import inspect import sys class KafkaError(RuntimeError): retriable = False # whether metadata should be refreshed on error invalid_metadata = False def __str__(self): if not self.args: return self.__class__.__name__ return '{0}: {1}'.format(self.__class__.__name__, super(KafkaError, self).__str__()) class IllegalStateError(KafkaError): pass class IllegalArgumentError(KafkaError): pass class NoBrokersAvailable(KafkaError): retriable = True invalid_metadata = True class NodeNotReadyError(KafkaError): retriable = True class CorrelationIdError(KafkaError): retriable = True class Cancelled(KafkaError): retriable = True class TooManyInFlightRequests(KafkaError): retriable = True class StaleMetadata(KafkaError): retriable = True invalid_metadata = True class UnrecognizedBrokerVersion(KafkaError): pass class CommitFailedError(KafkaError): pass class AuthenticationMethodNotSupported(KafkaError): pass class AuthenticationFailedError(KafkaError): retriable = False class BrokerResponseError(KafkaError): errno = None message = None description = None def __str__(self): """Add errno to standard KafkaError str""" return '[Error {0}] {1}'.format( self.errno, super(BrokerResponseError, self).__str__()) class NoError(BrokerResponseError): errno = 0 message = 'NO_ERROR' description = 'No error--it worked!' class UnknownError(BrokerResponseError): errno = -1 message = 'UNKNOWN' description = 'An unexpected server error.' class OffsetOutOfRangeError(BrokerResponseError): errno = 1 message = 'OFFSET_OUT_OF_RANGE' description = ('The requested offset is outside the range of offsets' ' maintained by the server for the given topic/partition.') class InvalidMessageError(BrokerResponseError): errno = 2 message = 'INVALID_MESSAGE' description = ('This message has failed its CRC checksum, exceeds the' ' valid size, or is otherwise corrupt.') class UnknownTopicOrPartitionError(BrokerResponseError): errno = 3 message = 'UNKNOWN_TOPIC_OR_PARTITION' description = ('This request is for a topic or partition that does not' ' exist on this broker.') invalid_metadata = True class InvalidFetchRequestError(BrokerResponseError): errno = 4 message = 'INVALID_FETCH_SIZE' description = 'The message has a negative size.' class LeaderNotAvailableError(BrokerResponseError): errno = 5 message = 'LEADER_NOT_AVAILABLE' description = ('This error is thrown if we are in the middle of a' ' leadership election and there is currently no leader for' ' this partition and hence it is unavailable for writes.') retriable = True invalid_metadata = True class NotLeaderForPartitionError(BrokerResponseError): errno = 6 message = 'NOT_LEADER_FOR_PARTITION' description = ('This error is thrown if the client attempts to send' ' messages to a replica that is not the leader for some' ' partition. It indicates that the clients metadata is out' ' of date.') retriable = True invalid_metadata = True class RequestTimedOutError(BrokerResponseError): errno = 7 message = 'REQUEST_TIMED_OUT' description = ('This error is thrown if the request exceeds the' ' user-specified time limit in the request.') retriable = True class BrokerNotAvailableError(BrokerResponseError): errno = 8 message = 'BROKER_NOT_AVAILABLE' description = ('This is not a client facing error and is used mostly by' ' tools when a broker is not alive.') class ReplicaNotAvailableError(BrokerResponseError): errno = 9 message = 'REPLICA_NOT_AVAILABLE' description = ('If replica is expected on a broker, but is not (this can be' ' safely ignored).') class MessageSizeTooLargeError(BrokerResponseError): errno = 10 message = 'MESSAGE_SIZE_TOO_LARGE' description = ('The server has a configurable maximum message size to avoid' ' unbounded memory allocation. This error is thrown if the' ' client attempt to produce a message larger than this' ' maximum.') class StaleControllerEpochError(BrokerResponseError): errno = 11 message = 'STALE_CONTROLLER_EPOCH' description = 'Internal error code for broker-to-broker communication.' class OffsetMetadataTooLargeError(BrokerResponseError): errno = 12 message = 'OFFSET_METADATA_TOO_LARGE' description = ('If you specify a string larger than configured maximum for' ' offset metadata.') # TODO is this deprecated? https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-ErrorCodes class StaleLeaderEpochCodeError(BrokerResponseError): errno = 13 message = 'STALE_LEADER_EPOCH_CODE' class GroupLoadInProgressError(BrokerResponseError): errno = 14 message = 'OFFSETS_LOAD_IN_PROGRESS' description = ('The broker returns this error code for an offset fetch' ' request if it is still loading offsets (after a leader' ' change for that offsets topic partition), or in response' ' to group membership requests (such as heartbeats) when' ' group metadata is being loaded by the coordinator.') retriable = True class GroupCoordinatorNotAvailableError(BrokerResponseError): errno = 15 message = 'CONSUMER_COORDINATOR_NOT_AVAILABLE' description = ('The broker returns this error code for group coordinator' ' requests, offset commits, and most group management' ' requests if the offsets topic has not yet been created, or' ' if the group coordinator is not active.') retriable = True class NotCoordinatorForGroupError(BrokerResponseError): errno = 16 message = 'NOT_COORDINATOR_FOR_CONSUMER' description = ('The broker returns this error code if it receives an offset' ' fetch or commit request for a group that it is not a' ' coordinator for.') retriable = True class InvalidTopicError(BrokerResponseError): errno = 17 message = 'INVALID_TOPIC' description = ('For a request which attempts to access an invalid topic' ' (e.g. one which has an illegal name), or if an attempt' ' is made to write to an internal topic (such as the' ' consumer offsets topic).') class RecordListTooLargeError(BrokerResponseError): errno = 18 message = 'RECORD_LIST_TOO_LARGE' description = ('If a message batch in a produce request exceeds the maximum' ' configured segment size.') class NotEnoughReplicasError(BrokerResponseError): errno = 19 message = 'NOT_ENOUGH_REPLICAS' description = ('Returned from a produce request when the number of in-sync' ' replicas is lower than the configured minimum and' ' requiredAcks is -1.') class NotEnoughReplicasAfterAppendError(BrokerResponseError): errno = 20 message = 'NOT_ENOUGH_REPLICAS_AFTER_APPEND' description = ('Returned from a produce request when the message was' ' written to the log, but with fewer in-sync replicas than' ' required.') class InvalidRequiredAcksError(BrokerResponseError): errno = 21 message = 'INVALID_REQUIRED_ACKS' description = ('Returned from a produce request if the requested' ' requiredAcks is invalid (anything other than -1, 1, or 0).') class IllegalGenerationError(BrokerResponseError): errno = 22 message = 'ILLEGAL_GENERATION' description = ('Returned from group membership requests (such as heartbeats)' ' when the generation id provided in the request is not the' ' current generation.') class InconsistentGroupProtocolError(BrokerResponseError): errno = 23 message = 'INCONSISTENT_GROUP_PROTOCOL' description = ('Returned in join group when the member provides a protocol' ' type or set of protocols which is not compatible with the current group.') class InvalidGroupIdError(BrokerResponseError): errno = 24 message = 'INVALID_GROUP_ID' description = 'Returned in join group when the groupId is empty or null.' class UnknownMemberIdError(BrokerResponseError): errno = 25 message = 'UNKNOWN_MEMBER_ID' description = ('Returned from group requests (offset commits/fetches,' ' heartbeats, etc) when the memberId is not in the current' ' generation.') class InvalidSessionTimeoutError(BrokerResponseError): errno = 26 message = 'INVALID_SESSION_TIMEOUT' description = ('Return in join group when the requested session timeout is' ' outside of the allowed range on the broker') class RebalanceInProgressError(BrokerResponseError): errno = 27 message = 'REBALANCE_IN_PROGRESS' description = ('Returned in heartbeat requests when the coordinator has' ' begun rebalancing the group. This indicates to the client' ' that it should rejoin the group.') class InvalidCommitOffsetSizeError(BrokerResponseError): errno = 28 message = 'INVALID_COMMIT_OFFSET_SIZE' description = ('This error indicates that an offset commit was rejected' ' because of oversize metadata.') class TopicAuthorizationFailedError(BrokerResponseError): errno = 29 message = 'TOPIC_AUTHORIZATION_FAILED' description = ('Returned by the broker when the client is not authorized to' ' access the requested topic.') class GroupAuthorizationFailedError(BrokerResponseError): errno = 30 message = 'GROUP_AUTHORIZATION_FAILED' description = ('Returned by the broker when the client is not authorized to' ' access a particular groupId.') class ClusterAuthorizationFailedError(BrokerResponseError): errno = 31 message = 'CLUSTER_AUTHORIZATION_FAILED' description = ('Returned by the broker when the client is not authorized to' ' use an inter-broker or administrative API.') class InvalidTimestampError(BrokerResponseError): errno = 32 message = 'INVALID_TIMESTAMP' description = ('The timestamp of the message is out of acceptable range.') class UnsupportedSaslMechanismError(BrokerResponseError): errno = 33 message = 'UNSUPPORTED_SASL_MECHANISM' description = ('The broker does not support the requested SASL mechanism.') class IllegalSaslStateError(BrokerResponseError): errno = 34 message = 'ILLEGAL_SASL_STATE' description = ('Request is not valid given the current SASL state.') class KafkaUnavailableError(KafkaError): pass class KafkaTimeoutError(KafkaError): pass class FailedPayloadsError(KafkaError): def __init__(self, payload, *args): super(FailedPayloadsError, self).__init__(*args) self.payload = payload class ConnectionError(KafkaError): retriable = True invalid_metadata = True class BufferUnderflowError(KafkaError): pass class ChecksumError(KafkaError): pass class ConsumerFetchSizeTooSmall(KafkaError): pass class ConsumerNoMoreData(KafkaError): pass class ConsumerTimeout(KafkaError): pass class ProtocolError(KafkaError): pass class UnsupportedCodecError(KafkaError): pass class KafkaConfigurationError(KafkaError): pass class QuotaViolationError(KafkaError): pass class AsyncProducerQueueFull(KafkaError): def __init__(self, failed_msgs, *args): super(AsyncProducerQueueFull, self).__init__(*args) self.failed_msgs = failed_msgs def _iter_broker_errors(): for name, obj in inspect.getmembers(sys.modules[__name__]): if inspect.isclass(obj) and issubclass(obj, BrokerResponseError) and obj != BrokerResponseError: yield obj kafka_errors = dict([(x.errno, x) for x in _iter_broker_errors()]) def for_code(error_code): return kafka_errors.get(error_code, UnknownError) def check_error(response): if isinstance(response, Exception): raise response if response.error: error_class = kafka_errors.get(response.error, UnknownError) raise error_class(response) RETRY_BACKOFF_ERROR_TYPES = ( KafkaUnavailableError, LeaderNotAvailableError, ConnectionError, FailedPayloadsError ) RETRY_REFRESH_ERROR_TYPES = ( NotLeaderForPartitionError, UnknownTopicOrPartitionError, LeaderNotAvailableError, ConnectionError ) RETRY_ERROR_TYPES = RETRY_BACKOFF_ERROR_TYPES + RETRY_REFRESH_ERROR_TYPES kafka-1.3.2/kafka/future.py0000644001271300127130000000465213025302127015323 0ustar dpowers00000000000000from __future__ import absolute_import import functools import logging log = logging.getLogger(__name__) class Future(object): error_on_callbacks = False # and errbacks def __init__(self): self.is_done = False self.value = None self.exception = None self._callbacks = [] self._errbacks = [] def succeeded(self): return self.is_done and not bool(self.exception) def failed(self): return self.is_done and bool(self.exception) def retriable(self): try: return self.exception.retriable except AttributeError: return False def success(self, value): assert not self.is_done, 'Future is already complete' self.value = value self.is_done = True if self._callbacks: self._call_backs('callback', self._callbacks, self.value) return self def failure(self, e): assert not self.is_done, 'Future is already complete' self.exception = e if type(e) is not type else e() assert isinstance(self.exception, BaseException), ( 'future failed without an exception') self.is_done = True self._call_backs('errback', self._errbacks, self.exception) return self def add_callback(self, f, *args, **kwargs): if args or kwargs: f = functools.partial(f, *args, **kwargs) if self.is_done and not self.exception: self._call_backs('callback', [f], self.value) else: self._callbacks.append(f) return self def add_errback(self, f, *args, **kwargs): if args or kwargs: f = functools.partial(f, *args, **kwargs) if self.is_done and self.exception: self._call_backs('errback', [f], self.exception) else: self._errbacks.append(f) return self def add_both(self, f, *args, **kwargs): self.add_callback(f, *args, **kwargs) self.add_errback(f, *args, **kwargs) return self def chain(self, future): self.add_callback(future.success) self.add_errback(future.failure) return self def _call_backs(self, back_type, backs, value): for f in backs: try: f(value) except Exception as e: log.exception('Error processing %s', back_type) if self.error_on_callbacks: raise e kafka-1.3.2/kafka/metrics/0000755001271300127130000000000013031057517015105 5ustar dpowers00000000000000kafka-1.3.2/kafka/metrics/__init__.py0000644001271300127130000000072613025302127017214 0ustar dpowers00000000000000from __future__ import absolute_import from .compound_stat import NamedMeasurable from .dict_reporter import DictReporter from .kafka_metric import KafkaMetric from .measurable import AnonMeasurable from .metric_config import MetricConfig from .metric_name import MetricName from .metrics import Metrics from .quota import Quota __all__ = [ 'AnonMeasurable', 'DictReporter', 'KafkaMetric', 'MetricConfig', 'MetricName', 'Metrics', 'NamedMeasurable', 'Quota' ] kafka-1.3.2/kafka/metrics/compound_stat.py0000644001271300127130000000141013025302127020323 0ustar dpowers00000000000000from __future__ import absolute_import import abc from kafka.metrics.stat import AbstractStat class AbstractCompoundStat(AbstractStat): """ A compound stat is a stat where a single measurement and associated data structure feeds many metrics. This is the example for a histogram which has many associated percentiles. """ __metaclass__ = abc.ABCMeta def stats(self): """ Return list of NamedMeasurable """ raise NotImplementedError class NamedMeasurable(object): def __init__(self, metric_name, measurable_stat): self._name = metric_name self._stat = measurable_stat @property def name(self): return self._name @property def stat(self): return self._stat kafka-1.3.2/kafka/metrics/dict_reporter.py0000644001271300127130000000500713025302127020317 0ustar dpowers00000000000000from __future__ import absolute_import import logging import threading from kafka.metrics.metrics_reporter import AbstractMetricsReporter logger = logging.getLogger(__name__) class DictReporter(AbstractMetricsReporter): """A basic dictionary based metrics reporter. Store all metrics in a two level dictionary of category > name > metric. """ def __init__(self, prefix=''): self._lock = threading.Lock() self._prefix = prefix if prefix else '' # never allow None self._store = {} def snapshot(self): """ Return a nested dictionary snapshot of all metrics and their values at this time. Example: { 'category': { 'metric1_name': 42.0, 'metric2_name': 'foo' } } """ return dict((category, dict((name, metric.value()) for name, metric in list(metrics.items()))) for category, metrics in list(self._store.items())) def init(self, metrics): for metric in metrics: self.metric_change(metric) def metric_change(self, metric): with self._lock: category = self.get_category(metric) if category not in self._store: self._store[category] = {} self._store[category][metric.metric_name.name] = metric def metric_removal(self, metric): with self._lock: category = self.get_category(metric) metrics = self._store.get(category, {}) removed = metrics.pop(metric.metric_name.name, None) if not metrics: self._store.pop(category, None) return removed def get_category(self, metric): """ Return a string category for the metric. The category is made up of this reporter's prefix and the metric's group and tags. Examples: prefix = 'foo', group = 'bar', tags = {'a': 1, 'b': 2} returns: 'foo.bar.a=1,b=2' prefix = 'foo', group = 'bar', tags = None returns: 'foo.bar' prefix = None, group = 'bar', tags = None returns: 'bar' """ tags = ','.join('%s=%s' % (k, v) for k, v in sorted(metric.metric_name.tags.items())) return '.'.join(x for x in [self._prefix, metric.metric_name.group, tags] if x) def configure(self, configs): pass def close(self): pass kafka-1.3.2/kafka/metrics/kafka_metric.py0000644001271300127130000000164513025302127020076 0ustar dpowers00000000000000from __future__ import absolute_import import time class KafkaMetric(object): # NOTE java constructor takes a lock instance def __init__(self, metric_name, measurable, config): if not metric_name: raise ValueError('metric_name must be non-empty') if not measurable: raise ValueError('measurable must be non-empty') self._metric_name = metric_name self._measurable = measurable self._config = config @property def metric_name(self): return self._metric_name @property def measurable(self): return self._measurable @property def config(self): return self._config @config.setter def config(self, config): self._config = config def value(self, time_ms=None): if time_ms is None: time_ms = time.time() * 1000 return self.measurable.measure(self.config, time_ms) kafka-1.3.2/kafka/metrics/measurable.py0000644001271300127130000000140213025302127017565 0ustar dpowers00000000000000from __future__ import absolute_import import abc class AbstractMeasurable(object): """A measurable quantity that can be registered as a metric""" @abc.abstractmethod def measure(self, config, now): """ Measure this quantity and return the result Arguments: config (MetricConfig): The configuration for this metric now (int): The POSIX time in milliseconds the measurement is being taken Returns: The measured value """ raise NotImplementedError class AnonMeasurable(AbstractMeasurable): def __init__(self, measure_fn): self._measure_fn = measure_fn def measure(self, config, now): return float(self._measure_fn(config, now)) kafka-1.3.2/kafka/metrics/measurable_stat.py0000644001271300127130000000076713025302127020635 0ustar dpowers00000000000000from __future__ import absolute_import import abc from kafka.metrics.measurable import AbstractMeasurable from kafka.metrics.stat import AbstractStat class AbstractMeasurableStat(AbstractStat, AbstractMeasurable): """ An AbstractMeasurableStat is an AbstractStat that is also an AbstractMeasurable (i.e. can produce a single floating point value). This is the interface used for most of the simple statistics such as Avg, Max, Count, etc. """ __metaclass__ = abc.ABCMeta kafka-1.3.2/kafka/metrics/metric_config.py0000644001271300127130000000220213025302127020254 0ustar dpowers00000000000000from __future__ import absolute_import import sys class MetricConfig(object): """Configuration values for metrics""" def __init__(self, quota=None, samples=2, event_window=sys.maxsize, time_window_ms=30 * 1000, tags=None): """ Arguments: quota (Quota, optional): Upper or lower bound of a value. samples (int, optional): Max number of samples kept per metric. event_window (int, optional): Max number of values per sample. time_window_ms (int, optional): Max age of an individual sample. tags (dict of {str: str}, optional): Tags for each metric. """ self.quota = quota self._samples = samples self.event_window = event_window self.time_window_ms = time_window_ms # tags should be OrderedDict (not supported in py26) self.tags = tags if tags else {} @property def samples(self): return self._samples @samples.setter def samples(self, value): if value < 1: raise ValueError('The number of samples must be at least 1.') self._samples = value kafka-1.3.2/kafka/metrics/metric_name.py0000644001271300127130000000653113025302127017740 0ustar dpowers00000000000000from __future__ import absolute_import import copy class MetricName(object): """ This class encapsulates a metric's name, logical group and its related attributes (tags). group, tags parameters can be used to create unique metric names. e.g. domainName:type=group,key1=val1,key2=val2 Usage looks something like this: # set up metrics: metric_tags = {'client-id': 'producer-1', 'topic': 'topic'} metric_config = MetricConfig(tags=metric_tags) # metrics is the global repository of metrics and sensors metrics = Metrics(metric_config) sensor = metrics.sensor('message-sizes') metric_name = metrics.metric_name('message-size-avg', 'producer-metrics', 'average message size') sensor.add(metric_name, Avg()) metric_name = metrics.metric_name('message-size-max', sensor.add(metric_name, Max()) tags = {'client-id': 'my-client', 'topic': 'my-topic'} metric_name = metrics.metric_name('message-size-min', 'producer-metrics', 'message minimum size', tags) sensor.add(metric_name, Min()) # as messages are sent we record the sizes sensor.record(message_size) """ def __init__(self, name, group, description=None, tags=None): """ Arguments: name (str): The name of the metric. group (str): The logical group name of the metrics to which this metric belongs. description (str, optional): A human-readable description to include in the metric. tags (dict, optional): Additional key/val attributes of the metric. """ if not (name and group): raise Exception('name and group must be non-empty.') if tags is not None and not isinstance(tags, dict): raise Exception('tags must be a dict if present.') self._name = name self._group = group self._description = description self._tags = copy.copy(tags) self._hash = 0 @property def name(self): return self._name @property def group(self): return self._group @property def description(self): return self._description @property def tags(self): return copy.copy(self._tags) def __hash__(self): if self._hash != 0: return self._hash prime = 31 result = 1 result = prime * result + hash(self.group) result = prime * result + hash(self.name) tags_hash = hash(frozenset(self.tags.items())) if self.tags else 0 result = prime * result + tags_hash self._hash = result return result def __eq__(self, other): if self is other: return True if other is None: return False return (type(self) == type(other) and self.group == other.group and self.name == other.name and self.tags == other.tags) def __ne__(self, other): return not self.__eq__(other) def __str__(self): return 'MetricName(name=%s, group=%s, description=%s, tags=%s)' % ( self.name, self.group, self.description, self.tags) kafka-1.3.2/kafka/metrics/metrics.py0000644001271300127130000002405013025302127017117 0ustar dpowers00000000000000from __future__ import absolute_import import logging import sys import time import threading from kafka.metrics import AnonMeasurable, KafkaMetric, MetricConfig, MetricName from kafka.metrics.stats import Sensor logger = logging.getLogger(__name__) class Metrics(object): """ A registry of sensors and metrics. A metric is a named, numerical measurement. A sensor is a handle to record numerical measurements as they occur. Each Sensor has zero or more associated metrics. For example a Sensor might represent message sizes and we might associate with this sensor a metric for the average, maximum, or other statistics computed off the sequence of message sizes that are recorded by the sensor. Usage looks something like this: # set up metrics: metrics = Metrics() # the global repository of metrics and sensors sensor = metrics.sensor('message-sizes') metric_name = MetricName('message-size-avg', 'producer-metrics') sensor.add(metric_name, Avg()) metric_name = MetricName('message-size-max', 'producer-metrics') sensor.add(metric_name, Max()) # as messages are sent we record the sizes sensor.record(message_size); """ def __init__(self, default_config=None, reporters=None, enable_expiration=False): """ Create a metrics repository with a default config, given metric reporters and the ability to expire eligible sensors Arguments: default_config (MetricConfig, optional): The default config reporters (list of AbstractMetricsReporter, optional): The metrics reporters enable_expiration (bool, optional): true if the metrics instance can garbage collect inactive sensors, false otherwise """ self._lock = threading.RLock() self._config = default_config or MetricConfig() self._sensors = {} self._metrics = {} self._children_sensors = {} self._reporters = reporters or [] for reporter in self._reporters: reporter.init([]) if enable_expiration: def expire_loop(): while True: # delay 30 seconds time.sleep(30) self.ExpireSensorTask.run(self) metrics_scheduler = threading.Thread(target=expire_loop) # Creating a daemon thread to not block shutdown metrics_scheduler.daemon = True metrics_scheduler.start() self.add_metric(self.metric_name('count', 'kafka-metrics-count', 'total number of registered metrics'), AnonMeasurable(lambda config, now: len(self._metrics))) @property def config(self): return self._config @property def metrics(self): """ Get all the metrics currently maintained and indexed by metricName """ return self._metrics def metric_name(self, name, group, description='', tags=None): """ Create a MetricName with the given name, group, description and tags, plus default tags specified in the metric configuration. Tag in tags takes precedence if the same tag key is specified in the default metric configuration. Arguments: name (str): The name of the metric group (str): logical group name of the metrics to which this metric belongs description (str, optional): A human-readable description to include in the metric tags (dict, optionals): additional key/value attributes of the metric """ combined_tags = dict(self.config.tags) combined_tags.update(tags or {}) return MetricName(name, group, description, combined_tags) def get_sensor(self, name): """ Get the sensor with the given name if it exists Arguments: name (str): The name of the sensor Returns: Sensor: The sensor or None if no such sensor exists """ if not name: raise ValueError('name must be non-empty') return self._sensors.get(name, None) def sensor(self, name, config=None, inactive_sensor_expiration_time_seconds=sys.maxsize, parents=None): """ Get or create a sensor with the given unique name and zero or more parent sensors. All parent sensors will receive every value recorded with this sensor. Arguments: name (str): The name of the sensor config (MetricConfig, optional): A default configuration to use for this sensor for metrics that don't have their own config inactive_sensor_expiration_time_seconds (int, optional): If no value if recorded on the Sensor for this duration of time, it is eligible for removal parents (list of Sensor): The parent sensors Returns: Sensor: The sensor that is created """ sensor = self.get_sensor(name) if sensor: return sensor with self._lock: sensor = self.get_sensor(name) if not sensor: sensor = Sensor(self, name, parents, config or self.config, inactive_sensor_expiration_time_seconds) self._sensors[name] = sensor if parents: for parent in parents: children = self._children_sensors.get(parent) if not children: children = [] self._children_sensors[parent] = children children.append(sensor) logger.debug('Added sensor with name %s', name) return sensor def remove_sensor(self, name): """ Remove a sensor (if it exists), associated metrics and its children. Arguments: name (str): The name of the sensor to be removed """ sensor = self._sensors.get(name) if sensor: child_sensors = None with sensor._lock: with self._lock: val = self._sensors.pop(name, None) if val and val == sensor: for metric in sensor.metrics: self.remove_metric(metric.metric_name) logger.debug('Removed sensor with name %s', name) child_sensors = self._children_sensors.pop(sensor, None) if child_sensors: for child_sensor in child_sensors: self.remove_sensor(child_sensor.name) def add_metric(self, metric_name, measurable, config=None): """ Add a metric to monitor an object that implements measurable. This metric won't be associated with any sensor. This is a way to expose existing values as metrics. Arguments: metricName (MetricName): The name of the metric measurable (AbstractMeasurable): The measurable that will be measured by this metric config (MetricConfig, optional): The configuration to use when measuring this measurable """ # NOTE there was a lock here, but i don't think it's needed metric = KafkaMetric(metric_name, measurable, config or self.config) self.register_metric(metric) def remove_metric(self, metric_name): """ Remove a metric if it exists and return it. Return None otherwise. If a metric is removed, `metric_removal` will be invoked for each reporter. Arguments: metric_name (MetricName): The name of the metric Returns: KafkaMetric: the removed `KafkaMetric` or None if no such metric exists """ with self._lock: metric = self._metrics.pop(metric_name, None) if metric: for reporter in self._reporters: reporter.metric_removal(metric) return metric def add_reporter(self, reporter): """Add a MetricReporter""" with self._lock: reporter.init(list(self.metrics.values())) self._reporters.append(reporter) def register_metric(self, metric): with self._lock: if metric.metric_name in self.metrics: raise ValueError('A metric named "%s" already exists, cannot' ' register another one.' % metric.metric_name) self.metrics[metric.metric_name] = metric for reporter in self._reporters: reporter.metric_change(metric) class ExpireSensorTask(object): """ This iterates over every Sensor and triggers a remove_sensor if it has expired. Package private for testing """ @staticmethod def run(metrics): items = list(metrics._sensors.items()) for name, sensor in items: # remove_sensor also locks the sensor object. This is fine # because synchronized is reentrant. There is however a minor # race condition here. Assume we have a parent sensor P and # child sensor C. Calling record on C would cause a record on # P as well. So expiration time for P == expiration time for C. # If the record on P happens via C just after P is removed, # that will cause C to also get removed. Since the expiration # time is typically high it is not expected to be a significant # concern and thus not necessary to optimize with sensor._lock: if sensor.has_expired(): logger.debug('Removing expired sensor %s', name) metrics.remove_sensor(name) def close(self): """Close this metrics repository.""" for reporter in self._reporters: reporter.close() kafka-1.3.2/kafka/metrics/metrics_reporter.py0000644001271300127130000000256613025302127021051 0ustar dpowers00000000000000from __future__ import absolute_import import abc class AbstractMetricsReporter(object): """ An abstract class to allow things to listen as new metrics are created so they can be reported. """ __metaclass__ = abc.ABCMeta @abc.abstractmethod def init(self, metrics): """ This is called when the reporter is first registered to initially register all existing metrics Arguments: metrics (list of KafkaMetric): All currently existing metrics """ raise NotImplementedError @abc.abstractmethod def metric_change(self, metric): """ This is called whenever a metric is updated or added Arguments: metric (KafkaMetric) """ raise NotImplementedError @abc.abstractmethod def metric_removal(self, metric): """ This is called whenever a metric is removed Arguments: metric (KafkaMetric) """ raise NotImplementedError @abc.abstractmethod def configure(self, configs): """ Configure this class with the given key-value pairs Arguments: configs (dict of {str, ?}) """ raise NotImplementedError @abc.abstractmethod def close(self): """Called when the metrics repository is closed.""" raise NotImplementedError kafka-1.3.2/kafka/metrics/quota.py0000644001271300127130000000215013025302127016577 0ustar dpowers00000000000000from __future__ import absolute_import class Quota(object): """An upper or lower bound for metrics""" def __init__(self, bound, is_upper): self._bound = bound self._upper = is_upper @staticmethod def upper_bound(upper_bound): return Quota(upper_bound, True) @staticmethod def lower_bound(lower_bound): return Quota(lower_bound, False) def is_upper_bound(self): return self._upper @property def bound(self): return self._bound def is_acceptable(self, value): return ((self.is_upper_bound() and value <= self.bound) or (not self.is_upper_bound() and value >= self.bound)) def __hash__(self): prime = 31 result = prime + self.bound return prime * result + self.is_upper_bound() def __eq__(self, other): if self is other: return True return (type(self) == type(other) and self.bound == other.bound and self.is_upper_bound() == other.is_upper_bound()) def __ne__(self, other): return not self.__eq__(other) kafka-1.3.2/kafka/metrics/stat.py0000644001271300127130000000116413025302127016425 0ustar dpowers00000000000000from __future__ import absolute_import import abc class AbstractStat(object): """ An AbstractStat is a quantity such as average, max, etc that is computed off the stream of updates to a sensor """ __metaclass__ = abc.ABCMeta @abc.abstractmethod def record(self, config, value, time_ms): """ Record the given value Arguments: config (MetricConfig): The configuration to use for this metric value (float): The value to record timeMs (int): The POSIX time in milliseconds this value occurred """ raise NotImplementedError kafka-1.3.2/kafka/metrics/stats/0000755001271300127130000000000013031057517016243 5ustar dpowers00000000000000kafka-1.3.2/kafka/metrics/stats/__init__.py0000644001271300127130000000066713025302127020356 0ustar dpowers00000000000000from __future__ import absolute_import from .avg import Avg from .count import Count from .histogram import Histogram from .max_stat import Max from .min_stat import Min from .percentile import Percentile from .percentiles import Percentiles from .rate import Rate from .sensor import Sensor from .total import Total __all__ = [ 'Avg', 'Count', 'Histogram', 'Max', 'Min', 'Percentile', 'Percentiles', 'Rate', 'Sensor', 'Total' ] kafka-1.3.2/kafka/metrics/stats/avg.py0000644001271300127130000000124613025302127017366 0ustar dpowers00000000000000from __future__ import absolute_import from kafka.metrics.stats.sampled_stat import AbstractSampledStat class Avg(AbstractSampledStat): """ An AbstractSampledStat that maintains a simple average over its samples. """ def __init__(self): super(Avg, self).__init__(0.0) def update(self, sample, config, value, now): sample.value += value def combine(self, samples, config, now): total_sum = 0 total_count = 0 for sample in samples: total_sum += sample.value total_count += sample.event_count if not total_count: return 0 return float(total_sum) / total_count kafka-1.3.2/kafka/metrics/stats/count.py0000644001271300127130000000074713025302127017746 0ustar dpowers00000000000000from __future__ import absolute_import from kafka.metrics.stats.sampled_stat import AbstractSampledStat class Count(AbstractSampledStat): """ An AbstractSampledStat that maintains a simple count of what it has seen. """ def __init__(self): super(Count, self).__init__(0.0) def update(self, sample, config, value, now): sample.value += 1.0 def combine(self, samples, config, now): return float(sum(sample.value for sample in samples)) kafka-1.3.2/kafka/metrics/stats/histogram.py0000644001271300127130000000547213025302127020613 0ustar dpowers00000000000000from __future__ import absolute_import import math class Histogram(object): def __init__(self, bin_scheme): self._hist = [0.0] * bin_scheme.bins self._count = 0.0 self._bin_scheme = bin_scheme def record(self, value): self._hist[self._bin_scheme.to_bin(value)] += 1.0 self._count += 1.0 def value(self, quantile): if self._count == 0.0: return float('NaN') _sum = 0.0 quant = float(quantile) for i, value in enumerate(self._hist[:-1]): _sum += value if _sum / self._count > quant: return self._bin_scheme.from_bin(i) return float('inf') @property def counts(self): return self._hist def clear(self): for i in range(self._hist): self._hist[i] = 0.0 self._count = 0 def __str__(self): values = ['%.10f:%.0f' % (self._bin_scheme.from_bin(i), value) for i, value in enumerate(self._hist[:-1])] values.append('%s:%s' % (float('inf'), self._hist[-1])) return '{%s}' % ','.join(values) class ConstantBinScheme(object): def __init__(self, bins, min_val, max_val): if bins < 2: raise ValueError('Must have at least 2 bins.') self._min = float(min_val) self._max = float(max_val) self._bins = int(bins) self._bucket_width = (max_val - min_val) / (bins - 2) @property def bins(self): return self._bins def from_bin(self, b): if b == 0: return float('-inf') elif b == self._bins - 1: return float('inf') else: return self._min + (b - 1) * self._bucket_width def to_bin(self, x): if x < self._min: return 0 elif x > self._max: return self._bins - 1 else: return int(((x - self._min) / self._bucket_width) + 1) class LinearBinScheme(object): def __init__(self, num_bins, max_val): self._bins = num_bins self._max = max_val self._scale = max_val / (num_bins * (num_bins - 1) / 2) @property def bins(self): return self._bins def from_bin(self, b): if b == self._bins - 1: return float('inf') else: unscaled = (b * (b + 1.0)) / 2.0 return unscaled * self._scale def to_bin(self, x): if x < 0.0: raise ValueError('Values less than 0.0 not accepted.') elif x > self._max: return self._bins - 1 else: scaled = x / self._scale return int(-0.5 + math.sqrt(2.0 * scaled + 0.25)) kafka-1.3.2/kafka/metrics/stats/max_stat.py0000644001271300127130000000104213025302127020423 0ustar dpowers00000000000000from __future__ import absolute_import from kafka.metrics.stats.sampled_stat import AbstractSampledStat class Max(AbstractSampledStat): """An AbstractSampledStat that gives the max over its samples.""" def __init__(self): super(Max, self).__init__(float('-inf')) def update(self, sample, config, value, now): sample.value = max(sample.value, value) def combine(self, samples, config, now): if not samples: return float('-inf') return float(max(sample.value for sample in samples)) kafka-1.3.2/kafka/metrics/stats/min_stat.py0000644001271300127130000000107013025302127020422 0ustar dpowers00000000000000from __future__ import absolute_import import sys from kafka.metrics.stats.sampled_stat import AbstractSampledStat class Min(AbstractSampledStat): """An AbstractSampledStat that gives the min over its samples.""" def __init__(self): super(Min, self).__init__(float(sys.maxsize)) def update(self, sample, config, value, now): sample.value = min(sample.value, value) def combine(self, samples, config, now): if not samples: return float(sys.maxsize) return float(min(sample.value for sample in samples)) kafka-1.3.2/kafka/metrics/stats/percentile.py0000644001271300127130000000052613025302127020743 0ustar dpowers00000000000000from __future__ import absolute_import class Percentile(object): def __init__(self, metric_name, percentile): self._metric_name = metric_name self._percentile = float(percentile) @property def name(self): return self._metric_name @property def percentile(self): return self._percentile kafka-1.3.2/kafka/metrics/stats/percentiles.py0000644001271300127130000000552213025302127021127 0ustar dpowers00000000000000from __future__ import absolute_import from kafka.metrics import AnonMeasurable, NamedMeasurable from kafka.metrics.compound_stat import AbstractCompoundStat from kafka.metrics.stats import Histogram from kafka.metrics.stats.sampled_stat import AbstractSampledStat class BucketSizing(object): CONSTANT = 0 LINEAR = 1 class Percentiles(AbstractSampledStat, AbstractCompoundStat): """A compound stat that reports one or more percentiles""" def __init__(self, size_in_bytes, bucketing, max_val, min_val=0.0, percentiles=None): super(Percentiles, self).__init__(0.0) self._percentiles = percentiles or [] self._buckets = int(size_in_bytes / 4) if bucketing == BucketSizing.CONSTANT: self._bin_scheme = Histogram.ConstantBinScheme(self._buckets, min_val, max_val) elif bucketing == BucketSizing.LINEAR: if min_val != 0.0: raise ValueError('Linear bucket sizing requires min_val' ' to be 0.0.') self.bin_scheme = Histogram.LinearBinScheme(self._buckets, max_val) else: ValueError('Unknown bucket type: %s' % bucketing) def stats(self): measurables = [] def make_measure_fn(pct): return lambda config, now: self.value(config, now, pct / 100.0) for percentile in self._percentiles: measure_fn = make_measure_fn(percentile.percentile) stat = NamedMeasurable(percentile.name, AnonMeasurable(measure_fn)) measurables.append(stat) return measurables def value(self, config, now, quantile): self.purge_obsolete_samples(config, now) count = sum(sample.event_count for sample in self._samples) if count == 0.0: return float('NaN') sum_val = 0.0 quant = float(quantile) for b in range(self._buckets): for sample in self._samples: assert type(sample) is self.HistogramSample hist = sample.histogram.counts sum_val += hist[b] if sum_val / count > quant: return self._bin_scheme.from_bin(b) return float('inf') def combine(self, samples, config, now): return self.value(config, now, 0.5) def new_sample(self, time_ms): return Percentiles.HistogramSample(self._bin_scheme, time_ms) def update(self, sample, config, value, time_ms): assert type(sample) is self.HistogramSample sample.histogram.record(value) class HistogramSample(AbstractSampledStat.Sample): def __init__(self, scheme, now): super(Percentiles.HistogramSample, self).__init__(0.0, now) self.histogram = Histogram(scheme) kafka-1.3.2/kafka/metrics/stats/rate.py0000644001271300127130000001066213025302127017546 0ustar dpowers00000000000000from __future__ import absolute_import from kafka.metrics.measurable_stat import AbstractMeasurableStat from kafka.metrics.stats.sampled_stat import AbstractSampledStat class TimeUnit(object): _names = { 'nanosecond': 0, 'microsecond': 1, 'millisecond': 2, 'second': 3, 'minute': 4, 'hour': 5, 'day': 6, } NANOSECONDS = _names['nanosecond'] MICROSECONDS = _names['microsecond'] MILLISECONDS = _names['millisecond'] SECONDS = _names['second'] MINUTES = _names['minute'] HOURS = _names['hour'] DAYS = _names['day'] @staticmethod def get_name(time_unit): return TimeUnit._names[time_unit] class Rate(AbstractMeasurableStat): """ The rate of the given quantity. By default this is the total observed over a set of samples from a sampled statistic divided by the elapsed time over the sample windows. Alternative AbstractSampledStat implementations can be provided, however, to record the rate of occurrences (e.g. the count of values measured over the time interval) or other such values. """ def __init__(self, time_unit=TimeUnit.SECONDS, sampled_stat=None): self._stat = sampled_stat or SampledTotal() self._unit = time_unit def unit_name(self): return TimeUnit.get_name(self._unit) def record(self, config, value, time_ms): self._stat.record(config, value, time_ms) def measure(self, config, now): value = self._stat.measure(config, now) return float(value) / self.convert(self.window_size(config, now)) def window_size(self, config, now): # purge old samples before we compute the window size self._stat.purge_obsolete_samples(config, now) """ Here we check the total amount of time elapsed since the oldest non-obsolete window. This give the total window_size of the batch which is the time used for Rate computation. However, there is an issue if we do not have sufficient data for e.g. if only 1 second has elapsed in a 30 second window, the measured rate will be very high. Hence we assume that the elapsed time is always N-1 complete windows plus whatever fraction of the final window is complete. Note that we could simply count the amount of time elapsed in the current window and add n-1 windows to get the total time, but this approach does not account for sleeps. AbstractSampledStat only creates samples whenever record is called, if no record is called for a period of time that time is not accounted for in window_size and produces incorrect results. """ total_elapsed_time_ms = now - self._stat.oldest(now).last_window_ms # Check how many full windows of data we have currently retained num_full_windows = int(total_elapsed_time_ms / config.time_window_ms) min_full_windows = config.samples - 1 # If the available windows are less than the minimum required, # add the difference to the totalElapsedTime if num_full_windows < min_full_windows: total_elapsed_time_ms += ((min_full_windows - num_full_windows) * config.time_window_ms) return total_elapsed_time_ms def convert(self, time_ms): if self._unit == TimeUnit.NANOSECONDS: return time_ms * 1000.0 * 1000.0 elif self._unit == TimeUnit.MICROSECONDS: return time_ms * 1000.0 elif self._unit == TimeUnit.MILLISECONDS: return time_ms elif self._unit == TimeUnit.SECONDS: return time_ms / 1000.0 elif self._unit == TimeUnit.MINUTES: return time_ms / (60.0 * 1000.0) elif self._unit == TimeUnit.HOURS: return time_ms / (60.0 * 60.0 * 1000.0) elif self._unit == TimeUnit.DAYS: return time_ms / (24.0 * 60.0 * 60.0 * 1000.0) else: raise ValueError('Unknown unit: %s' % self._unit) class SampledTotal(AbstractSampledStat): def __init__(self, initial_value=None): if initial_value is not None: raise ValueError('initial_value cannot be set on SampledTotal') super(SampledTotal, self).__init__(0.0) def update(self, sample, config, value, time_ms): sample.value += value def combine(self, samples, config, now): return float(sum(sample.value for sample in samples)) kafka-1.3.2/kafka/metrics/stats/sampled_stat.py0000644001271300127130000000660213025302127021272 0ustar dpowers00000000000000from __future__ import absolute_import import abc from kafka.metrics.measurable_stat import AbstractMeasurableStat class AbstractSampledStat(AbstractMeasurableStat): """ An AbstractSampledStat records a single scalar value measured over one or more samples. Each sample is recorded over a configurable window. The window can be defined by number of events or elapsed time (or both, if both are given the window is complete when *either* the event count or elapsed time criterion is met). All the samples are combined to produce the measurement. When a window is complete the oldest sample is cleared and recycled to begin recording the next sample. Subclasses of this class define different statistics measured using this basic pattern. """ __metaclass__ = abc.ABCMeta def __init__(self, initial_value): self._initial_value = initial_value self._samples = [] self._current = 0 @abc.abstractmethod def update(self, sample, config, value, time_ms): raise NotImplementedError @abc.abstractmethod def combine(self, samples, config, now): raise NotImplementedError def record(self, config, value, time_ms): sample = self.current(time_ms) if sample.is_complete(time_ms, config): sample = self._advance(config, time_ms) self.update(sample, config, float(value), time_ms) sample.event_count += 1 def new_sample(self, time_ms): return self.Sample(self._initial_value, time_ms) def measure(self, config, now): self.purge_obsolete_samples(config, now) return float(self.combine(self._samples, config, now)) def current(self, time_ms): if not self._samples: self._samples.append(self.new_sample(time_ms)) return self._samples[self._current] def oldest(self, now): if not self._samples: self._samples.append(self.new_sample(now)) oldest = self._samples[0] for sample in self._samples[1:]: if sample.last_window_ms < oldest.last_window_ms: oldest = sample return oldest def purge_obsolete_samples(self, config, now): """ Timeout any windows that have expired in the absence of any events """ expire_age = config.samples * config.time_window_ms for sample in self._samples: if now - sample.last_window_ms >= expire_age: sample.reset(now) def _advance(self, config, time_ms): self._current = (self._current + 1) % config.samples if self._current >= len(self._samples): sample = self.new_sample(time_ms) self._samples.append(sample) return sample else: sample = self.current(time_ms) sample.reset(time_ms) return sample class Sample(object): def __init__(self, initial_value, now): self.initial_value = initial_value self.event_count = 0 self.last_window_ms = now self.value = initial_value def reset(self, now): self.event_count = 0 self.last_window_ms = now self.value = self.initial_value def is_complete(self, time_ms, config): return (time_ms - self.last_window_ms >= config.time_window_ms or self.event_count >= config.event_window) kafka-1.3.2/kafka/metrics/stats/sensor.py0000644001271300127130000001200613031057471020124 0ustar dpowers00000000000000from __future__ import absolute_import import threading import time from kafka.errors import QuotaViolationError from kafka.metrics import KafkaMetric class Sensor(object): """ A sensor applies a continuous sequence of numerical values to a set of associated metrics. For example a sensor on message size would record a sequence of message sizes using the `record(double)` api and would maintain a set of metrics about request sizes such as the average or max. """ def __init__(self, registry, name, parents, config, inactive_sensor_expiration_time_seconds): if not name: raise ValueError('name must be non-empty') self._lock = threading.RLock() self._registry = registry self._name = name self._parents = parents or [] self._metrics = [] self._stats = [] self._config = config self._inactive_sensor_expiration_time_ms = ( inactive_sensor_expiration_time_seconds * 1000) self._last_record_time = time.time() * 1000 self._check_forest(set()) def _check_forest(self, sensors): """Validate that this sensor doesn't end up referencing itself.""" if self in sensors: raise ValueError('Circular dependency in sensors: %s is its own' 'parent.' % self.name) sensors.add(self) for parent in self._parents: parent._check_forest(sensors) @property def name(self): """ The name this sensor is registered with. This name will be unique among all registered sensors. """ return self._name @property def metrics(self): return tuple(self._metrics) def record(self, value=1.0, time_ms=None): """ Record a value at a known time. Arguments: value (double): The value we are recording time_ms (int): A POSIX timestamp in milliseconds. Default: The time when record() is evaluated (now) Raises: QuotaViolationException: if recording this value moves a metric beyond its configured maximum or minimum bound """ if time_ms is None: time_ms = time.time() * 1000 self._last_record_time = time_ms with self._lock: # XXX high volume, might be performance issue # increment all the stats for stat in self._stats: stat.record(self._config, value, time_ms) self._check_quotas(time_ms) for parent in self._parents: parent.record(value, time_ms) def _check_quotas(self, time_ms): """ Check if we have violated our quota for any metric that has a configured quota """ for metric in self._metrics: if metric.config and metric.config.quota: value = metric.value(time_ms) if not metric.config.quota.is_acceptable(value): raise QuotaViolationError("'%s' violated quota. Actual: " "%d, Threshold: %d" % (metric.metric_name, value, metric.config.quota.bound)) def add_compound(self, compound_stat, config=None): """ Register a compound statistic with this sensor which yields multiple measurable quantities (like a histogram) Arguments: stat (AbstractCompoundStat): The stat to register config (MetricConfig): The configuration for this stat. If None then the stat will use the default configuration for this sensor. """ if not compound_stat: raise ValueError('compound stat must be non-empty') self._stats.append(compound_stat) for named_measurable in compound_stat.stats(): metric = KafkaMetric(named_measurable.name, named_measurable.stat, config or self._config) self._registry.register_metric(metric) self._metrics.append(metric) def add(self, metric_name, stat, config=None): """ Register a metric with this sensor Arguments: metric_name (MetricName): The name of the metric stat (AbstractMeasurableStat): The statistic to keep config (MetricConfig): A special configuration for this metric. If None use the sensor default configuration. """ with self._lock: metric = KafkaMetric(metric_name, stat, config or self._config) self._registry.register_metric(metric) self._metrics.append(metric) self._stats.append(stat) def has_expired(self): """ Return True if the Sensor is eligible for removal due to inactivity. """ return ((time.time() * 1000 - self._last_record_time) > self._inactive_sensor_expiration_time_ms) kafka-1.3.2/kafka/metrics/stats/total.py0000644001271300127130000000064213025302127017733 0ustar dpowers00000000000000from __future__ import absolute_import from kafka.metrics.measurable_stat import AbstractMeasurableStat class Total(AbstractMeasurableStat): """An un-windowed cumulative total maintained over all time.""" def __init__(self, value=0.0): self._total = value def record(self, config, value, now): self._total += value def measure(self, config, now): return float(self._total) kafka-1.3.2/kafka/partitioner/0000755001271300127130000000000013031057517015777 5ustar dpowers00000000000000kafka-1.3.2/kafka/partitioner/__init__.py0000644001271300127130000000052013031057471020104 0ustar dpowers00000000000000from __future__ import absolute_import from .default import DefaultPartitioner from .hashed import HashedPartitioner, Murmur2Partitioner, LegacyPartitioner from .roundrobin import RoundRobinPartitioner __all__ = [ 'DefaultPartitioner', 'RoundRobinPartitioner', 'HashedPartitioner', 'Murmur2Partitioner', 'LegacyPartitioner' ] kafka-1.3.2/kafka/partitioner/base.py0000644001271300127130000000161213031057471017262 0ustar dpowers00000000000000from __future__ import absolute_import class Partitioner(object): """ Base class for a partitioner """ def __init__(self, partitions=None): """ Initialize the partitioner Arguments: partitions: A list of available partitions (during startup) OPTIONAL. """ self.partitions = partitions def __call__(self, key, all_partitions=None, available_partitions=None): """ Takes a string key, num_partitions and available_partitions as argument and returns a partition to be used for the message Arguments: key: the key to use for partitioning. all_partitions: a list of the topic's partitions. available_partitions: a list of the broker's currently avaliable partitions(optional). """ raise NotImplementedError('partition function has to be implemented') kafka-1.3.2/kafka/partitioner/default.py0000644001271300127130000000177513031057471020006 0ustar dpowers00000000000000from __future__ import absolute_import import random from .hashed import murmur2 class DefaultPartitioner(object): """Default partitioner. Hashes key to partition using murmur2 hashing (from java client) If key is None, selects partition randomly from available, or from all partitions if none are currently available """ @classmethod def __call__(cls, key, all_partitions, available): """ Get the partition corresponding to key :param key: partitioning key :param all_partitions: list of all partitions sorted by partition ID :param available: list of available partitions in no particular order :return: one of the values from all_partitions or available """ if key is None: if available: return random.choice(available) return random.choice(all_partitions) idx = murmur2(key) idx &= 0x7fffffff idx %= len(all_partitions) return all_partitions[idx] kafka-1.3.2/kafka/partitioner/hashed.py0000644001271300127130000000627713031057471017620 0ustar dpowers00000000000000from __future__ import absolute_import from kafka.vendor import six from .base import Partitioner class Murmur2Partitioner(Partitioner): """ Implements a partitioner which selects the target partition based on the hash of the key. Attempts to apply the same hashing function as mainline java client. """ def __call__(self, key, partitions=None, available=None): if available: return self.partition(key, available) return self.partition(key, partitions) def partition(self, key, partitions=None): if not partitions: partitions = self.partitions # https://github.com/apache/kafka/blob/0.8.2/clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java#L69 idx = (murmur2(key) & 0x7fffffff) % len(partitions) return partitions[idx] class LegacyPartitioner(object): """DEPRECATED -- See Issue 374 Implements a partitioner which selects the target partition based on the hash of the key """ def __init__(self, partitions): self.partitions = partitions def partition(self, key, partitions=None): if not partitions: partitions = self.partitions size = len(partitions) idx = hash(key) % size return partitions[idx] # Default will change to Murmur2 in 0.10 release HashedPartitioner = LegacyPartitioner # https://github.com/apache/kafka/blob/0.8.2/clients/src/main/java/org/apache/kafka/common/utils/Utils.java#L244 def murmur2(data): """Pure-python Murmur2 implementation. Based on java client, see org.apache.kafka.common.utils.Utils.murmur2 Args: data (bytes): opaque bytes Returns: MurmurHash2 of data """ # Python2 bytes is really a str, causing the bitwise operations below to fail # so convert to bytearray. if six.PY2: data = bytearray(bytes(data)) length = len(data) seed = 0x9747b28c # 'm' and 'r' are mixing constants generated offline. # They're not really 'magic', they just happen to work well. m = 0x5bd1e995 r = 24 # Initialize the hash to a random value h = seed ^ length length4 = length // 4 for i in range(length4): i4 = i * 4 k = ((data[i4 + 0] & 0xff) + ((data[i4 + 1] & 0xff) << 8) + ((data[i4 + 2] & 0xff) << 16) + ((data[i4 + 3] & 0xff) << 24)) k &= 0xffffffff k *= m k &= 0xffffffff k ^= (k % 0x100000000) >> r # k ^= k >>> r k &= 0xffffffff k *= m k &= 0xffffffff h *= m h &= 0xffffffff h ^= k h &= 0xffffffff # Handle the last few bytes of the input array extra_bytes = length % 4 if extra_bytes >= 3: h ^= (data[(length & ~3) + 2] & 0xff) << 16 h &= 0xffffffff if extra_bytes >= 2: h ^= (data[(length & ~3) + 1] & 0xff) << 8 h &= 0xffffffff if extra_bytes >= 1: h ^= (data[length & ~3] & 0xff) h &= 0xffffffff h *= m h &= 0xffffffff h ^= (h % 0x100000000) >> 13 # h >>> 13; h &= 0xffffffff h *= m h &= 0xffffffff h ^= (h % 0x100000000) >> 15 # h >>> 15; h &= 0xffffffff return h kafka-1.3.2/kafka/partitioner/roundrobin.py0000644001271300127130000000454013031057471020534 0ustar dpowers00000000000000from __future__ import absolute_import from .base import Partitioner class RoundRobinPartitioner(Partitioner): def __init__(self, partitions=None): self.partitions_iterable = CachedPartitionCycler(partitions) if partitions: self._set_partitions(partitions) else: self.partitions = None def __call__(self, key, all_partitions=None, available_partitions=None): if available_partitions: cur_partitions = available_partitions else: cur_partitions = all_partitions if not self.partitions: self._set_partitions(cur_partitions) elif cur_partitions != self.partitions_iterable.partitions and cur_partitions is not None: self._set_partitions(cur_partitions) return next(self.partitions_iterable) def _set_partitions(self, available_partitions): self.partitions = available_partitions self.partitions_iterable.set_partitions(available_partitions) def partition(self, key, all_partitions=None, available_partitions=None): return self.__call__(key, all_partitions, available_partitions) class CachedPartitionCycler(object): def __init__(self, partitions=None): self.partitions = partitions if partitions: assert type(partitions) is list self.cur_pos = None def __next__(self): return self.next() @staticmethod def _index_available(cur_pos, partitions): return cur_pos < len(partitions) def set_partitions(self, partitions): if self.cur_pos: if not self._index_available(self.cur_pos, partitions): self.cur_pos = 0 self.partitions = partitions return None self.partitions = partitions next_item = self.partitions[self.cur_pos] if next_item in partitions: self.cur_pos = partitions.index(next_item) else: self.cur_pos = 0 return None self.partitions = partitions def next(self): assert self.partitions is not None if self.cur_pos is None or not self._index_available(self.cur_pos, self.partitions): self.cur_pos = 1 return self.partitions[0] cur_item = self.partitions[self.cur_pos] self.cur_pos += 1 return cur_item kafka-1.3.2/kafka/producer/0000755001271300127130000000000013031057517015262 5ustar dpowers00000000000000kafka-1.3.2/kafka/producer/__init__.py0000644001271300127130000000034413025302127017365 0ustar dpowers00000000000000from __future__ import absolute_import from .kafka import KafkaProducer from .simple import SimpleProducer from .keyed import KeyedProducer __all__ = [ 'KafkaProducer', 'SimpleProducer', 'KeyedProducer' # deprecated ] kafka-1.3.2/kafka/producer/base.py0000644001271300127130000004733613031057471016562 0ustar dpowers00000000000000from __future__ import absolute_import import atexit import logging import time try: from queue import Empty, Full, Queue # pylint: disable=import-error except ImportError: from Queue import Empty, Full, Queue # pylint: disable=import-error from collections import defaultdict from threading import Thread, Event from kafka.vendor import six from kafka.structs import ( ProduceRequestPayload, ProduceResponsePayload, TopicPartition, RetryOptions) from kafka.errors import ( kafka_errors, UnsupportedCodecError, FailedPayloadsError, RequestTimedOutError, AsyncProducerQueueFull, UnknownError, RETRY_ERROR_TYPES, RETRY_BACKOFF_ERROR_TYPES, RETRY_REFRESH_ERROR_TYPES) from kafka.protocol import CODEC_NONE, ALL_CODECS, create_message_set log = logging.getLogger('kafka.producer') BATCH_SEND_DEFAULT_INTERVAL = 20 BATCH_SEND_MSG_COUNT = 20 # unlimited ASYNC_QUEUE_MAXSIZE = 0 ASYNC_QUEUE_PUT_TIMEOUT = 0 # unlimited retries by default ASYNC_RETRY_LIMIT = None ASYNC_RETRY_BACKOFF_MS = 100 ASYNC_RETRY_ON_TIMEOUTS = True ASYNC_LOG_MESSAGES_ON_ERROR = True STOP_ASYNC_PRODUCER = -1 ASYNC_STOP_TIMEOUT_SECS = 30 SYNC_FAIL_ON_ERROR_DEFAULT = True def _send_upstream(queue, client, codec, batch_time, batch_size, req_acks, ack_timeout, retry_options, stop_event, log_messages_on_error=ASYNC_LOG_MESSAGES_ON_ERROR, stop_timeout=ASYNC_STOP_TIMEOUT_SECS, codec_compresslevel=None): """Private method to manage producing messages asynchronously Listens on the queue for a specified number of messages or until a specified timeout and then sends messages to the brokers in grouped requests (one per broker). Messages placed on the queue should be tuples that conform to this format: ((topic, partition), message, key) Currently does not mark messages with task_done. Do not attempt to join()! Arguments: queue (threading.Queue): the queue from which to get messages client (kafka.SimpleClient): instance to use for communicating with brokers codec (kafka.protocol.ALL_CODECS): compression codec to use batch_time (int): interval in seconds to send message batches batch_size (int): count of messages that will trigger an immediate send req_acks: required acks to use with ProduceRequests. see server protocol ack_timeout: timeout to wait for required acks. see server protocol retry_options (RetryOptions): settings for retry limits, backoff etc stop_event (threading.Event): event to monitor for shutdown signal. when this event is 'set', the producer will stop sending messages. log_messages_on_error (bool, optional): log stringified message-contents on any produce error, otherwise only log a hash() of the contents, defaults to True. stop_timeout (int or float, optional): number of seconds to continue retrying messages after stop_event is set, defaults to 30. """ request_tries = {} while not stop_event.is_set(): try: client.reinit() except Exception as e: log.warn('Async producer failed to connect to brokers; backoff for %s(ms) before retrying', retry_options.backoff_ms) time.sleep(float(retry_options.backoff_ms) / 1000) else: break stop_at = None while not (stop_event.is_set() and queue.empty() and not request_tries): # Handle stop_timeout if stop_event.is_set(): if not stop_at: stop_at = stop_timeout + time.time() if time.time() > stop_at: log.debug('Async producer stopping due to stop_timeout') break timeout = batch_time count = batch_size send_at = time.time() + timeout msgset = defaultdict(list) # Merging messages will require a bit more work to manage correctly # for now, don't look for new batches if we have old ones to retry if request_tries: count = 0 log.debug('Skipping new batch collection to handle retries') else: log.debug('Batching size: %s, timeout: %s', count, timeout) # Keep fetching till we gather enough messages or a # timeout is reached while count > 0 and timeout >= 0: try: topic_partition, msg, key = queue.get(timeout=timeout) except Empty: break # Check if the controller has requested us to stop if topic_partition == STOP_ASYNC_PRODUCER: stop_event.set() break # Adjust the timeout to match the remaining period count -= 1 timeout = send_at - time.time() msgset[topic_partition].append((msg, key)) # Send collected requests upstream for topic_partition, msg in msgset.items(): messages = create_message_set(msg, codec, key, codec_compresslevel) req = ProduceRequestPayload( topic_partition.topic, topic_partition.partition, tuple(messages)) request_tries[req] = 0 if not request_tries: continue reqs_to_retry, error_cls = [], None retry_state = { 'do_backoff': False, 'do_refresh': False } def _handle_error(error_cls, request): if issubclass(error_cls, RETRY_ERROR_TYPES) or (retry_options.retry_on_timeouts and issubclass(error_cls, RequestTimedOutError)): reqs_to_retry.append(request) if issubclass(error_cls, RETRY_BACKOFF_ERROR_TYPES): retry_state['do_backoff'] |= True if issubclass(error_cls, RETRY_REFRESH_ERROR_TYPES): retry_state['do_refresh'] |= True requests = list(request_tries.keys()) log.debug('Sending: %s', requests) responses = client.send_produce_request(requests, acks=req_acks, timeout=ack_timeout, fail_on_error=False) log.debug('Received: %s', responses) for i, response in enumerate(responses): error_cls = None if isinstance(response, FailedPayloadsError): error_cls = response.__class__ orig_req = response.payload elif isinstance(response, ProduceResponsePayload) and response.error: error_cls = kafka_errors.get(response.error, UnknownError) orig_req = requests[i] if error_cls: _handle_error(error_cls, orig_req) log.error('%s sending ProduceRequestPayload (#%d of %d) ' 'to %s:%d with msgs %s', error_cls.__name__, (i + 1), len(requests), orig_req.topic, orig_req.partition, orig_req.messages if log_messages_on_error else hash(orig_req.messages)) if not reqs_to_retry: request_tries = {} continue # doing backoff before next retry if retry_state['do_backoff'] and retry_options.backoff_ms: log.warn('Async producer backoff for %s(ms) before retrying', retry_options.backoff_ms) time.sleep(float(retry_options.backoff_ms) / 1000) # refresh topic metadata before next retry if retry_state['do_refresh']: log.warn('Async producer forcing metadata refresh metadata before retrying') try: client.load_metadata_for_topics() except Exception: log.exception("Async producer couldn't reload topic metadata.") # Apply retry limit, dropping messages that are over request_tries = dict( (key, count + 1) for (key, count) in request_tries.items() if key in reqs_to_retry and (retry_options.limit is None or (count < retry_options.limit)) ) # Log messages we are going to retry for orig_req in request_tries.keys(): log.info('Retrying ProduceRequestPayload to %s:%d with msgs %s', orig_req.topic, orig_req.partition, orig_req.messages if log_messages_on_error else hash(orig_req.messages)) if request_tries or not queue.empty(): log.error('Stopped producer with {0} unsent messages' .format(len(request_tries) + queue.qsize())) class Producer(object): """ Base class to be used by producers Arguments: client (kafka.SimpleClient): instance to use for broker communications. If async=True, the background thread will use client.copy(), which is expected to return a thread-safe object. codec (kafka.protocol.ALL_CODECS): compression codec to use. req_acks (int, optional): A value indicating the acknowledgements that the server must receive before responding to the request, defaults to 1 (local ack). ack_timeout (int, optional): millisecond timeout to wait for the configured req_acks, defaults to 1000. sync_fail_on_error (bool, optional): whether sync producer should raise exceptions (True), or just return errors (False), defaults to True. async (bool, optional): send message using a background thread, defaults to False. batch_send_every_n (int, optional): If async is True, messages are sent in batches of this size, defaults to 20. batch_send_every_t (int or float, optional): If async is True, messages are sent immediately after this timeout in seconds, even if there are fewer than batch_send_every_n, defaults to 20. async_retry_limit (int, optional): number of retries for failed messages or None for unlimited, defaults to None / unlimited. async_retry_backoff_ms (int, optional): milliseconds to backoff on failed messages, defaults to 100. async_retry_on_timeouts (bool, optional): whether to retry on RequestTimedOutError, defaults to True. async_queue_maxsize (int, optional): limit to the size of the internal message queue in number of messages (not size), defaults to 0 (no limit). async_queue_put_timeout (int or float, optional): timeout seconds for queue.put in send_messages for async producers -- will only apply if async_queue_maxsize > 0 and the queue is Full, defaults to 0 (fail immediately on full queue). async_log_messages_on_error (bool, optional): set to False and the async producer will only log hash() contents on failed produce requests, defaults to True (log full messages). Hash logging will not allow you to identify the specific message that failed, but it will allow you to match failures with retries. async_stop_timeout (int or float, optional): seconds to continue attempting to send queued messages after producer.stop(), defaults to 30. Deprecated Arguments: batch_send (bool, optional): If True, messages are sent by a background thread in batches, defaults to False. Deprecated, use 'async' """ ACK_NOT_REQUIRED = 0 # No ack is required ACK_AFTER_LOCAL_WRITE = 1 # Send response after it is written to log ACK_AFTER_CLUSTER_COMMIT = -1 # Send response after data is committed DEFAULT_ACK_TIMEOUT = 1000 def __init__(self, client, req_acks=ACK_AFTER_LOCAL_WRITE, ack_timeout=DEFAULT_ACK_TIMEOUT, codec=None, codec_compresslevel=None, sync_fail_on_error=SYNC_FAIL_ON_ERROR_DEFAULT, async=False, batch_send=False, # deprecated, use async batch_send_every_n=BATCH_SEND_MSG_COUNT, batch_send_every_t=BATCH_SEND_DEFAULT_INTERVAL, async_retry_limit=ASYNC_RETRY_LIMIT, async_retry_backoff_ms=ASYNC_RETRY_BACKOFF_MS, async_retry_on_timeouts=ASYNC_RETRY_ON_TIMEOUTS, async_queue_maxsize=ASYNC_QUEUE_MAXSIZE, async_queue_put_timeout=ASYNC_QUEUE_PUT_TIMEOUT, async_log_messages_on_error=ASYNC_LOG_MESSAGES_ON_ERROR, async_stop_timeout=ASYNC_STOP_TIMEOUT_SECS): if async: assert batch_send_every_n > 0 assert batch_send_every_t > 0 assert async_queue_maxsize >= 0 self.client = client self.async = async self.req_acks = req_acks self.ack_timeout = ack_timeout self.stopped = False if codec is None: codec = CODEC_NONE elif codec not in ALL_CODECS: raise UnsupportedCodecError("Codec 0x%02x unsupported" % codec) self.codec = codec self.codec_compresslevel = codec_compresslevel if self.async: # Messages are sent through this queue self.queue = Queue(async_queue_maxsize) self.async_queue_put_timeout = async_queue_put_timeout async_retry_options = RetryOptions( limit=async_retry_limit, backoff_ms=async_retry_backoff_ms, retry_on_timeouts=async_retry_on_timeouts) self.thread_stop_event = Event() self.thread = Thread( target=_send_upstream, args=(self.queue, self.client.copy(), self.codec, batch_send_every_t, batch_send_every_n, self.req_acks, self.ack_timeout, async_retry_options, self.thread_stop_event), kwargs={'log_messages_on_error': async_log_messages_on_error, 'stop_timeout': async_stop_timeout, 'codec_compresslevel': self.codec_compresslevel} ) # Thread will die if main thread exits self.thread.daemon = True self.thread.start() def cleanup(obj): if not obj.stopped: obj.stop() self._cleanup_func = cleanup atexit.register(cleanup, self) else: self.sync_fail_on_error = sync_fail_on_error def send_messages(self, topic, partition, *msg): """Helper method to send produce requests. Note that msg type *must* be encoded to bytes by user. Passing unicode message will not work, for example you should encode before calling send_messages via something like `unicode_message.encode('utf-8')` All messages will set the message 'key' to None. Arguments: topic (str): name of topic for produce request partition (int): partition number for produce request *msg (bytes): one or more message payloads Returns: ResponseRequest returned by server Raises: FailedPayloadsError: low-level connection error, can be caused by networking failures, or a malformed request. ConnectionError: KafkaUnavailableError: all known brokers are down when attempting to refresh metadata. LeaderNotAvailableError: topic or partition is initializing or a broker failed and leadership election is in progress. NotLeaderForPartitionError: metadata is out of sync; the broker that the request was sent to is not the leader for the topic or partition. UnknownTopicOrPartitionError: the topic or partition has not been created yet and auto-creation is not available. AsyncProducerQueueFull: in async mode, if too many messages are unsent and remain in the internal queue. """ return self._send_messages(topic, partition, *msg) def _send_messages(self, topic, partition, *msg, **kwargs): key = kwargs.pop('key', None) # Guarantee that msg is actually a list or tuple (should always be true) if not isinstance(msg, (list, tuple)): raise TypeError("msg is not a list or tuple!") for m in msg: # The protocol allows to have key & payload with null values both, # (https://goo.gl/o694yN) but having (null,null) pair doesn't make sense. if m is None: if key is None: raise TypeError("key and payload can't be null in one") # Raise TypeError if any non-null message is not encoded as bytes elif not isinstance(m, six.binary_type): raise TypeError("all produce message payloads must be null or type bytes") # Raise TypeError if the key is not encoded as bytes if key is not None and not isinstance(key, six.binary_type): raise TypeError("the key must be type bytes") if self.async: for idx, m in enumerate(msg): try: item = (TopicPartition(topic, partition), m, key) if self.async_queue_put_timeout == 0: self.queue.put_nowait(item) else: self.queue.put(item, True, self.async_queue_put_timeout) except Full: raise AsyncProducerQueueFull( msg[idx:], 'Producer async queue overfilled. ' 'Current queue size %d.' % self.queue.qsize()) resp = [] else: messages = create_message_set([(m, key) for m in msg], self.codec, key, self.codec_compresslevel) req = ProduceRequestPayload(topic, partition, messages) try: resp = self.client.send_produce_request( [req], acks=self.req_acks, timeout=self.ack_timeout, fail_on_error=self.sync_fail_on_error ) except Exception: log.exception("Unable to send messages") raise return resp def stop(self, timeout=None): """ Stop the producer (async mode). Blocks until async thread completes. """ if timeout is not None: log.warning('timeout argument to stop() is deprecated - ' 'it will be removed in future release') if not self.async: log.warning('producer.stop() called, but producer is not async') return if self.stopped: log.warning('producer.stop() called, but producer is already stopped') return if self.async: self.queue.put((STOP_ASYNC_PRODUCER, None, None)) self.thread_stop_event.set() self.thread.join() if hasattr(self, '_cleanup_func'): # Remove cleanup handler now that we've stopped # py3 supports unregistering if hasattr(atexit, 'unregister'): atexit.unregister(self._cleanup_func) # pylint: disable=no-member # py2 requires removing from private attribute... else: # ValueError on list.remove() if the exithandler no longer exists # but that is fine here try: atexit._exithandlers.remove( # pylint: disable=no-member (self._cleanup_func, (self,), {})) except ValueError: pass del self._cleanup_func self.stopped = True def __del__(self): if self.async and not self.stopped: self.stop() kafka-1.3.2/kafka/producer/buffer.py0000644001271300127130000004052213025302127017101 0ustar dpowers00000000000000from __future__ import absolute_import, division import collections import io import threading import time from ..codec import (has_gzip, has_snappy, has_lz4, gzip_encode, snappy_encode, lz4_encode, lz4_encode_old_kafka) from .. import errors as Errors from ..metrics.stats import Rate from ..protocol.types import Int32, Int64 from ..protocol.message import MessageSet, Message class MessageSetBuffer(object): """Wrap a buffer for writing MessageSet batches. Arguments: buf (IO stream): a buffer for writing data. Typically BytesIO. batch_size (int): maximum number of bytes to write to the buffer. Keyword Arguments: compression_type ('gzip', 'snappy', None): compress messages before publishing. Default: None. """ _COMPRESSORS = { 'gzip': (has_gzip, gzip_encode, Message.CODEC_GZIP), 'snappy': (has_snappy, snappy_encode, Message.CODEC_SNAPPY), 'lz4': (has_lz4, lz4_encode, Message.CODEC_LZ4), 'lz4-old-kafka': (has_lz4, lz4_encode_old_kafka, Message.CODEC_LZ4), } def __init__(self, buf, batch_size, compression_type=None, message_version=0): if compression_type is not None: assert compression_type in self._COMPRESSORS, 'Unrecognized compression type' # Kafka 0.8/0.9 had a quirky lz4... if compression_type == 'lz4' and message_version == 0: compression_type = 'lz4-old-kafka' checker, encoder, attributes = self._COMPRESSORS[compression_type] assert checker(), 'Compression Libraries Not Found' self._compressor = encoder self._compression_attributes = attributes else: self._compressor = None self._compression_attributes = None self._message_version = message_version self._buffer = buf # Init MessageSetSize to 0 -- update on close self._buffer.seek(0) self._buffer.write(Int32.encode(0)) self._batch_size = batch_size self._closed = False self._messages = 0 self._bytes_written = 4 # Int32 header is 4 bytes self._final_size = None def append(self, offset, message): """Append a Message to the MessageSet. Arguments: offset (int): offset of the message message (Message or bytes): message struct or encoded bytes Returns: bytes written """ if isinstance(message, Message): encoded = message.encode() else: encoded = bytes(message) msg = Int64.encode(offset) + Int32.encode(len(encoded)) + encoded self._buffer.write(msg) self._messages += 1 self._bytes_written += len(msg) return len(msg) def has_room_for(self, key, value): if self._closed: return False if not self._messages: return True needed_bytes = MessageSet.HEADER_SIZE + Message.HEADER_SIZE if key is not None: needed_bytes += len(key) if value is not None: needed_bytes += len(value) return self._buffer.tell() + needed_bytes < self._batch_size def is_full(self): if self._closed: return True return self._buffer.tell() >= self._batch_size def close(self): # This method may be called multiple times on the same batch # i.e., on retries # we need to make sure we only close it out once # otherwise compressed messages may be double-compressed # see Issue 718 if not self._closed: if self._compressor: # TODO: avoid copies with bytearray / memoryview uncompressed_size = self._buffer.tell() self._buffer.seek(4) msg = Message(self._compressor(self._buffer.read(uncompressed_size - 4)), attributes=self._compression_attributes, magic=self._message_version) encoded = msg.encode() self._buffer.seek(4) self._buffer.write(Int64.encode(0)) # offset 0 for wrapper msg self._buffer.write(Int32.encode(len(encoded))) self._buffer.write(encoded) # Update the message set size (less the 4 byte header), # and return with buffer ready for full read() self._final_size = self._buffer.tell() self._buffer.seek(0) self._buffer.write(Int32.encode(self._final_size - 4)) self._buffer.seek(0) self._closed = True def size_in_bytes(self): return self._final_size or self._buffer.tell() def compression_rate(self): return self.size_in_bytes() / self._bytes_written def buffer(self): return self._buffer class SimpleBufferPool(object): """A simple pool of BytesIO objects with a weak memory ceiling.""" def __init__(self, memory, poolable_size, metrics=None, metric_group_prefix='producer-metrics'): """Create a new buffer pool. Arguments: memory (int): maximum memory that this buffer pool can allocate poolable_size (int): memory size per buffer to cache in the free list rather than deallocating """ self._poolable_size = poolable_size self._lock = threading.RLock() buffers = int(memory / poolable_size) if poolable_size else 0 self._free = collections.deque([io.BytesIO() for _ in range(buffers)]) self._waiters = collections.deque() self.wait_time = None if metrics: self.wait_time = metrics.sensor('bufferpool-wait-time') self.wait_time.add(metrics.metric_name( 'bufferpool-wait-ratio', metric_group_prefix, 'The fraction of time an appender waits for space allocation.'), Rate()) def allocate(self, size, max_time_to_block_ms): """ Allocate a buffer of the given size. This method blocks if there is not enough memory and the buffer pool is configured with blocking mode. Arguments: size (int): The buffer size to allocate in bytes [ignored] max_time_to_block_ms (int): The maximum time in milliseconds to block for buffer memory to be available Returns: io.BytesIO """ with self._lock: # check if we have a free buffer of the right size pooled if self._free: return self._free.popleft() elif self._poolable_size == 0: return io.BytesIO() else: # we are out of buffers and will have to block buf = None more_memory = threading.Condition(self._lock) self._waiters.append(more_memory) # loop over and over until we have a buffer or have reserved # enough memory to allocate one while buf is None: start_wait = time.time() more_memory.wait(max_time_to_block_ms / 1000.0) end_wait = time.time() if self.wait_time: self.wait_time.record(end_wait - start_wait) if self._free: buf = self._free.popleft() else: raise Errors.KafkaTimeoutError( "Failed to allocate memory within the configured" " max blocking time") # remove the condition for this thread to let the next thread # in line start getting memory removed = self._waiters.popleft() assert removed is more_memory, 'Wrong condition' # signal any additional waiters if there is more memory left # over for them if self._free and self._waiters: self._waiters[0].notify() # unlock and return the buffer return buf def deallocate(self, buf): """ Return buffers to the pool. If they are of the poolable size add them to the free list, otherwise just mark the memory as free. Arguments: buffer_ (io.BytesIO): The buffer to return """ with self._lock: # BytesIO.truncate here makes the pool somewhat pointless # but we stick with the BufferPool API until migrating to # bytesarray / memoryview. The buffer we return must not # expose any prior data on read(). buf.truncate(0) self._free.append(buf) if self._waiters: self._waiters[0].notify() def queued(self): """The number of threads blocked waiting on memory.""" with self._lock: return len(self._waiters) ''' class BufferPool(object): """ A pool of ByteBuffers kept under a given memory limit. This class is fairly specific to the needs of the producer. In particular it has the following properties: * There is a special "poolable size" and buffers of this size are kept in a free list and recycled * It is fair. That is all memory is given to the longest waiting thread until it has sufficient memory. This prevents starvation or deadlock when a thread asks for a large chunk of memory and needs to block until multiple buffers are deallocated. """ def __init__(self, memory, poolable_size): """Create a new buffer pool. Arguments: memory (int): maximum memory that this buffer pool can allocate poolable_size (int): memory size per buffer to cache in the free list rather than deallocating """ self._poolable_size = poolable_size self._lock = threading.RLock() self._free = collections.deque() self._waiters = collections.deque() self._total_memory = memory self._available_memory = memory #self.metrics = metrics; #self.waitTime = this.metrics.sensor("bufferpool-wait-time"); #MetricName metricName = metrics.metricName("bufferpool-wait-ratio", metricGrpName, "The fraction of time an appender waits for space allocation."); #this.waitTime.add(metricName, new Rate(TimeUnit.NANOSECONDS)); def allocate(self, size, max_time_to_block_ms): """ Allocate a buffer of the given size. This method blocks if there is not enough memory and the buffer pool is configured with blocking mode. Arguments: size (int): The buffer size to allocate in bytes max_time_to_block_ms (int): The maximum time in milliseconds to block for buffer memory to be available Returns: buffer Raises: InterruptedException If the thread is interrupted while blocked IllegalArgumentException if size is larger than the total memory controlled by the pool (and hence we would block forever) """ assert size <= self._total_memory, ( "Attempt to allocate %d bytes, but there is a hard limit of %d on" " memory allocations." % (size, self._total_memory)) with self._lock: # check if we have a free buffer of the right size pooled if (size == self._poolable_size and len(self._free) > 0): return self._free.popleft() # now check if the request is immediately satisfiable with the # memory on hand or if we need to block free_list_size = len(self._free) * self._poolable_size if self._available_memory + free_list_size >= size: # we have enough unallocated or pooled memory to immediately # satisfy the request self._free_up(size) self._available_memory -= size raise NotImplementedError() #return ByteBuffer.allocate(size) else: # we are out of memory and will have to block accumulated = 0 buf = None more_memory = threading.Condition(self._lock) self._waiters.append(more_memory) # loop over and over until we have a buffer or have reserved # enough memory to allocate one while (accumulated < size): start_wait = time.time() if not more_memory.wait(max_time_to_block_ms / 1000.0): raise Errors.KafkaTimeoutError( "Failed to allocate memory within the configured" " max blocking time") end_wait = time.time() #this.waitTime.record(endWait - startWait, time.milliseconds()); # check if we can satisfy this request from the free list, # otherwise allocate memory if (accumulated == 0 and size == self._poolable_size and self._free): # just grab a buffer from the free list buf = self._free.popleft() accumulated = size else: # we'll need to allocate memory, but we may only get # part of what we need on this iteration self._free_up(size - accumulated) got = min(size - accumulated, self._available_memory) self._available_memory -= got accumulated += got # remove the condition for this thread to let the next thread # in line start getting memory removed = self._waiters.popleft() assert removed is more_memory, 'Wrong condition' # signal any additional waiters if there is more memory left # over for them if (self._available_memory > 0 or len(self._free) > 0): if len(self._waiters) > 0: self._waiters[0].notify() # unlock and return the buffer if buf is None: raise NotImplementedError() #return ByteBuffer.allocate(size) else: return buf def _free_up(self, size): """ Attempt to ensure we have at least the requested number of bytes of memory for allocation by deallocating pooled buffers (if needed) """ while self._free and self._available_memory < size: self._available_memory += self._free.pop().capacity def deallocate(self, buffer_, size=None): """ Return buffers to the pool. If they are of the poolable size add them to the free list, otherwise just mark the memory as free. Arguments: buffer (io.BytesIO): The buffer to return size (int): The size of the buffer to mark as deallocated, note that this maybe smaller than buffer.capacity since the buffer may re-allocate itself during in-place compression """ with self._lock: if size is None: size = buffer_.capacity if (size == self._poolable_size and size == buffer_.capacity): buffer_.seek(0) buffer_.truncate() self._free.append(buffer_) else: self._available_memory += size if self._waiters: more_mem = self._waiters[0] more_mem.notify() def available_memory(self): """The total free memory both unallocated and in the free list.""" with self._lock: return self._available_memory + len(self._free) * self._poolable_size def unallocated_memory(self): """Get the unallocated memory (not in the free list or in use).""" with self._lock: return self._available_memory def queued(self): """The number of threads blocked waiting on memory.""" with self._lock: return len(self._waiters) def poolable_size(self): """The buffer size that will be retained in the free list after use.""" return self._poolable_size def total_memory(self): """The total memory managed by this pool.""" return self._total_memory ''' kafka-1.3.2/kafka/producer/future.py0000644001271300127130000000517013025302127017142 0ustar dpowers00000000000000from __future__ import absolute_import import collections import threading from .. import errors as Errors from ..future import Future class FutureProduceResult(Future): def __init__(self, topic_partition): super(FutureProduceResult, self).__init__() self.topic_partition = topic_partition self._latch = threading.Event() def success(self, value): ret = super(FutureProduceResult, self).success(value) self._latch.set() return ret def failure(self, error): ret = super(FutureProduceResult, self).failure(error) self._latch.set() return ret def wait(self, timeout=None): # wait() on python2.6 returns None instead of the flag value return self._latch.wait(timeout) or self._latch.is_set() class FutureRecordMetadata(Future): def __init__(self, produce_future, relative_offset, timestamp_ms, checksum, serialized_key_size, serialized_value_size): super(FutureRecordMetadata, self).__init__() self._produce_future = produce_future # packing args as a tuple is a minor speed optimization self.args = (relative_offset, timestamp_ms, checksum, serialized_key_size, serialized_value_size) produce_future.add_callback(self._produce_success) produce_future.add_errback(self.failure) def _produce_success(self, offset_and_timestamp): offset, produce_timestamp_ms = offset_and_timestamp # Unpacking from args tuple is minor speed optimization (relative_offset, timestamp_ms, checksum, serialized_key_size, serialized_value_size) = self.args if produce_timestamp_ms is not None: timestamp_ms = produce_timestamp_ms if offset != -1 and relative_offset is not None: offset += relative_offset tp = self._produce_future.topic_partition metadata = RecordMetadata(tp[0], tp[1], tp, offset, timestamp_ms, checksum, serialized_key_size, serialized_value_size) self.success(metadata) def get(self, timeout=None): if not self.is_done and not self._produce_future.wait(timeout): raise Errors.KafkaTimeoutError( "Timeout after waiting for %s secs." % timeout) assert self.is_done if self.failed(): raise self.exception # pylint: disable-msg=raising-bad-type return self.value RecordMetadata = collections.namedtuple( 'RecordMetadata', ['topic', 'partition', 'topic_partition', 'offset', 'timestamp', 'checksum', 'serialized_key_size', 'serialized_value_size']) kafka-1.3.2/kafka/producer/kafka.py0000644001271300127130000007653013031057471016723 0ustar dpowers00000000000000from __future__ import absolute_import import atexit import copy import logging import socket import threading import time import weakref from .. import errors as Errors from ..client_async import KafkaClient, selectors from ..metrics import MetricConfig, Metrics from ..partitioner.default import DefaultPartitioner from ..protocol.message import Message, MessageSet from ..serializer import Serializer from ..structs import TopicPartition from .future import FutureRecordMetadata, FutureProduceResult from .record_accumulator import AtomicInteger, RecordAccumulator from .sender import Sender log = logging.getLogger(__name__) PRODUCER_CLIENT_ID_SEQUENCE = AtomicInteger() class KafkaProducer(object): """A Kafka client that publishes records to the Kafka cluster. The producer is thread safe and sharing a single producer instance across threads will generally be faster than having multiple instances. The producer consists of a pool of buffer space that holds records that haven't yet been transmitted to the server as well as a background I/O thread that is responsible for turning these records into requests and transmitting them to the cluster. The send() method is asynchronous. When called it adds the record to a buffer of pending record sends and immediately returns. This allows the producer to batch together individual records for efficiency. The 'acks' config controls the criteria under which requests are considered complete. The "all" setting will result in blocking on the full commit of the record, the slowest but most durable setting. If the request fails, the producer can automatically retry, unless 'retries' is configured to 0. Enabling retries also opens up the possibility of duplicates (see the documentation on message delivery semantics for details: http://kafka.apache.org/documentation.html#semantics ). The producer maintains buffers of unsent records for each partition. These buffers are of a size specified by the 'batch_size' config. Making this larger can result in more batching, but requires more memory (since we will generally have one of these buffers for each active partition). By default a buffer is available to send immediately even if there is additional unused space in the buffer. However if you want to reduce the number of requests you can set 'linger_ms' to something greater than 0. This will instruct the producer to wait up to that number of milliseconds before sending a request in hope that more records will arrive to fill up the same batch. This is analogous to Nagle's algorithm in TCP. Note that records that arrive close together in time will generally batch together even with linger_ms=0 so under heavy load batching will occur regardless of the linger configuration; however setting this to something larger than 0 can lead to fewer, more efficient requests when not under maximal load at the cost of a small amount of latency. The buffer_memory controls the total amount of memory available to the producer for buffering. If records are sent faster than they can be transmitted to the server then this buffer space will be exhausted. When the buffer space is exhausted additional send calls will block. The key_serializer and value_serializer instruct how to turn the key and value objects the user provides into bytes. Keyword Arguments: bootstrap_servers: 'host[:port]' string (or list of 'host[:port]' strings) that the producer should contact to bootstrap initial cluster metadata. This does not have to be the full node list. It just needs to have at least one broker that will respond to a Metadata API Request. Default port is 9092. If no servers are specified, will default to localhost:9092. client_id (str): a name for this client. This string is passed in each request to servers and can be used to identify specific server-side log entries that correspond to this client. Default: 'kafka-python-producer-#' (appended with a unique number per instance) key_serializer (callable): used to convert user-supplied keys to bytes If not None, called as f(key), should return bytes. Default: None. value_serializer (callable): used to convert user-supplied message values to bytes. If not None, called as f(value), should return bytes. Default: None. acks (0, 1, 'all'): The number of acknowledgments the producer requires the leader to have received before considering a request complete. This controls the durability of records that are sent. The following settings are common: 0: Producer will not wait for any acknowledgment from the server. The message will immediately be added to the socket buffer and considered sent. No guarantee can be made that the server has received the record in this case, and the retries configuration will not take effect (as the client won't generally know of any failures). The offset given back for each record will always be set to -1. 1: Wait for leader to write the record to its local log only. Broker will respond without awaiting full acknowledgement from all followers. In this case should the leader fail immediately after acknowledging the record but before the followers have replicated it then the record will be lost. all: Wait for the full set of in-sync replicas to write the record. This guarantees that the record will not be lost as long as at least one in-sync replica remains alive. This is the strongest available guarantee. If unset, defaults to acks=1. compression_type (str): The compression type for all data generated by the producer. Valid values are 'gzip', 'snappy', 'lz4', or None. Compression is of full batches of data, so the efficacy of batching will also impact the compression ratio (more batching means better compression). Default: None. retries (int): Setting a value greater than zero will cause the client to resend any record whose send fails with a potentially transient error. Note that this retry is no different than if the client resent the record upon receiving the error. Allowing retries without setting max_in_flight_requests_per_connection to 1 will potentially change the ordering of records because if two batches are sent to a single partition, and the first fails and is retried but the second succeeds, then the records in the second batch may appear first. Default: 0. batch_size (int): Requests sent to brokers will contain multiple batches, one for each partition with data available to be sent. A small batch size will make batching less common and may reduce throughput (a batch size of zero will disable batching entirely). Default: 16384 linger_ms (int): The producer groups together any records that arrive in between request transmissions into a single batched request. Normally this occurs only under load when records arrive faster than they can be sent out. However in some circumstances the client may want to reduce the number of requests even under moderate load. This setting accomplishes this by adding a small amount of artificial delay; that is, rather than immediately sending out a record the producer will wait for up to the given delay to allow other records to be sent so that the sends can be batched together. This can be thought of as analogous to Nagle's algorithm in TCP. This setting gives the upper bound on the delay for batching: once we get batch_size worth of records for a partition it will be sent immediately regardless of this setting, however if we have fewer than this many bytes accumulated for this partition we will 'linger' for the specified time waiting for more records to show up. This setting defaults to 0 (i.e. no delay). Setting linger_ms=5 would have the effect of reducing the number of requests sent but would add up to 5ms of latency to records sent in the absense of load. Default: 0. partitioner (callable): Callable used to determine which partition each message is assigned to. Called (after key serialization): partitioner(key_bytes, all_partitions, available_partitions). The default partitioner implementation hashes each non-None key using the same murmur2 algorithm as the java client so that messages with the same key are assigned to the same partition. When a key is None, the message is delivered to a random partition (filtered to partitions with available leaders only, if possible). buffer_memory (int): The total bytes of memory the producer should use to buffer records waiting to be sent to the server. If records are sent faster than they can be delivered to the server the producer will block up to max_block_ms, raising an exception on timeout. In the current implementation, this setting is an approximation. Default: 33554432 (32MB) max_block_ms (int): Number of milliseconds to block during send() and partitions_for(). These methods can be blocked either because the buffer is full or metadata unavailable. Blocking in the user-supplied serializers or partitioner will not be counted against this timeout. Default: 60000. max_request_size (int): The maximum size of a request. This is also effectively a cap on the maximum record size. Note that the server has its own cap on record size which may be different from this. This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests. Default: 1048576. metadata_max_age_ms (int): The period of time in milliseconds after which we force a refresh of metadata even if we haven't seen any partition leadership changes to proactively discover any new brokers or partitions. Default: 300000 retry_backoff_ms (int): Milliseconds to backoff when retrying on errors. Default: 100. request_timeout_ms (int): Client request timeout in milliseconds. Default: 30000. receive_buffer_bytes (int): The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. Default: None (relies on system defaults). Java client defaults to 32768. send_buffer_bytes (int): The size of the TCP send buffer (SO_SNDBUF) to use when sending data. Default: None (relies on system defaults). Java client defaults to 131072. socket_options (list): List of tuple-arguments to socket.setsockopt to apply to broker connection sockets. Default: [(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)] reconnect_backoff_ms (int): The amount of time in milliseconds to wait before attempting to reconnect to a given host. Default: 50. max_in_flight_requests_per_connection (int): Requests are pipelined to kafka brokers up to this number of maximum requests per broker connection. Default: 5. security_protocol (str): Protocol used to communicate with brokers. Valid values are: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL. Default: PLAINTEXT. ssl_context (ssl.SSLContext): pre-configured SSLContext for wrapping socket connections. If provided, all other ssl_* configurations will be ignored. Default: None. ssl_check_hostname (bool): flag to configure whether ssl handshake should verify that the certificate matches the brokers hostname. default: true. ssl_cafile (str): optional filename of ca file to use in certificate veriication. default: none. ssl_certfile (str): optional filename of file in pem format containing the client certificate, as well as any ca certificates needed to establish the certificate's authenticity. default: none. ssl_keyfile (str): optional filename containing the client private key. default: none. ssl_password (str): optional password to be used when loading the certificate chain. default: none. ssl_crlfile (str): optional filename containing the CRL to check for certificate expiration. By default, no CRL check is done. When providing a file, only the leaf certificate will be checked against this CRL. The CRL can only be checked with Python 3.4+ or 2.7.9+. default: none. api_version (tuple): specify which kafka API version to use. For a full list of supported versions, see KafkaClient.API_VERSIONS If set to None, the client will attempt to infer the broker version by probing various APIs. Default: None api_version_auto_timeout_ms (int): number of milliseconds to throw a timeout exception from the constructor when checking the broker api version. Only applies if api_version set to 'auto' metric_reporters (list): A list of classes to use as metrics reporters. Implementing the AbstractMetricsReporter interface allows plugging in classes that will be notified of new metric creation. Default: [] metrics_num_samples (int): The number of samples maintained to compute metrics. Default: 2 metrics_sample_window_ms (int): The maximum age in milliseconds of samples used to compute metrics. Default: 30000 selector (selectors.BaseSelector): Provide a specific selector implementation to use for I/O multiplexing. Default: selectors.DefaultSelector sasl_mechanism (str): string picking sasl mechanism when security_protocol is SASL_PLAINTEXT or SASL_SSL. Currently only PLAIN is supported. Default: None sasl_plain_username (str): username for sasl PLAIN authentication. Default: None sasl_plain_password (str): password for sasl PLAIN authentication. Default: None Note: Configuration parameters are described in more detail at https://kafka.apache.org/0100/configuration.html#producerconfigs """ DEFAULT_CONFIG = { 'bootstrap_servers': 'localhost', 'client_id': None, 'key_serializer': None, 'value_serializer': None, 'acks': 1, 'compression_type': None, 'retries': 0, 'batch_size': 16384, 'linger_ms': 0, 'partitioner': DefaultPartitioner(), 'buffer_memory': 33554432, 'connections_max_idle_ms': 600000, # not implemented yet 'max_block_ms': 60000, 'max_request_size': 1048576, 'metadata_max_age_ms': 300000, 'retry_backoff_ms': 100, 'request_timeout_ms': 30000, 'receive_buffer_bytes': None, 'send_buffer_bytes': None, 'socket_options': [(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)], 'reconnect_backoff_ms': 50, 'max_in_flight_requests_per_connection': 5, 'security_protocol': 'PLAINTEXT', 'ssl_context': None, 'ssl_check_hostname': True, 'ssl_cafile': None, 'ssl_certfile': None, 'ssl_keyfile': None, 'ssl_crlfile': None, 'ssl_password': None, 'api_version': None, 'api_version_auto_timeout_ms': 2000, 'metric_reporters': [], 'metrics_num_samples': 2, 'metrics_sample_window_ms': 30000, 'selector': selectors.DefaultSelector, 'sasl_mechanism': None, 'sasl_plain_username': None, 'sasl_plain_password': None, } def __init__(self, **configs): log.debug("Starting the Kafka producer") # trace self.config = copy.copy(self.DEFAULT_CONFIG) for key in self.config: if key in configs: self.config[key] = configs.pop(key) # Only check for extra config keys in top-level class assert not configs, 'Unrecognized configs: %s' % configs if self.config['client_id'] is None: self.config['client_id'] = 'kafka-python-producer-%s' % \ PRODUCER_CLIENT_ID_SEQUENCE.increment() if self.config['acks'] == 'all': self.config['acks'] = -1 # api_version was previously a str. accept old format for now if isinstance(self.config['api_version'], str): deprecated = self.config['api_version'] if deprecated == 'auto': self.config['api_version'] = None else: self.config['api_version'] = tuple(map(int, deprecated.split('.'))) log.warning('use api_version=%s [tuple] -- "%s" as str is deprecated', str(self.config['api_version']), deprecated) # Configure metrics metrics_tags = {'client-id': self.config['client_id']} metric_config = MetricConfig(samples=self.config['metrics_num_samples'], time_window_ms=self.config['metrics_sample_window_ms'], tags=metrics_tags) reporters = [reporter() for reporter in self.config['metric_reporters']] self._metrics = Metrics(metric_config, reporters) client = KafkaClient(metrics=self._metrics, metric_group_prefix='producer', **self.config) # Get auto-discovered version from client if necessary if self.config['api_version'] is None: self.config['api_version'] = client.config['api_version'] if self.config['compression_type'] == 'lz4': assert self.config['api_version'] >= (0, 8, 2), 'LZ4 Requires >= Kafka 0.8.2 Brokers' message_version = 1 if self.config['api_version'] >= (0, 10) else 0 self._accumulator = RecordAccumulator(message_version=message_version, metrics=self._metrics, **self.config) self._metadata = client.cluster guarantee_message_order = bool(self.config['max_in_flight_requests_per_connection'] == 1) self._sender = Sender(client, self._metadata, self._accumulator, self._metrics, guarantee_message_order=guarantee_message_order, **self.config) self._sender.daemon = True self._sender.start() self._closed = False self._cleanup = self._cleanup_factory() atexit.register(self._cleanup) log.debug("Kafka producer started") def _cleanup_factory(self): """Build a cleanup clojure that doesn't increase our ref count""" _self = weakref.proxy(self) def wrapper(): try: _self.close() except (ReferenceError, AttributeError): pass return wrapper def _unregister_cleanup(self): if getattr(self, '_cleanup', None): if hasattr(atexit, 'unregister'): atexit.unregister(self._cleanup) # pylint: disable=no-member # py2 requires removing from private attribute... else: # ValueError on list.remove() if the exithandler no longer exists # but that is fine here try: atexit._exithandlers.remove( # pylint: disable=no-member (self._cleanup, (), {})) except ValueError: pass self._cleanup = None def __del__(self): self.close(timeout=0) def close(self, timeout=None): """Close this producer. Arguments: timeout (float, optional): timeout in seconds to wait for completion. """ # drop our atexit handler now to avoid leaks self._unregister_cleanup() if not hasattr(self, '_closed') or self._closed: log.info('Kafka producer closed') return if timeout is None: timeout = 999999999 assert timeout >= 0 log.info("Closing the Kafka producer with %s secs timeout.", timeout) #first_exception = AtomicReference() # this will keep track of the first encountered exception invoked_from_callback = bool(threading.current_thread() is self._sender) if timeout > 0: if invoked_from_callback: log.warning("Overriding close timeout %s secs to 0 in order to" " prevent useless blocking due to self-join. This" " means you have incorrectly invoked close with a" " non-zero timeout from the producer call-back.", timeout) else: # Try to close gracefully. if self._sender is not None: self._sender.initiate_close() self._sender.join(timeout) if self._sender is not None and self._sender.is_alive(): log.info("Proceeding to force close the producer since pending" " requests could not be completed within timeout %s.", timeout) self._sender.force_close() # Only join the sender thread when not calling from callback. if not invoked_from_callback: self._sender.join() self._metrics.close() try: self.config['key_serializer'].close() except AttributeError: pass try: self.config['value_serializer'].close() except AttributeError: pass self._closed = True log.debug("The Kafka producer has closed.") def partitions_for(self, topic): """Returns set of all known partitions for the topic.""" max_wait = self.config['max_block_ms'] / 1000.0 return self._wait_on_metadata(topic, max_wait) def send(self, topic, value=None, key=None, partition=None, timestamp_ms=None): """Publish a message to a topic. Arguments: topic (str): topic where the message will be published value (optional): message value. Must be type bytes, or be serializable to bytes via configured value_serializer. If value is None, key is required and message acts as a 'delete'. See kafka compaction documentation for more details: http://kafka.apache.org/documentation.html#compaction (compaction requires kafka >= 0.8.1) partition (int, optional): optionally specify a partition. If not set, the partition will be selected using the configured 'partitioner'. key (optional): a key to associate with the message. Can be used to determine which partition to send the message to. If partition is None (and producer's partitioner config is left as default), then messages with the same key will be delivered to the same partition (but if key is None, partition is chosen randomly). Must be type bytes, or be serializable to bytes via configured key_serializer. timestamp_ms (int, optional): epoch milliseconds (from Jan 1 1970 UTC) to use as the message timestamp. Defaults to current time. Returns: FutureRecordMetadata: resolves to RecordMetadata Raises: KafkaTimeoutError: if unable to fetch topic metadata, or unable to obtain memory buffer prior to configured max_block_ms """ assert value is not None or self.config['api_version'] >= (0, 8, 1), ( 'Null messages require kafka >= 0.8.1') assert not (value is None and key is None), 'Need at least one: key or value' key_bytes = value_bytes = None try: # first make sure the metadata for the topic is # available self._wait_on_metadata(topic, self.config['max_block_ms'] / 1000.0) key_bytes = self._serialize( self.config['key_serializer'], topic, key) value_bytes = self._serialize( self.config['value_serializer'], topic, value) partition = self._partition(topic, partition, key, value, key_bytes, value_bytes) message_size = MessageSet.HEADER_SIZE + Message.HEADER_SIZE if key_bytes is not None: message_size += len(key_bytes) if value_bytes is not None: message_size += len(value_bytes) self._ensure_valid_record_size(message_size) tp = TopicPartition(topic, partition) if timestamp_ms is None: timestamp_ms = int(time.time() * 1000) log.debug("Sending (key=%s value=%s) to %s", key, value, tp) result = self._accumulator.append(tp, timestamp_ms, key_bytes, value_bytes, self.config['max_block_ms']) future, batch_is_full, new_batch_created = result if batch_is_full or new_batch_created: log.debug("Waking up the sender since %s is either full or" " getting a new batch", tp) self._sender.wakeup() return future # handling exceptions and record the errors; # for API exceptions return them in the future, # for other exceptions raise directly except Errors.KafkaTimeoutError: raise except AssertionError: raise except Exception as e: log.debug("Exception occurred during message send: %s", e) return FutureRecordMetadata( FutureProduceResult(TopicPartition(topic, partition)), -1, None, None, len(key_bytes) if key_bytes is not None else -1, len(value_bytes) if value_bytes is not None else -1 ).failure(e) def flush(self, timeout=None): """ Invoking this method makes all buffered records immediately available to send (even if linger_ms is greater than 0) and blocks on the completion of the requests associated with these records. The post-condition of flush() is that any previously sent record will have completed (e.g. Future.is_done() == True). A request is considered completed when either it is successfully acknowledged according to the 'acks' configuration for the producer, or it results in an error. Other threads can continue sending messages while one thread is blocked waiting for a flush call to complete; however, no guarantee is made about the completion of messages sent after the flush call begins. Arguments: timeout (float, optional): timeout in seconds to wait for completion. """ log.debug("Flushing accumulated records in producer.") # trace self._accumulator.begin_flush() self._sender.wakeup() self._accumulator.await_flush_completion(timeout=timeout) def _ensure_valid_record_size(self, size): """Validate that the record size isn't too large.""" if size > self.config['max_request_size']: raise Errors.MessageSizeTooLargeError( "The message is %d bytes when serialized which is larger than" " the maximum request size you have configured with the" " max_request_size configuration" % size) if size > self.config['buffer_memory']: raise Errors.MessageSizeTooLargeError( "The message is %d bytes when serialized which is larger than" " the total memory buffer you have configured with the" " buffer_memory configuration." % size) def _wait_on_metadata(self, topic, max_wait): """ Wait for cluster metadata including partitions for the given topic to be available. Arguments: topic (str): topic we want metadata for max_wait (float): maximum time in secs for waiting on the metadata Returns: set: partition ids for the topic Raises: TimeoutException: if partitions for topic were not obtained before specified max_wait timeout """ # add topic to metadata topic list if it is not there already. self._sender.add_topic(topic) begin = time.time() elapsed = 0.0 metadata_event = None while True: partitions = self._metadata.partitions_for_topic(topic) if partitions is not None: return partitions if not metadata_event: metadata_event = threading.Event() log.debug("Requesting metadata update for topic %s", topic) metadata_event.clear() future = self._metadata.request_update() future.add_both(lambda e, *args: e.set(), metadata_event) self._sender.wakeup() metadata_event.wait(max_wait - elapsed) elapsed = time.time() - begin if not metadata_event.is_set(): raise Errors.KafkaTimeoutError( "Failed to update metadata after %s secs.", max_wait) elif topic in self._metadata.unauthorized_topics: raise Errors.TopicAuthorizationFailedError(topic) else: log.debug("_wait_on_metadata woke after %s secs.", elapsed) def _serialize(self, f, topic, data): if not f: return data if isinstance(f, Serializer): return f.serialize(topic, data) return f(data) def _partition(self, topic, partition, key, value, serialized_key, serialized_value): if partition is not None: assert partition >= 0 assert partition in self._metadata.partitions_for_topic(topic), 'Unrecognized partition' return partition all_partitions = sorted(self._metadata.partitions_for_topic(topic)) available = list(self._metadata.available_partitions_for_topic(topic)) return self.config['partitioner'](serialized_key, all_partitions, available) def metrics(self, raw=False): """Warning: this is an unstable interface. It may change in future releases without warning""" if raw: return self._metrics.metrics metrics = {} for k, v in self._metrics.metrics.items(): if k.group not in metrics: metrics[k.group] = {} if k.name not in metrics[k.group]: metrics[k.group][k.name] = {} metrics[k.group][k.name] = v.value() return metrics kafka-1.3.2/kafka/producer/keyed.py0000644001271300127130000000320712702044544016737 0ustar dpowers00000000000000from __future__ import absolute_import import logging import warnings from .base import Producer from ..partitioner import HashedPartitioner log = logging.getLogger(__name__) class KeyedProducer(Producer): """ A producer which distributes messages to partitions based on the key See Producer class for Arguments Additional Arguments: partitioner: A partitioner class that will be used to get the partition to send the message to. Must be derived from Partitioner. Defaults to HashedPartitioner. """ def __init__(self, *args, **kwargs): self.partitioner_class = kwargs.pop('partitioner', HashedPartitioner) self.partitioners = {} super(KeyedProducer, self).__init__(*args, **kwargs) def _next_partition(self, topic, key): if topic not in self.partitioners: if not self.client.has_metadata_for_topic(topic): self.client.load_metadata_for_topics(topic, ignore_leadernotavailable=True) self.partitioners[topic] = self.partitioner_class(self.client.get_partition_ids_for_topic(topic)) partitioner = self.partitioners[topic] return partitioner.partition(key) def send_messages(self, topic, key, *msg): partition = self._next_partition(topic, key) return self._send_messages(topic, partition, *msg, key=key) # DEPRECATED def send(self, topic, key, msg): warnings.warn("KeyedProducer.send is deprecated in favor of send_messages", DeprecationWarning) return self.send_messages(topic, key, msg) def __repr__(self): return '' % self.async kafka-1.3.2/kafka/producer/record_accumulator.py0000644001271300127130000006027213031057471021517 0ustar dpowers00000000000000from __future__ import absolute_import import collections import copy import logging import threading import time from .. import errors as Errors from ..protocol.message import Message, MessageSet from .buffer import MessageSetBuffer, SimpleBufferPool from .future import FutureRecordMetadata, FutureProduceResult from ..structs import TopicPartition log = logging.getLogger(__name__) class AtomicInteger(object): def __init__(self, val=0): self._lock = threading.Lock() self._val = val def increment(self): with self._lock: self._val += 1 return self._val def decrement(self): with self._lock: self._val -= 1 return self._val def get(self): return self._val class RecordBatch(object): def __init__(self, tp, records, message_version=0): self.record_count = 0 self.max_record_size = 0 now = time.time() self.created = now self.drained = None self.attempts = 0 self.last_attempt = now self.last_append = now self.records = records self.message_version = message_version self.topic_partition = tp self.produce_future = FutureProduceResult(tp) self._retry = False def try_append(self, timestamp_ms, key, value): if not self.records.has_room_for(key, value): return None if self.message_version == 0: msg = Message(value, key=key, magic=self.message_version) else: msg = Message(value, key=key, magic=self.message_version, timestamp=timestamp_ms) record_size = self.records.append(self.record_count, msg) checksum = msg.crc # crc is recalculated during records.append() self.max_record_size = max(self.max_record_size, record_size) self.last_append = time.time() future = FutureRecordMetadata(self.produce_future, self.record_count, timestamp_ms, checksum, len(key) if key is not None else -1, len(value) if value is not None else -1) self.record_count += 1 return future def done(self, base_offset=None, timestamp_ms=None, exception=None): log.debug("Produced messages to topic-partition %s with base offset" " %s and error %s.", self.topic_partition, base_offset, exception) # trace if self.produce_future.is_done: log.warning('Batch is already closed -- ignoring batch.done()') return elif exception is None: self.produce_future.success((base_offset, timestamp_ms)) else: self.produce_future.failure(exception) def maybe_expire(self, request_timeout_ms, retry_backoff_ms, linger_ms, is_full): """Expire batches if metadata is not available A batch whose metadata is not available should be expired if one of the following is true: * the batch is not in retry AND request timeout has elapsed after it is ready (full or linger.ms has reached). * the batch is in retry AND request timeout has elapsed after the backoff period ended. """ now = time.time() since_append = now - self.last_append since_ready = now - (self.created + linger_ms / 1000.0) since_backoff = now - (self.last_attempt + retry_backoff_ms / 1000.0) timeout = request_timeout_ms / 1000.0 if ((not self.in_retry() and is_full and timeout < since_append) or (not self.in_retry() and timeout < since_ready) or (self.in_retry() and timeout < since_backoff)): self.records.close() self.done(-1, None, Errors.KafkaTimeoutError( "Batch containing %s record(s) expired due to timeout while" " requesting metadata from brokers for %s", self.record_count, self.topic_partition)) return True return False def in_retry(self): return self._retry def set_retry(self): self._retry = True def __str__(self): return 'RecordBatch(topic_partition=%s, record_count=%d)' % ( self.topic_partition, self.record_count) class RecordAccumulator(object): """ This class maintains a dequeue per TopicPartition that accumulates messages into MessageSets to be sent to the server. The accumulator attempts to bound memory use, and append calls will block when that memory is exhausted. Keyword Arguments: batch_size (int): Requests sent to brokers will contain multiple batches, one for each partition with data available to be sent. A small batch size will make batching less common and may reduce throughput (a batch size of zero will disable batching entirely). Default: 16384 buffer_memory (int): The total bytes of memory the producer should use to buffer records waiting to be sent to the server. If records are sent faster than they can be delivered to the server the producer will block up to max_block_ms, raising an exception on timeout. In the current implementation, this setting is an approximation. Default: 33554432 (32MB) compression_type (str): The compression type for all data generated by the producer. Valid values are 'gzip', 'snappy', 'lz4', or None. Compression is of full batches of data, so the efficacy of batching will also impact the compression ratio (more batching means better compression). Default: None. linger_ms (int): An artificial delay time to add before declaring a messageset (that isn't full) ready for sending. This allows time for more records to arrive. Setting a non-zero linger_ms will trade off some latency for potentially better throughput due to more batching (and hence fewer, larger requests). Default: 0 retry_backoff_ms (int): An artificial delay time to retry the produce request upon receiving an error. This avoids exhausting all retries in a short period of time. Default: 100 """ DEFAULT_CONFIG = { 'buffer_memory': 33554432, 'batch_size': 16384, 'compression_type': None, 'linger_ms': 0, 'retry_backoff_ms': 100, 'message_version': 0, 'metrics': None, 'metric_group_prefix': 'producer-metrics', } def __init__(self, **configs): self.config = copy.copy(self.DEFAULT_CONFIG) for key in self.config: if key in configs: self.config[key] = configs.pop(key) self._closed = False self._flushes_in_progress = AtomicInteger() self._appends_in_progress = AtomicInteger() self._batches = collections.defaultdict(collections.deque) # TopicPartition: [RecordBatch] self._tp_locks = {None: threading.Lock()} # TopicPartition: Lock, plus a lock to add entries self._free = SimpleBufferPool(self.config['buffer_memory'], self.config['batch_size'], metrics=self.config['metrics'], metric_group_prefix=self.config['metric_group_prefix']) self._incomplete = IncompleteRecordBatches() # The following variables should only be accessed by the sender thread, # so we don't need to protect them w/ locking. self.muted = set() self._drain_index = 0 def append(self, tp, timestamp_ms, key, value, max_time_to_block_ms): """Add a record to the accumulator, return the append result. The append result will contain the future metadata, and flag for whether the appended batch is full or a new batch is created Arguments: tp (TopicPartition): The topic/partition to which this record is being sent timestamp_ms (int): The timestamp of the record (epoch ms) key (bytes): The key for the record value (bytes): The value for the record max_time_to_block_ms (int): The maximum time in milliseconds to block for buffer memory to be available Returns: tuple: (future, batch_is_full, new_batch_created) """ assert isinstance(tp, TopicPartition), 'not TopicPartition' assert not self._closed, 'RecordAccumulator is closed' # We keep track of the number of appending thread to make sure we do not miss batches in # abortIncompleteBatches(). self._appends_in_progress.increment() try: if tp not in self._tp_locks: with self._tp_locks[None]: if tp not in self._tp_locks: self._tp_locks[tp] = threading.Lock() with self._tp_locks[tp]: # check if we have an in-progress batch dq = self._batches[tp] if dq: last = dq[-1] future = last.try_append(timestamp_ms, key, value) if future is not None: batch_is_full = len(dq) > 1 or last.records.is_full() return future, batch_is_full, False # we don't have an in-progress record batch try to allocate a new batch message_size = MessageSet.HEADER_SIZE + Message.HEADER_SIZE if key is not None: message_size += len(key) if value is not None: message_size += len(value) assert message_size <= self.config['buffer_memory'], 'message too big' size = max(self.config['batch_size'], message_size) log.debug("Allocating a new %d byte message buffer for %s", size, tp) # trace buf = self._free.allocate(size, max_time_to_block_ms) with self._tp_locks[tp]: # Need to check if producer is closed again after grabbing the # dequeue lock. assert not self._closed, 'RecordAccumulator is closed' if dq: last = dq[-1] future = last.try_append(timestamp_ms, key, value) if future is not None: # Somebody else found us a batch, return the one we # waited for! Hopefully this doesn't happen often... self._free.deallocate(buf) batch_is_full = len(dq) > 1 or last.records.is_full() return future, batch_is_full, False records = MessageSetBuffer(buf, self.config['batch_size'], self.config['compression_type'], self.config['message_version']) batch = RecordBatch(tp, records, self.config['message_version']) future = batch.try_append(timestamp_ms, key, value) if not future: raise Exception() dq.append(batch) self._incomplete.add(batch) batch_is_full = len(dq) > 1 or batch.records.is_full() return future, batch_is_full, True finally: self._appends_in_progress.decrement() def abort_expired_batches(self, request_timeout_ms, cluster): """Abort the batches that have been sitting in RecordAccumulator for more than the configured request_timeout due to metadata being unavailable. Arguments: request_timeout_ms (int): milliseconds to timeout cluster (ClusterMetadata): current metadata for kafka cluster Returns: list of RecordBatch that were expired """ expired_batches = [] to_remove = [] count = 0 for tp in list(self._batches.keys()): assert tp in self._tp_locks, 'TopicPartition not in locks dict' # We only check if the batch should be expired if the partition # does not have a batch in flight. This is to avoid the later # batches get expired when an earlier batch is still in progress. # This protection only takes effect when user sets # max.in.flight.request.per.connection=1. Otherwise the expiration # order is not guranteed. if tp in self.muted: continue with self._tp_locks[tp]: # iterate over the batches and expire them if they have stayed # in accumulator for more than request_timeout_ms dq = self._batches[tp] for batch in dq: is_full = bool(bool(batch != dq[-1]) or batch.records.is_full()) # check if the batch is expired if batch.maybe_expire(request_timeout_ms, self.config['retry_backoff_ms'], self.config['linger_ms'], is_full): expired_batches.append(batch) to_remove.append(batch) count += 1 self.deallocate(batch) else: # Stop at the first batch that has not expired. break # Python does not allow us to mutate the dq during iteration # Assuming expired batches are infrequent, this is better than # creating a new copy of the deque for iteration on every loop if to_remove: for batch in to_remove: dq.remove(batch) to_remove = [] if expired_batches: log.debug("Expired %d batches in accumulator", count) # trace return expired_batches def reenqueue(self, batch): """Re-enqueue the given record batch in the accumulator to retry.""" now = time.time() batch.attempts += 1 batch.last_attempt = now batch.last_append = now batch.set_retry() assert batch.topic_partition in self._tp_locks, 'TopicPartition not in locks dict' assert batch.topic_partition in self._batches, 'TopicPartition not in batches' dq = self._batches[batch.topic_partition] with self._tp_locks[batch.topic_partition]: dq.appendleft(batch) def ready(self, cluster): """ Get a list of nodes whose partitions are ready to be sent, and the earliest time at which any non-sendable partition will be ready; Also return the flag for whether there are any unknown leaders for the accumulated partition batches. A destination node is ready to send if: * There is at least one partition that is not backing off its send * and those partitions are not muted (to prevent reordering if max_in_flight_requests_per_connection is set to 1) * and any of the following are true: * The record set is full * The record set has sat in the accumulator for at least linger_ms milliseconds * The accumulator is out of memory and threads are blocking waiting for data (in this case all partitions are immediately considered ready). * The accumulator has been closed Arguments: cluster (ClusterMetadata): Returns: tuple: ready_nodes (set): node_ids that have ready batches next_ready_check (float): secs until next ready after backoff unknown_leaders_exist (bool): True if metadata refresh needed """ ready_nodes = set() next_ready_check = 9999999.99 unknown_leaders_exist = False now = time.time() exhausted = bool(self._free.queued() > 0) # several threads are accessing self._batches -- to simplify # concurrent access, we iterate over a snapshot of partitions # and lock each partition separately as needed partitions = list(self._batches.keys()) for tp in partitions: leader = cluster.leader_for_partition(tp) if leader is None or leader == -1: unknown_leaders_exist = True continue elif leader in ready_nodes: continue elif tp in self.muted: continue with self._tp_locks[tp]: dq = self._batches[tp] if not dq: continue batch = dq[0] retry_backoff = self.config['retry_backoff_ms'] / 1000.0 linger = self.config['linger_ms'] / 1000.0 backing_off = bool(batch.attempts > 0 and batch.last_attempt + retry_backoff > now) waited_time = now - batch.last_attempt time_to_wait = retry_backoff if backing_off else linger time_left = max(time_to_wait - waited_time, 0) full = bool(len(dq) > 1 or batch.records.is_full()) expired = bool(waited_time >= time_to_wait) sendable = (full or expired or exhausted or self._closed or self._flush_in_progress()) if sendable and not backing_off: ready_nodes.add(leader) else: # Note that this results in a conservative estimate since # an un-sendable partition may have a leader that will # later be found to have sendable data. However, this is # good enough since we'll just wake up and then sleep again # for the remaining time. next_ready_check = min(time_left, next_ready_check) return ready_nodes, next_ready_check, unknown_leaders_exist def has_unsent(self): """Return whether there is any unsent record in the accumulator.""" for tp in list(self._batches.keys()): with self._tp_locks[tp]: dq = self._batches[tp] if len(dq): return True return False def drain(self, cluster, nodes, max_size): """ Drain all the data for the given nodes and collate them into a list of batches that will fit within the specified size on a per-node basis. This method attempts to avoid choosing the same topic-node repeatedly. Arguments: cluster (ClusterMetadata): The current cluster metadata nodes (list): list of node_ids to drain max_size (int): maximum number of bytes to drain Returns: dict: {node_id: list of RecordBatch} with total size less than the requested max_size. """ if not nodes: return {} now = time.time() batches = {} for node_id in nodes: size = 0 partitions = list(cluster.partitions_for_broker(node_id)) ready = [] # to make starvation less likely this loop doesn't start at 0 self._drain_index %= len(partitions) start = self._drain_index while True: tp = partitions[self._drain_index] if tp in self._batches and tp not in self.muted: with self._tp_locks[tp]: dq = self._batches[tp] if dq: first = dq[0] backoff = ( bool(first.attempts > 0) and bool(first.last_attempt + self.config['retry_backoff_ms'] / 1000.0 > now) ) # Only drain the batch if it is not during backoff if not backoff: if (size + first.records.size_in_bytes() > max_size and len(ready) > 0): # there is a rare case that a single batch # size is larger than the request size due # to compression; in this case we will # still eventually send this batch in a # single request break else: batch = dq.popleft() batch.records.close() size += batch.records.size_in_bytes() ready.append(batch) batch.drained = now self._drain_index += 1 self._drain_index %= len(partitions) if start == self._drain_index: break batches[node_id] = ready return batches def deallocate(self, batch): """Deallocate the record batch.""" self._incomplete.remove(batch) self._free.deallocate(batch.records.buffer()) def _flush_in_progress(self): """Are there any threads currently waiting on a flush?""" return self._flushes_in_progress.get() > 0 def begin_flush(self): """ Initiate the flushing of data from the accumulator...this makes all requests immediately ready """ self._flushes_in_progress.increment() def await_flush_completion(self, timeout=None): """ Mark all partitions as ready to send and block until the send is complete """ try: for batch in self._incomplete.all(): log.debug('Waiting on produce to %s', batch.produce_future.topic_partition) assert batch.produce_future.wait(timeout=timeout), 'Timeout waiting for future' assert batch.produce_future.is_done, 'Future not done?' if batch.produce_future.failed(): log.warning(batch.produce_future.exception) finally: self._flushes_in_progress.decrement() def abort_incomplete_batches(self): """ This function is only called when sender is closed forcefully. It will fail all the incomplete batches and return. """ # We need to keep aborting the incomplete batch until no thread is trying to append to # 1. Avoid losing batches. # 2. Free up memory in case appending threads are blocked on buffer full. # This is a tight loop but should be able to get through very quickly. while True: self._abort_batches() if not self._appends_in_progress.get(): break # After this point, no thread will append any messages because they will see the close # flag set. We need to do the last abort after no thread was appending in case the there was a new # batch appended by the last appending thread. self._abort_batches() self._batches.clear() def _abort_batches(self): """Go through incomplete batches and abort them.""" error = Errors.IllegalStateError("Producer is closed forcefully.") for batch in self._incomplete.all(): tp = batch.topic_partition # Close the batch before aborting with self._tp_locks[tp]: batch.records.close() batch.done(exception=error) self.deallocate(batch) def close(self): """Close this accumulator and force all the record buffers to be drained.""" self._closed = True class IncompleteRecordBatches(object): """A threadsafe helper class to hold RecordBatches that haven't been ack'd yet""" def __init__(self): self._incomplete = set() self._lock = threading.Lock() def add(self, batch): with self._lock: return self._incomplete.add(batch) def remove(self, batch): with self._lock: return self._incomplete.remove(batch) def all(self): with self._lock: return list(self._incomplete) kafka-1.3.2/kafka/producer/sender.py0000644001271300127130000005205413025302127017113 0ustar dpowers00000000000000from __future__ import absolute_import, division import collections import copy import logging import threading import time from kafka.vendor import six from .. import errors as Errors from ..metrics.measurable import AnonMeasurable from ..metrics.stats import Avg, Count, Max, Rate from ..protocol.produce import ProduceRequest from ..structs import TopicPartition from ..version import __version__ log = logging.getLogger(__name__) class Sender(threading.Thread): """ The background thread that handles the sending of produce requests to the Kafka cluster. This thread makes metadata requests to renew its view of the cluster and then sends produce requests to the appropriate nodes. """ DEFAULT_CONFIG = { 'max_request_size': 1048576, 'acks': 1, 'retries': 0, 'request_timeout_ms': 30000, 'guarantee_message_order': False, 'client_id': 'kafka-python-' + __version__, 'api_version': (0, 8, 0), } def __init__(self, client, metadata, accumulator, metrics, **configs): super(Sender, self).__init__() self.config = copy.copy(self.DEFAULT_CONFIG) for key in self.config: if key in configs: self.config[key] = configs.pop(key) self.name = self.config['client_id'] + '-network-thread' self._client = client self._accumulator = accumulator self._metadata = client.cluster self._running = True self._force_close = False self._topics_to_add = set() self._sensors = SenderMetrics(metrics, self._client, self._metadata) def run(self): """The main run loop for the sender thread.""" log.debug("Starting Kafka producer I/O thread.") # main loop, runs until close is called while self._running: try: self.run_once() except Exception: log.exception("Uncaught error in kafka producer I/O thread") log.debug("Beginning shutdown of Kafka producer I/O thread, sending" " remaining records.") # okay we stopped accepting requests but there may still be # requests in the accumulator or waiting for acknowledgment, # wait until these are completed. while (not self._force_close and (self._accumulator.has_unsent() or self._client.in_flight_request_count() > 0)): try: self.run_once() except Exception: log.exception("Uncaught error in kafka producer I/O thread") if self._force_close: # We need to fail all the incomplete batches and wake up the # threads waiting on the futures. self._accumulator.abort_incomplete_batches() try: self._client.close() except Exception: log.exception("Failed to close network client") log.debug("Shutdown of Kafka producer I/O thread has completed.") def run_once(self): """Run a single iteration of sending.""" while self._topics_to_add: self._client.add_topic(self._topics_to_add.pop()) # get the list of partitions with data ready to send result = self._accumulator.ready(self._metadata) ready_nodes, next_ready_check_delay, unknown_leaders_exist = result # if there are any partitions whose leaders are not known yet, force # metadata update if unknown_leaders_exist: log.debug('Unknown leaders exist, requesting metadata update') self._metadata.request_update() # remove any nodes we aren't ready to send to not_ready_timeout = 999999999 for node in list(ready_nodes): if not self._client.ready(node): log.debug('Node %s not ready; delaying produce of accumulated batch', node) ready_nodes.remove(node) not_ready_timeout = min(not_ready_timeout, self._client.connection_delay(node)) # create produce requests batches_by_node = self._accumulator.drain( self._metadata, ready_nodes, self.config['max_request_size']) if self.config['guarantee_message_order']: # Mute all the partitions drained for batch_list in six.itervalues(batches_by_node): for batch in batch_list: self._accumulator.muted.add(batch.topic_partition) expired_batches = self._accumulator.abort_expired_batches( self.config['request_timeout_ms'], self._metadata) for expired_batch in expired_batches: self._sensors.record_errors(expired_batch.topic_partition.topic, expired_batch.record_count) self._sensors.update_produce_request_metrics(batches_by_node) requests = self._create_produce_requests(batches_by_node) # If we have any nodes that are ready to send + have sendable data, # poll with 0 timeout so this can immediately loop and try sending more # data. Otherwise, the timeout is determined by nodes that have # partitions with data that isn't yet sendable (e.g. lingering, backing # off). Note that this specifically does not include nodes with # sendable data that aren't ready to send since they would cause busy # looping. poll_timeout_ms = min(next_ready_check_delay * 1000, not_ready_timeout) if ready_nodes: log.debug("Nodes with data ready to send: %s", ready_nodes) # trace log.debug("Created %d produce requests: %s", len(requests), requests) # trace poll_timeout_ms = 0 for node_id, request in six.iteritems(requests): batches = batches_by_node[node_id] log.debug('Sending Produce Request: %r', request) (self._client.send(node_id, request) .add_callback( self._handle_produce_response, node_id, time.time(), batches) .add_errback( self._failed_produce, batches, node_id)) # if some partitions are already ready to be sent, the select time # would be 0; otherwise if some partition already has some data # accumulated but not ready yet, the select time will be the time # difference between now and its linger expiry time; otherwise the # select time will be the time difference between now and the # metadata expiry time self._client.poll(poll_timeout_ms, sleep=True) def initiate_close(self): """Start closing the sender (won't complete until all data is sent).""" self._running = False self._accumulator.close() self.wakeup() def force_close(self): """Closes the sender without sending out any pending messages.""" self._force_close = True self.initiate_close() def add_topic(self, topic): # This is generally called from a separate thread # so this needs to be a thread-safe operation # we assume that checking set membership across threads # is ok where self._client._topics should never # remove topics for a producer instance, only add them. if topic not in self._client._topics: self._topics_to_add.add(topic) self.wakeup() def _failed_produce(self, batches, node_id, error): log.debug("Error sending produce request to node %d: %s", node_id, error) # trace for batch in batches: self._complete_batch(batch, error, -1, None) def _handle_produce_response(self, node_id, send_time, batches, response): """Handle a produce response.""" # if we have a response, parse it log.debug('Parsing produce response: %r', response) if response: batches_by_partition = dict([(batch.topic_partition, batch) for batch in batches]) for topic, partitions in response.topics: for partition_info in partitions: if response.API_VERSION < 2: partition, error_code, offset = partition_info ts = None else: partition, error_code, offset, ts = partition_info tp = TopicPartition(topic, partition) error = Errors.for_code(error_code) batch = batches_by_partition[tp] self._complete_batch(batch, error, offset, ts) if response.API_VERSION > 0: self._sensors.record_throttle_time(response.throttle_time_ms, node=node_id) else: # this is the acks = 0 case, just complete all requests for batch in batches: self._complete_batch(batch, None, -1, None) def _complete_batch(self, batch, error, base_offset, timestamp_ms=None): """Complete or retry the given batch of records. Arguments: batch (RecordBatch): The record batch error (Exception): The error (or None if none) base_offset (int): The base offset assigned to the records if successful timestamp_ms (int, optional): The timestamp returned by the broker for this batch """ # Standardize no-error to None if error is Errors.NoError: error = None if error is not None and self._can_retry(batch, error): # retry log.warning("Got error produce response on topic-partition %s," " retrying (%d attempts left). Error: %s", batch.topic_partition, self.config['retries'] - batch.attempts - 1, error) self._accumulator.reenqueue(batch) self._sensors.record_retries(batch.topic_partition.topic, batch.record_count) else: if error is Errors.TopicAuthorizationFailedError: error = error(batch.topic_partition.topic) # tell the user the result of their request batch.done(base_offset, timestamp_ms, error) self._accumulator.deallocate(batch) if error is not None: self._sensors.record_errors(batch.topic_partition.topic, batch.record_count) if getattr(error, 'invalid_metadata', False): self._metadata.request_update() # Unmute the completed partition. if self.config['guarantee_message_order']: self._accumulator.muted.remove(batch.topic_partition) def _can_retry(self, batch, error): """ We can retry a send if the error is transient and the number of attempts taken is fewer than the maximum allowed """ return (batch.attempts < self.config['retries'] and getattr(error, 'retriable', False)) def _create_produce_requests(self, collated): """ Transfer the record batches into a list of produce requests on a per-node basis. Arguments: collated: {node_id: [RecordBatch]} Returns: dict: {node_id: ProduceRequest} (version depends on api_version) """ requests = {} for node_id, batches in six.iteritems(collated): requests[node_id] = self._produce_request( node_id, self.config['acks'], self.config['request_timeout_ms'], batches) return requests def _produce_request(self, node_id, acks, timeout, batches): """Create a produce request from the given record batches. Returns: ProduceRequest (version depends on api_version) """ produce_records_by_partition = collections.defaultdict(dict) for batch in batches: topic = batch.topic_partition.topic partition = batch.topic_partition.partition # TODO: bytearray / memoryview buf = batch.records.buffer() produce_records_by_partition[topic][partition] = buf if self.config['api_version'] >= (0, 10): version = 2 elif self.config['api_version'] == (0, 9): version = 1 else: version = 0 return ProduceRequest[version]( required_acks=acks, timeout=timeout, topics=[(topic, list(partition_info.items())) for topic, partition_info in six.iteritems(produce_records_by_partition)] ) def wakeup(self): """Wake up the selector associated with this send thread.""" self._client.wakeup() class SenderMetrics(object): def __init__(self, metrics, client, metadata): self.metrics = metrics self._client = client self._metadata = metadata sensor_name = 'batch-size' self.batch_size_sensor = self.metrics.sensor(sensor_name) self.add_metric('batch-size-avg', Avg(), sensor_name=sensor_name, description='The average number of bytes sent per partition per-request.') self.add_metric('batch-size-max', Max(), sensor_name=sensor_name, description='The max number of bytes sent per partition per-request.') sensor_name = 'compression-rate' self.compression_rate_sensor = self.metrics.sensor(sensor_name) self.add_metric('compression-rate-avg', Avg(), sensor_name=sensor_name, description='The average compression rate of record batches.') sensor_name = 'queue-time' self.queue_time_sensor = self.metrics.sensor(sensor_name) self.add_metric('record-queue-time-avg', Avg(), sensor_name=sensor_name, description='The average time in ms record batches spent in the record accumulator.') self.add_metric('record-queue-time-max', Max(), sensor_name=sensor_name, description='The maximum time in ms record batches spent in the record accumulator.') sensor_name = 'produce-throttle-time' self.produce_throttle_time_sensor = self.metrics.sensor(sensor_name) self.add_metric('produce-throttle-time-avg', Avg(), sensor_name=sensor_name, description='The average throttle time in ms') self.add_metric('produce-throttle-time-max', Max(), sensor_name=sensor_name, description='The maximum throttle time in ms') sensor_name = 'records-per-request' self.records_per_request_sensor = self.metrics.sensor(sensor_name) self.add_metric('record-send-rate', Rate(), sensor_name=sensor_name, description='The average number of records sent per second.') self.add_metric('records-per-request-avg', Avg(), sensor_name=sensor_name, description='The average number of records per request.') sensor_name = 'bytes' self.byte_rate_sensor = self.metrics.sensor(sensor_name) self.add_metric('byte-rate', Rate(), sensor_name=sensor_name, description='The average number of bytes sent per second.') sensor_name = 'record-retries' self.retry_sensor = self.metrics.sensor(sensor_name) self.add_metric('record-retry-rate', Rate(), sensor_name=sensor_name, description='The average per-second number of retried record sends') sensor_name = 'errors' self.error_sensor = self.metrics.sensor(sensor_name) self.add_metric('record-error-rate', Rate(), sensor_name=sensor_name, description='The average per-second number of record sends that resulted in errors') sensor_name = 'record-size-max' self.max_record_size_sensor = self.metrics.sensor(sensor_name) self.add_metric('record-size-max', Max(), sensor_name=sensor_name, description='The maximum record size across all batches') self.add_metric('record-size-avg', Avg(), sensor_name=sensor_name, description='The average maximum record size per batch') self.add_metric('requests-in-flight', AnonMeasurable(lambda *_: self._client.in_flight_request_count()), description='The current number of in-flight requests awaiting a response.') self.add_metric('metadata-age', AnonMeasurable(lambda _, now: (now - self._metadata._last_successful_refresh_ms) / 1000), description='The age in seconds of the current producer metadata being used.') def add_metric(self, metric_name, measurable, group_name='producer-metrics', description=None, tags=None, sensor_name=None): m = self.metrics metric = m.metric_name(metric_name, group_name, description, tags) if sensor_name: sensor = m.sensor(sensor_name) sensor.add(metric, measurable) else: m.add_metric(metric, measurable) def maybe_register_topic_metrics(self, topic): def sensor_name(name): return 'topic.{0}.{1}'.format(topic, name) # if one sensor of the metrics has been registered for the topic, # then all other sensors should have been registered; and vice versa if not self.metrics.get_sensor(sensor_name('records-per-batch')): self.add_metric('record-send-rate', Rate(), sensor_name=sensor_name('records-per-batch'), group_name='producer-topic-metrics.' + topic, description= 'Records sent per second for topic ' + topic) self.add_metric('byte-rate', Rate(), sensor_name=sensor_name('bytes'), group_name='producer-topic-metrics.' + topic, description='Bytes per second for topic ' + topic) self.add_metric('compression-rate', Avg(), sensor_name=sensor_name('compression-rate'), group_name='producer-topic-metrics.' + topic, description='Average Compression ratio for topic ' + topic) self.add_metric('record-retry-rate', Rate(), sensor_name=sensor_name('record-retries'), group_name='producer-topic-metrics.' + topic, description='Record retries per second for topic ' + topic) self.add_metric('record-error-rate', Rate(), sensor_name=sensor_name('record-errors'), group_name='producer-topic-metrics.' + topic, description='Record errors per second for topic ' + topic) def update_produce_request_metrics(self, batches_map): for node_batch in batches_map.values(): records = 0 total_bytes = 0 for batch in node_batch: # register all per-topic metrics at once topic = batch.topic_partition.topic self.maybe_register_topic_metrics(topic) # per-topic record send rate topic_records_count = self.metrics.get_sensor( 'topic.' + topic + '.records-per-batch') topic_records_count.record(batch.record_count) # per-topic bytes send rate topic_byte_rate = self.metrics.get_sensor( 'topic.' + topic + '.bytes') topic_byte_rate.record(batch.records.size_in_bytes()) # per-topic compression rate topic_compression_rate = self.metrics.get_sensor( 'topic.' + topic + '.compression-rate') topic_compression_rate.record(batch.records.compression_rate()) # global metrics self.batch_size_sensor.record(batch.records.size_in_bytes()) if batch.drained: self.queue_time_sensor.record(batch.drained - batch.created) self.compression_rate_sensor.record(batch.records.compression_rate()) self.max_record_size_sensor.record(batch.max_record_size) records += batch.record_count total_bytes += batch.records.size_in_bytes() self.records_per_request_sensor.record(records) self.byte_rate_sensor.record(total_bytes) def record_retries(self, topic, count): self.retry_sensor.record(count) sensor = self.metrics.get_sensor('topic.' + topic + '.record-retries') if sensor: sensor.record(count) def record_errors(self, topic, count): self.error_sensor.record(count) sensor = self.metrics.get_sensor('topic.' + topic + '.record-errors') if sensor: sensor.record(count) def record_throttle_time(self, throttle_time_ms, node=None): self.produce_throttle_time_sensor.record(throttle_time_ms) kafka-1.3.2/kafka/producer/simple.py0000644001271300127130000000344313025302127017122 0ustar dpowers00000000000000from __future__ import absolute_import from itertools import cycle import logging import random from kafka.vendor.six.moves import xrange # pylint: disable=import-error from .base import Producer log = logging.getLogger(__name__) class SimpleProducer(Producer): """A simple, round-robin producer. See Producer class for Base Arguments Additional Arguments: random_start (bool, optional): randomize the initial partition which the first message block will be published to, otherwise if false, the first message block will always publish to partition 0 before cycling through each partition, defaults to True. """ def __init__(self, *args, **kwargs): self.partition_cycles = {} self.random_start = kwargs.pop('random_start', True) super(SimpleProducer, self).__init__(*args, **kwargs) def _next_partition(self, topic): if topic not in self.partition_cycles: if not self.client.has_metadata_for_topic(topic): self.client.ensure_topic_exists(topic) self.partition_cycles[topic] = cycle(self.client.get_partition_ids_for_topic(topic)) # Randomize the initial partition that is returned if self.random_start: num_partitions = len(self.client.get_partition_ids_for_topic(topic)) for _ in xrange(random.randint(0, num_partitions-1)): next(self.partition_cycles[topic]) return next(self.partition_cycles[topic]) def send_messages(self, topic, *msg): partition = self._next_partition(topic) return super(SimpleProducer, self).send_messages( topic, partition, *msg ) def __repr__(self): return '' % self.async kafka-1.3.2/kafka/protocol/0000755001271300127130000000000013031057517015300 5ustar dpowers00000000000000kafka-1.3.2/kafka/protocol/__init__.py0000644001271300127130000000036713025302127017410 0ustar dpowers00000000000000from __future__ import absolute_import from .legacy import ( create_message, create_gzip_message, create_snappy_message, create_message_set, CODEC_NONE, CODEC_GZIP, CODEC_SNAPPY, ALL_CODECS, ATTRIBUTE_CODEC_MASK, KafkaProtocol, ) kafka-1.3.2/kafka/protocol/abstract.py0000644001271300127130000000060113025302127017443 0ustar dpowers00000000000000from __future__ import absolute_import import abc class AbstractType(object): __metaclass__ = abc.ABCMeta @abc.abstractmethod def encode(cls, value): # pylint: disable=no-self-argument pass @abc.abstractmethod def decode(cls, data): # pylint: disable=no-self-argument pass @classmethod def repr(cls, value): return repr(value) kafka-1.3.2/kafka/protocol/admin.py0000644001271300127130000000466713025302127016750 0ustar dpowers00000000000000from __future__ import absolute_import from .struct import Struct from .types import Array, Bytes, Int16, Schema, String class ApiVersionResponse_v0(Struct): API_KEY = 18 API_VERSION = 0 SCHEMA = Schema( ('error_code', Int16), ('api_versions', Array( ('api_key', Int16), ('min_version', Int16), ('max_version', Int16)))) class ApiVersionRequest_v0(Struct): API_KEY = 18 API_VERSION = 0 RESPONSE_TYPE = ApiVersionResponse_v0 SCHEMA = Schema() ApiVersionRequest = [ApiVersionRequest_v0] ApiVersionResponse = [ApiVersionResponse_v0] class ListGroupsResponse_v0(Struct): API_KEY = 16 API_VERSION = 0 SCHEMA = Schema( ('error_code', Int16), ('groups', Array( ('group', String('utf-8')), ('protocol_type', String('utf-8')))) ) class ListGroupsRequest_v0(Struct): API_KEY = 16 API_VERSION = 0 RESPONSE_TYPE = ListGroupsResponse_v0 SCHEMA = Schema() ListGroupsRequest = [ListGroupsRequest_v0] ListGroupsResponse = [ListGroupsResponse_v0] class DescribeGroupsResponse_v0(Struct): API_KEY = 15 API_VERSION = 0 SCHEMA = Schema( ('groups', Array( ('error_code', Int16), ('group', String('utf-8')), ('state', String('utf-8')), ('protocol_type', String('utf-8')), ('protocol', String('utf-8')), ('members', Array( ('member_id', String('utf-8')), ('client_id', String('utf-8')), ('client_host', String('utf-8')), ('member_metadata', Bytes), ('member_assignment', Bytes))))) ) class DescribeGroupsRequest_v0(Struct): API_KEY = 15 API_VERSION = 0 RESPONSE_TYPE = DescribeGroupsResponse_v0 SCHEMA = Schema( ('groups', Array(String('utf-8'))) ) DescribeGroupsRequest = [DescribeGroupsRequest_v0] DescribeGroupsResponse = [DescribeGroupsResponse_v0] class SaslHandShakeResponse_v0(Struct): API_KEY = 17 API_VERSION = 0 SCHEMA = Schema( ('error_code', Int16), ('enabled_mechanisms', Array(String('utf-8'))) ) class SaslHandShakeRequest_v0(Struct): API_KEY = 17 API_VERSION = 0 RESPONSE_TYPE = SaslHandShakeResponse_v0 SCHEMA = Schema( ('mechanism', String('utf-8')) ) SaslHandShakeRequest = [SaslHandShakeRequest_v0] SaslHandShakeResponse = [SaslHandShakeResponse_v0] kafka-1.3.2/kafka/protocol/api.py0000644001271300127130000000100413025302127016407 0ustar dpowers00000000000000from __future__ import absolute_import from .struct import Struct from .types import Int16, Int32, String, Schema class RequestHeader(Struct): SCHEMA = Schema( ('api_key', Int16), ('api_version', Int16), ('correlation_id', Int32), ('client_id', String('utf-8')) ) def __init__(self, request, correlation_id=0, client_id='kafka-python'): super(RequestHeader, self).__init__( request.API_KEY, request.API_VERSION, correlation_id, client_id ) kafka-1.3.2/kafka/protocol/commit.py0000644001271300127130000001152213025302127017134 0ustar dpowers00000000000000from __future__ import absolute_import from .struct import Struct from .types import Array, Int16, Int32, Int64, Schema, String class OffsetCommitResponse_v0(Struct): API_KEY = 8 API_VERSION = 0 SCHEMA = Schema( ('topics', Array( ('topic', String('utf-8')), ('partitions', Array( ('partition', Int32), ('error_code', Int16))))) ) class OffsetCommitResponse_v1(Struct): API_KEY = 8 API_VERSION = 1 SCHEMA = Schema( ('topics', Array( ('topic', String('utf-8')), ('partitions', Array( ('partition', Int32), ('error_code', Int16))))) ) class OffsetCommitResponse_v2(Struct): API_KEY = 8 API_VERSION = 2 SCHEMA = Schema( ('topics', Array( ('topic', String('utf-8')), ('partitions', Array( ('partition', Int32), ('error_code', Int16))))) ) class OffsetCommitRequest_v0(Struct): API_KEY = 8 API_VERSION = 0 # Zookeeper-backed storage RESPONSE_TYPE = OffsetCommitResponse_v0 SCHEMA = Schema( ('consumer_group', String('utf-8')), ('topics', Array( ('topic', String('utf-8')), ('partitions', Array( ('partition', Int32), ('offset', Int64), ('metadata', String('utf-8')))))) ) class OffsetCommitRequest_v1(Struct): API_KEY = 8 API_VERSION = 1 # Kafka-backed storage RESPONSE_TYPE = OffsetCommitResponse_v1 SCHEMA = Schema( ('consumer_group', String('utf-8')), ('consumer_group_generation_id', Int32), ('consumer_id', String('utf-8')), ('topics', Array( ('topic', String('utf-8')), ('partitions', Array( ('partition', Int32), ('offset', Int64), ('timestamp', Int64), ('metadata', String('utf-8')))))) ) class OffsetCommitRequest_v2(Struct): API_KEY = 8 API_VERSION = 2 # added retention_time, dropped timestamp RESPONSE_TYPE = OffsetCommitResponse_v2 SCHEMA = Schema( ('consumer_group', String('utf-8')), ('consumer_group_generation_id', Int32), ('consumer_id', String('utf-8')), ('retention_time', Int64), ('topics', Array( ('topic', String('utf-8')), ('partitions', Array( ('partition', Int32), ('offset', Int64), ('metadata', String('utf-8')))))) ) DEFAULT_GENERATION_ID = -1 DEFAULT_RETENTION_TIME = -1 OffsetCommitRequest = [OffsetCommitRequest_v0, OffsetCommitRequest_v1, OffsetCommitRequest_v2] OffsetCommitResponse = [OffsetCommitResponse_v0, OffsetCommitResponse_v1, OffsetCommitResponse_v2] class OffsetFetchResponse_v0(Struct): API_KEY = 9 API_VERSION = 0 SCHEMA = Schema( ('topics', Array( ('topic', String('utf-8')), ('partitions', Array( ('partition', Int32), ('offset', Int64), ('metadata', String('utf-8')), ('error_code', Int16))))) ) class OffsetFetchResponse_v1(Struct): API_KEY = 9 API_VERSION = 1 SCHEMA = Schema( ('topics', Array( ('topic', String('utf-8')), ('partitions', Array( ('partition', Int32), ('offset', Int64), ('metadata', String('utf-8')), ('error_code', Int16))))) ) class OffsetFetchRequest_v0(Struct): API_KEY = 9 API_VERSION = 0 # zookeeper-backed storage RESPONSE_TYPE = OffsetFetchResponse_v0 SCHEMA = Schema( ('consumer_group', String('utf-8')), ('topics', Array( ('topic', String('utf-8')), ('partitions', Array(Int32)))) ) class OffsetFetchRequest_v1(Struct): API_KEY = 9 API_VERSION = 1 # kafka-backed storage RESPONSE_TYPE = OffsetFetchResponse_v1 SCHEMA = Schema( ('consumer_group', String('utf-8')), ('topics', Array( ('topic', String('utf-8')), ('partitions', Array(Int32)))) ) OffsetFetchRequest = [OffsetFetchRequest_v0, OffsetFetchRequest_v1] OffsetFetchResponse = [OffsetFetchResponse_v0, OffsetFetchResponse_v1] class GroupCoordinatorResponse_v0(Struct): API_KEY = 10 API_VERSION = 0 SCHEMA = Schema( ('error_code', Int16), ('coordinator_id', Int32), ('host', String('utf-8')), ('port', Int32) ) class GroupCoordinatorRequest_v0(Struct): API_KEY = 10 API_VERSION = 0 RESPONSE_TYPE = GroupCoordinatorResponse_v0 SCHEMA = Schema( ('consumer_group', String('utf-8')) ) GroupCoordinatorRequest = [GroupCoordinatorRequest_v0] GroupCoordinatorResponse = [GroupCoordinatorResponse_v0] kafka-1.3.2/kafka/protocol/fetch.py0000644001271300127130000000362713025302127016744 0ustar dpowers00000000000000from __future__ import absolute_import from .message import MessageSet from .struct import Struct from .types import Array, Int16, Int32, Int64, Schema, String class FetchResponse_v0(Struct): API_KEY = 1 API_VERSION = 0 SCHEMA = Schema( ('topics', Array( ('topics', String('utf-8')), ('partitions', Array( ('partition', Int32), ('error_code', Int16), ('highwater_offset', Int64), ('message_set', MessageSet))))) ) class FetchResponse_v1(Struct): API_KEY = 1 API_VERSION = 1 SCHEMA = Schema( ('throttle_time_ms', Int32), ('topics', Array( ('topics', String('utf-8')), ('partitions', Array( ('partition', Int32), ('error_code', Int16), ('highwater_offset', Int64), ('message_set', MessageSet))))) ) class FetchResponse_v2(Struct): API_KEY = 1 API_VERSION = 2 SCHEMA = FetchResponse_v1.SCHEMA # message format changed internally class FetchRequest_v0(Struct): API_KEY = 1 API_VERSION = 0 RESPONSE_TYPE = FetchResponse_v0 SCHEMA = Schema( ('replica_id', Int32), ('max_wait_time', Int32), ('min_bytes', Int32), ('topics', Array( ('topic', String('utf-8')), ('partitions', Array( ('partition', Int32), ('offset', Int64), ('max_bytes', Int32))))) ) class FetchRequest_v1(Struct): API_KEY = 1 API_VERSION = 1 RESPONSE_TYPE = FetchResponse_v1 SCHEMA = FetchRequest_v0.SCHEMA class FetchRequest_v2(Struct): API_KEY = 1 API_VERSION = 2 RESPONSE_TYPE = FetchResponse_v2 SCHEMA = FetchRequest_v1.SCHEMA FetchRequest = [FetchRequest_v0, FetchRequest_v1, FetchRequest_v2] FetchResponse = [FetchResponse_v0, FetchResponse_v1, FetchResponse_v2] kafka-1.3.2/kafka/protocol/group.py0000644001271300127130000000572013025302127017003 0ustar dpowers00000000000000from __future__ import absolute_import from .struct import Struct from .types import Array, Bytes, Int16, Int32, Schema, String class JoinGroupResponse_v0(Struct): API_KEY = 11 API_VERSION = 0 SCHEMA = Schema( ('error_code', Int16), ('generation_id', Int32), ('group_protocol', String('utf-8')), ('leader_id', String('utf-8')), ('member_id', String('utf-8')), ('members', Array( ('member_id', String('utf-8')), ('member_metadata', Bytes))) ) class JoinGroupRequest_v0(Struct): API_KEY = 11 API_VERSION = 0 RESPONSE_TYPE = JoinGroupResponse_v0 SCHEMA = Schema( ('group', String('utf-8')), ('session_timeout', Int32), ('member_id', String('utf-8')), ('protocol_type', String('utf-8')), ('group_protocols', Array( ('protocol_name', String('utf-8')), ('protocol_metadata', Bytes))) ) UNKNOWN_MEMBER_ID = '' JoinGroupRequest = [JoinGroupRequest_v0] JoinGroupResponse = [JoinGroupResponse_v0] class ProtocolMetadata(Struct): SCHEMA = Schema( ('version', Int16), ('subscription', Array(String('utf-8'))), # topics list ('user_data', Bytes) ) class SyncGroupResponse_v0(Struct): API_KEY = 14 API_VERSION = 0 SCHEMA = Schema( ('error_code', Int16), ('member_assignment', Bytes) ) class SyncGroupRequest_v0(Struct): API_KEY = 14 API_VERSION = 0 RESPONSE_TYPE = SyncGroupResponse_v0 SCHEMA = Schema( ('group', String('utf-8')), ('generation_id', Int32), ('member_id', String('utf-8')), ('group_assignment', Array( ('member_id', String('utf-8')), ('member_metadata', Bytes))) ) SyncGroupRequest = [SyncGroupRequest_v0] SyncGroupResponse = [SyncGroupResponse_v0] class MemberAssignment(Struct): SCHEMA = Schema( ('version', Int16), ('assignment', Array( ('topic', String('utf-8')), ('partitions', Array(Int32)))), ('user_data', Bytes) ) class HeartbeatResponse_v0(Struct): API_KEY = 12 API_VERSION = 0 SCHEMA = Schema( ('error_code', Int16) ) class HeartbeatRequest_v0(Struct): API_KEY = 12 API_VERSION = 0 RESPONSE_TYPE = HeartbeatResponse_v0 SCHEMA = Schema( ('group', String('utf-8')), ('generation_id', Int32), ('member_id', String('utf-8')) ) HeartbeatRequest = [HeartbeatRequest_v0] HeartbeatResponse = [HeartbeatResponse_v0] class LeaveGroupResponse_v0(Struct): API_KEY = 13 API_VERSION = 0 SCHEMA = Schema( ('error_code', Int16) ) class LeaveGroupRequest_v0(Struct): API_KEY = 13 API_VERSION = 0 RESPONSE_TYPE = LeaveGroupResponse_v0 SCHEMA = Schema( ('group', String('utf-8')), ('member_id', String('utf-8')) ) LeaveGroupRequest = [LeaveGroupRequest_v0] LeaveGroupResponse = [LeaveGroupResponse_v0] kafka-1.3.2/kafka/protocol/legacy.py0000644001271300127130000003505013025302127017112 0ustar dpowers00000000000000from __future__ import absolute_import import logging import struct from kafka.vendor import six # pylint: disable=import-error from kafka.vendor.six.moves import xrange # pylint: disable=import-error import kafka.protocol.commit import kafka.protocol.fetch import kafka.protocol.message import kafka.protocol.metadata import kafka.protocol.offset import kafka.protocol.produce import kafka.structs from kafka.codec import ( gzip_encode, gzip_decode, snappy_encode, snappy_decode) from kafka.errors import ProtocolError, ChecksumError, UnsupportedCodecError from kafka.structs import ConsumerMetadataResponse from kafka.util import ( crc32, read_short_string, read_int_string, relative_unpack, write_short_string, write_int_string, group_by_topic_and_partition) log = logging.getLogger(__name__) ATTRIBUTE_CODEC_MASK = 0x03 CODEC_NONE = 0x00 CODEC_GZIP = 0x01 CODEC_SNAPPY = 0x02 ALL_CODECS = (CODEC_NONE, CODEC_GZIP, CODEC_SNAPPY) class KafkaProtocol(object): """ Class to encapsulate all of the protocol encoding/decoding. This class does not have any state associated with it, it is purely for organization. """ PRODUCE_KEY = 0 FETCH_KEY = 1 OFFSET_KEY = 2 METADATA_KEY = 3 OFFSET_COMMIT_KEY = 8 OFFSET_FETCH_KEY = 9 CONSUMER_METADATA_KEY = 10 ################### # Private API # ################### @classmethod def _encode_message_header(cls, client_id, correlation_id, request_key, version=0): """ Encode the common request envelope """ return struct.pack('>hhih%ds' % len(client_id), request_key, # ApiKey version, # ApiVersion correlation_id, # CorrelationId len(client_id), # ClientId size client_id) # ClientId @classmethod def _encode_message_set(cls, messages): """ Encode a MessageSet. Unlike other arrays in the protocol, MessageSets are not length-prefixed Format ====== MessageSet => [Offset MessageSize Message] Offset => int64 MessageSize => int32 """ message_set = [] for message in messages: encoded_message = KafkaProtocol._encode_message(message) message_set.append(struct.pack('>qi%ds' % len(encoded_message), 0, len(encoded_message), encoded_message)) return b''.join(message_set) @classmethod def _encode_message(cls, message): """ Encode a single message. The magic number of a message is a format version number. The only supported magic number right now is zero Format ====== Message => Crc MagicByte Attributes Key Value Crc => int32 MagicByte => int8 Attributes => int8 Key => bytes Value => bytes """ if message.magic == 0: msg = b''.join([ struct.pack('>BB', message.magic, message.attributes), write_int_string(message.key), write_int_string(message.value) ]) crc = crc32(msg) msg = struct.pack('>i%ds' % len(msg), crc, msg) else: raise ProtocolError("Unexpected magic number: %d" % message.magic) return msg ################## # Public API # ################## @classmethod def encode_produce_request(cls, payloads=(), acks=1, timeout=1000): """ Encode a ProduceRequest struct Arguments: payloads: list of ProduceRequestPayload acks: How "acky" you want the request to be 1: written to disk by the leader 0: immediate response -1: waits for all replicas to be in sync timeout: Maximum time (in ms) the server will wait for replica acks. This is _not_ a socket timeout Returns: ProduceRequest """ if acks not in (1, 0, -1): raise ValueError('ProduceRequest acks (%s) must be 1, 0, -1' % acks) return kafka.protocol.produce.ProduceRequest[0]( required_acks=acks, timeout=timeout, topics=[( topic, [( partition, [(0, kafka.protocol.message.Message( msg.value, key=msg.key, magic=msg.magic, attributes=msg.attributes ).encode()) for msg in payload.messages]) for partition, payload in topic_payloads.items()]) for topic, topic_payloads in group_by_topic_and_partition(payloads).items()]) @classmethod def decode_produce_response(cls, response): """ Decode ProduceResponse to ProduceResponsePayload Arguments: response: ProduceResponse Return: list of ProduceResponsePayload """ return [ kafka.structs.ProduceResponsePayload(topic, partition, error, offset) for topic, partitions in response.topics for partition, error, offset in partitions ] @classmethod def encode_fetch_request(cls, payloads=(), max_wait_time=100, min_bytes=4096): """ Encodes a FetchRequest struct Arguments: payloads: list of FetchRequestPayload max_wait_time (int, optional): ms to block waiting for min_bytes data. Defaults to 100. min_bytes (int, optional): minimum bytes required to return before max_wait_time. Defaults to 4096. Return: FetchRequest """ return kafka.protocol.fetch.FetchRequest[0]( replica_id=-1, max_wait_time=max_wait_time, min_bytes=min_bytes, topics=[( topic, [( partition, payload.offset, payload.max_bytes) for partition, payload in topic_payloads.items()]) for topic, topic_payloads in group_by_topic_and_partition(payloads).items()]) @classmethod def decode_fetch_response(cls, response): """ Decode FetchResponse struct to FetchResponsePayloads Arguments: response: FetchResponse """ return [ kafka.structs.FetchResponsePayload( topic, partition, error, highwater_offset, [ offset_and_msg for offset_and_msg in cls.decode_message_set(messages)]) for topic, partitions in response.topics for partition, error, highwater_offset, messages in partitions ] @classmethod def decode_message_set(cls, messages): for offset, _, message in messages: if isinstance(message, kafka.protocol.message.Message) and message.is_compressed(): inner_messages = message.decompress() for (inner_offset, _msg_size, inner_msg) in inner_messages: yield kafka.structs.OffsetAndMessage(inner_offset, inner_msg) else: yield kafka.structs.OffsetAndMessage(offset, message) @classmethod def encode_offset_request(cls, payloads=()): return kafka.protocol.offset.OffsetRequest[0]( replica_id=-1, topics=[( topic, [( partition, payload.time, payload.max_offsets) for partition, payload in six.iteritems(topic_payloads)]) for topic, topic_payloads in six.iteritems(group_by_topic_and_partition(payloads))]) @classmethod def decode_offset_response(cls, response): """ Decode OffsetResponse into OffsetResponsePayloads Arguments: response: OffsetResponse Returns: list of OffsetResponsePayloads """ return [ kafka.structs.OffsetResponsePayload(topic, partition, error, tuple(offsets)) for topic, partitions in response.topics for partition, error, offsets in partitions ] @classmethod def encode_metadata_request(cls, topics=(), payloads=None): """ Encode a MetadataRequest Arguments: topics: list of strings """ if payloads is not None: topics = payloads return kafka.protocol.metadata.MetadataRequest[0](topics) @classmethod def decode_metadata_response(cls, response): return response @classmethod def encode_consumer_metadata_request(cls, client_id, correlation_id, payloads): """ Encode a ConsumerMetadataRequest Arguments: client_id: string correlation_id: int payloads: string (consumer group) """ message = [] message.append(cls._encode_message_header(client_id, correlation_id, KafkaProtocol.CONSUMER_METADATA_KEY)) message.append(struct.pack('>h%ds' % len(payloads), len(payloads), payloads)) msg = b''.join(message) return write_int_string(msg) @classmethod def decode_consumer_metadata_response(cls, data): """ Decode bytes to a ConsumerMetadataResponse Arguments: data: bytes to decode """ ((correlation_id, error, nodeId), cur) = relative_unpack('>ihi', data, 0) (host, cur) = read_short_string(data, cur) ((port,), cur) = relative_unpack('>i', data, cur) return ConsumerMetadataResponse(error, nodeId, host, port) @classmethod def encode_offset_commit_request(cls, group, payloads): """ Encode an OffsetCommitRequest struct Arguments: group: string, the consumer group you are committing offsets for payloads: list of OffsetCommitRequestPayload """ return kafka.protocol.commit.OffsetCommitRequest[0]( consumer_group=group, topics=[( topic, [( partition, payload.offset, payload.metadata) for partition, payload in six.iteritems(topic_payloads)]) for topic, topic_payloads in six.iteritems(group_by_topic_and_partition(payloads))]) @classmethod def decode_offset_commit_response(cls, response): """ Decode OffsetCommitResponse to an OffsetCommitResponsePayload Arguments: response: OffsetCommitResponse """ return [ kafka.structs.OffsetCommitResponsePayload(topic, partition, error) for topic, partitions in response.topics for partition, error in partitions ] @classmethod def encode_offset_fetch_request(cls, group, payloads, from_kafka=False): """ Encode an OffsetFetchRequest struct. The request is encoded using version 0 if from_kafka is false, indicating a request for Zookeeper offsets. It is encoded using version 1 otherwise, indicating a request for Kafka offsets. Arguments: group: string, the consumer group you are fetching offsets for payloads: list of OffsetFetchRequestPayload from_kafka: bool, default False, set True for Kafka-committed offsets """ version = 1 if from_kafka else 0 return kafka.protocol.commit.OffsetFetchRequest[version]( consumer_group=group, topics=[( topic, list(topic_payloads.keys())) for topic, topic_payloads in six.iteritems(group_by_topic_and_partition(payloads))]) @classmethod def decode_offset_fetch_response(cls, response): """ Decode OffsetFetchResponse to OffsetFetchResponsePayloads Arguments: response: OffsetFetchResponse """ return [ kafka.structs.OffsetFetchResponsePayload( topic, partition, offset, metadata, error ) for topic, partitions in response.topics for partition, offset, metadata, error in partitions ] def create_message(payload, key=None): """ Construct a Message Arguments: payload: bytes, the payload to send to Kafka key: bytes, a key used for partition routing (optional) """ return kafka.structs.Message(0, 0, key, payload) def create_gzip_message(payloads, key=None, compresslevel=None): """ Construct a Gzipped Message containing multiple Messages The given payloads will be encoded, compressed, and sent as a single atomic message to Kafka. Arguments: payloads: list(bytes), a list of payload to send be sent to Kafka key: bytes, a key used for partition routing (optional) """ message_set = KafkaProtocol._encode_message_set( [create_message(payload, pl_key) for payload, pl_key in payloads]) gzipped = gzip_encode(message_set, compresslevel=compresslevel) codec = ATTRIBUTE_CODEC_MASK & CODEC_GZIP return kafka.structs.Message(0, 0x00 | codec, key, gzipped) def create_snappy_message(payloads, key=None): """ Construct a Snappy Message containing multiple Messages The given payloads will be encoded, compressed, and sent as a single atomic message to Kafka. Arguments: payloads: list(bytes), a list of payload to send be sent to Kafka key: bytes, a key used for partition routing (optional) """ message_set = KafkaProtocol._encode_message_set( [create_message(payload, pl_key) for payload, pl_key in payloads]) snapped = snappy_encode(message_set) codec = ATTRIBUTE_CODEC_MASK & CODEC_SNAPPY return kafka.structs.Message(0, 0x00 | codec, key, snapped) def create_message_set(messages, codec=CODEC_NONE, key=None, compresslevel=None): """Create a message set using the given codec. If codec is CODEC_NONE, return a list of raw Kafka messages. Otherwise, return a list containing a single codec-encoded message. """ if codec == CODEC_NONE: return [create_message(m, k) for m, k in messages] elif codec == CODEC_GZIP: return [create_gzip_message(messages, key, compresslevel)] elif codec == CODEC_SNAPPY: return [create_snappy_message(messages, key)] else: raise UnsupportedCodecError("Codec 0x%02x unsupported" % codec) kafka-1.3.2/kafka/protocol/message.py0000644001271300127130000001540213031057471017277 0ustar dpowers00000000000000from __future__ import absolute_import import io import time from ..codec import (has_gzip, has_snappy, has_lz4, gzip_decode, snappy_decode, lz4_decode, lz4_decode_old_kafka) from . import pickle from .struct import Struct from .types import ( Int8, Int32, Int64, Bytes, Schema, AbstractType ) from ..util import crc32 class Message(Struct): SCHEMAS = [ Schema( ('crc', Int32), ('magic', Int8), ('attributes', Int8), ('key', Bytes), ('value', Bytes)), Schema( ('crc', Int32), ('magic', Int8), ('attributes', Int8), ('timestamp', Int64), ('key', Bytes), ('value', Bytes)), ] SCHEMA = SCHEMAS[1] CODEC_MASK = 0x07 CODEC_GZIP = 0x01 CODEC_SNAPPY = 0x02 CODEC_LZ4 = 0x03 TIMESTAMP_TYPE_MASK = 0x08 HEADER_SIZE = 22 # crc(4), magic(1), attributes(1), timestamp(8), key+value size(4*2) def __init__(self, value, key=None, magic=0, attributes=0, crc=0, timestamp=None): assert value is None or isinstance(value, bytes), 'value must be bytes' assert key is None or isinstance(key, bytes), 'key must be bytes' assert magic > 0 or timestamp is None, 'timestamp not supported in v0' # Default timestamp to now for v1 messages if magic > 0 and timestamp is None: timestamp = int(time.time() * 1000) self.timestamp = timestamp self.crc = crc self.magic = magic self.attributes = attributes self.key = key self.value = value self.encode = self._encode_self @property def timestamp_type(self): """0 for CreateTime; 1 for LogAppendTime; None if unsupported. Value is determined by broker; produced messages should always set to 0 Requires Kafka >= 0.10 / message version >= 1 """ if self.magic == 0: return None elif self.attributes & self.TIMESTAMP_TYPE_MASK: return 1 else: return 0 def _encode_self(self, recalc_crc=True): version = self.magic if version == 1: fields = (self.crc, self.magic, self.attributes, self.timestamp, self.key, self.value) elif version == 0: fields = (self.crc, self.magic, self.attributes, self.key, self.value) else: raise ValueError('Unrecognized message version: %s' % version) message = Message.SCHEMAS[version].encode(fields) if not recalc_crc: return message self.crc = crc32(message[4:]) crc_field = self.SCHEMAS[version].fields[0] return crc_field.encode(self.crc) + message[4:] @classmethod def decode(cls, data): if isinstance(data, bytes): data = io.BytesIO(data) # Partial decode required to determine message version base_fields = cls.SCHEMAS[0].fields[0:3] crc, magic, attributes = [field.decode(data) for field in base_fields] remaining = cls.SCHEMAS[magic].fields[3:] fields = [field.decode(data) for field in remaining] if magic == 1: timestamp = fields[0] else: timestamp = None return cls(fields[-1], key=fields[-2], magic=magic, attributes=attributes, crc=crc, timestamp=timestamp) def validate_crc(self): raw_msg = self._encode_self(recalc_crc=False) crc = crc32(raw_msg[4:]) if crc == self.crc: return True return False def is_compressed(self): return self.attributes & self.CODEC_MASK != 0 def decompress(self): codec = self.attributes & self.CODEC_MASK assert codec in (self.CODEC_GZIP, self.CODEC_SNAPPY, self.CODEC_LZ4) if codec == self.CODEC_GZIP: assert has_gzip(), 'Gzip decompression unsupported' raw_bytes = gzip_decode(self.value) elif codec == self.CODEC_SNAPPY: assert has_snappy(), 'Snappy decompression unsupported' raw_bytes = snappy_decode(self.value) elif codec == self.CODEC_LZ4: assert has_lz4(), 'LZ4 decompression unsupported' if self.magic == 0: raw_bytes = lz4_decode_old_kafka(self.value) else: raw_bytes = lz4_decode(self.value) else: raise Exception('This should be impossible') return MessageSet.decode(raw_bytes, bytes_to_read=len(raw_bytes)) def __hash__(self): return hash(self._encode_self(recalc_crc=False)) class PartialMessage(bytes): def __repr__(self): return 'PartialMessage(%s)' % self class MessageSet(AbstractType): ITEM = Schema( ('offset', Int64), ('message', Bytes) ) HEADER_SIZE = 12 # offset + message_size @classmethod def encode(cls, items): # RecordAccumulator encodes messagesets internally if isinstance(items, io.BytesIO): size = Int32.decode(items) # rewind and return all the bytes items.seek(-4, 1) return items.read(size + 4) encoded_values = [] for (offset, message) in items: encoded_values.append(Int64.encode(offset)) encoded_values.append(Bytes.encode(message)) encoded = b''.join(encoded_values) return Bytes.encode(encoded) @classmethod def decode(cls, data, bytes_to_read=None): """Compressed messages should pass in bytes_to_read (via message size) otherwise, we decode from data as Int32 """ if isinstance(data, bytes): data = io.BytesIO(data) if bytes_to_read is None: bytes_to_read = Int32.decode(data) # if FetchRequest max_bytes is smaller than the available message set # the server returns partial data for the final message # So create an internal buffer to avoid over-reading raw = io.BytesIO(data.read(bytes_to_read)) items = [] while bytes_to_read: try: offset = Int64.decode(raw) msg_bytes = Bytes.decode(raw) bytes_to_read -= 8 + 4 + len(msg_bytes) items.append((offset, len(msg_bytes), Message.decode(msg_bytes))) except ValueError: # PartialMessage to signal that max_bytes may be too small items.append((None, None, PartialMessage())) break return items @classmethod def repr(cls, messages): if isinstance(messages, io.BytesIO): offset = messages.tell() decoded = cls.decode(messages) messages.seek(offset) messages = decoded return str([cls.ITEM.repr(m) for m in messages]) kafka-1.3.2/kafka/protocol/metadata.py0000644001271300127130000000360713025302127017431 0ustar dpowers00000000000000from __future__ import absolute_import from .struct import Struct from .types import Array, Boolean, Int16, Int32, Schema, String class MetadataResponse_v0(Struct): API_KEY = 3 API_VERSION = 0 SCHEMA = Schema( ('brokers', Array( ('node_id', Int32), ('host', String('utf-8')), ('port', Int32))), ('topics', Array( ('error_code', Int16), ('topic', String('utf-8')), ('partitions', Array( ('error_code', Int16), ('partition', Int32), ('leader', Int32), ('replicas', Array(Int32)), ('isr', Array(Int32)))))) ) class MetadataResponse_v1(Struct): API_KEY = 3 API_VERSION = 1 SCHEMA = Schema( ('brokers', Array( ('node_id', Int32), ('host', String('utf-8')), ('port', Int32), ('rack', String('utf-8')))), ('controller_id', Int32), ('topics', Array( ('error_code', Int16), ('topic', String('utf-8')), ('is_internal', Boolean), ('partitions', Array( ('error_code', Int16), ('partition', Int32), ('leader', Int32), ('replicas', Array(Int32)), ('isr', Array(Int32)))))) ) class MetadataRequest_v0(Struct): API_KEY = 3 API_VERSION = 0 RESPONSE_TYPE = MetadataResponse_v0 SCHEMA = Schema( ('topics', Array(String('utf-8'))) # Empty Array (len 0) for all topics ) class MetadataRequest_v1(Struct): API_KEY = 3 API_VERSION = 1 RESPONSE_TYPE = MetadataResponse_v1 SCHEMA = Schema( ('topics', Array(String('utf-8'))) # Null Array (len -1) for all topics ) MetadataRequest = [MetadataRequest_v0, MetadataRequest_v1] MetadataResponse = [MetadataResponse_v0, MetadataResponse_v1] kafka-1.3.2/kafka/protocol/offset.py0000644001271300127130000000201413025302127017126 0ustar dpowers00000000000000from __future__ import absolute_import from .struct import Struct from .types import Array, Int16, Int32, Int64, Schema, String class OffsetResetStrategy(object): LATEST = -1 EARLIEST = -2 NONE = 0 class OffsetResponse_v0(Struct): API_KEY = 2 API_VERSION = 0 SCHEMA = Schema( ('topics', Array( ('topic', String('utf-8')), ('partitions', Array( ('partition', Int32), ('error_code', Int16), ('offsets', Array(Int64)))))) ) class OffsetRequest_v0(Struct): API_KEY = 2 API_VERSION = 0 RESPONSE_TYPE = OffsetResponse_v0 SCHEMA = Schema( ('replica_id', Int32), ('topics', Array( ('topic', String('utf-8')), ('partitions', Array( ('partition', Int32), ('time', Int64), ('max_offsets', Int32))))) ) DEFAULTS = { 'replica_id': -1 } OffsetRequest = [OffsetRequest_v0] OffsetResponse = [OffsetResponse_v0] kafka-1.3.2/kafka/protocol/pickle.py0000644001271300127130000000163013025302127017112 0ustar dpowers00000000000000from __future__ import absolute_import try: import copyreg # pylint: disable=import-error except ImportError: import copy_reg as copyreg # pylint: disable=import-error import types def _pickle_method(method): try: func_name = method.__func__.__name__ obj = method.__self__ cls = method.__self__.__class__ except AttributeError: func_name = method.im_func.__name__ obj = method.im_self cls = method.im_class return _unpickle_method, (func_name, obj, cls) def _unpickle_method(func_name, obj, cls): for cls in cls.mro(): try: func = cls.__dict__[func_name] except KeyError: pass else: break return func.__get__(obj, cls) # https://bytes.com/topic/python/answers/552476-why-cant-you-pickle-instancemethods copyreg.pickle(types.MethodType, _pickle_method, _unpickle_method) kafka-1.3.2/kafka/protocol/produce.py0000644001271300127130000000400113025302127017277 0ustar dpowers00000000000000from __future__ import absolute_import from .message import MessageSet from .struct import Struct from .types import Int16, Int32, Int64, String, Array, Schema class ProduceResponse_v0(Struct): API_KEY = 0 API_VERSION = 0 SCHEMA = Schema( ('topics', Array( ('topic', String('utf-8')), ('partitions', Array( ('partition', Int32), ('error_code', Int16), ('offset', Int64))))) ) class ProduceResponse_v1(Struct): API_KEY = 0 API_VERSION = 1 SCHEMA = Schema( ('topics', Array( ('topic', String('utf-8')), ('partitions', Array( ('partition', Int32), ('error_code', Int16), ('offset', Int64))))), ('throttle_time_ms', Int32) ) class ProduceResponse_v2(Struct): API_KEY = 0 API_VERSION = 2 SCHEMA = Schema( ('topics', Array( ('topic', String('utf-8')), ('partitions', Array( ('partition', Int32), ('error_code', Int16), ('offset', Int64), ('timestamp', Int64))))), ('throttle_time_ms', Int32) ) class ProduceRequest_v0(Struct): API_KEY = 0 API_VERSION = 0 RESPONSE_TYPE = ProduceResponse_v0 SCHEMA = Schema( ('required_acks', Int16), ('timeout', Int32), ('topics', Array( ('topic', String('utf-8')), ('partitions', Array( ('partition', Int32), ('messages', MessageSet))))) ) class ProduceRequest_v1(Struct): API_KEY = 0 API_VERSION = 1 RESPONSE_TYPE = ProduceResponse_v1 SCHEMA = ProduceRequest_v0.SCHEMA class ProduceRequest_v2(Struct): API_KEY = 0 API_VERSION = 2 RESPONSE_TYPE = ProduceResponse_v2 SCHEMA = ProduceRequest_v1.SCHEMA ProduceRequest = [ProduceRequest_v0, ProduceRequest_v1, ProduceRequest_v2] ProduceResponse = [ProduceResponse_v0, ProduceResponse_v1, ProduceResponse_v2] kafka-1.3.2/kafka/protocol/struct.py0000644001271300127130000000400313025302127017164 0ustar dpowers00000000000000from __future__ import absolute_import #from collections import namedtuple from io import BytesIO from .abstract import AbstractType from .types import Schema class Struct(AbstractType): SCHEMA = Schema() def __init__(self, *args, **kwargs): if len(args) == len(self.SCHEMA.fields): for i, name in enumerate(self.SCHEMA.names): self.__dict__[name] = args[i] elif len(args) > 0: raise ValueError('Args must be empty or mirror schema') else: self.__dict__.update(kwargs) # overloading encode() to support both class and instance self.encode = self._encode_self @classmethod def encode(cls, item): # pylint: disable=E0202 bits = [] for i, field in enumerate(cls.SCHEMA.fields): bits.append(field.encode(item[i])) return b''.join(bits) def _encode_self(self): return self.SCHEMA.encode( [self.__dict__[name] for name in self.SCHEMA.names] ) @classmethod def decode(cls, data): if isinstance(data, bytes): data = BytesIO(data) return cls(*[field.decode(data) for field in cls.SCHEMA.fields]) def __repr__(self): key_vals = [] for name, field in zip(self.SCHEMA.names, self.SCHEMA.fields): key_vals.append('%s=%s' % (name, field.repr(self.__dict__[name]))) return self.__class__.__name__ + '(' + ', '.join(key_vals) + ')' def __hash__(self): return hash(self.encode()) def __eq__(self, other): if self.SCHEMA != other.SCHEMA: return False for attr in self.SCHEMA.names: if self.__dict__[attr] != other.__dict__[attr]: return False return True """ class MetaStruct(type): def __new__(cls, clsname, bases, dct): nt = namedtuple(clsname, [name for (name, _) in dct['SCHEMA']]) bases = tuple([Struct, nt] + list(bases)) return super(MetaStruct, cls).__new__(cls, clsname, bases, dct) """ kafka-1.3.2/kafka/protocol/types.py0000644001271300127130000001061013025302127017005 0ustar dpowers00000000000000from __future__ import absolute_import from struct import pack, unpack, error from .abstract import AbstractType def _pack(f, value): try: return pack(f, value) except error: raise ValueError(error) def _unpack(f, data): try: (value,) = unpack(f, data) return value except error: raise ValueError(error) class Int8(AbstractType): @classmethod def encode(cls, value): return _pack('>b', value) @classmethod def decode(cls, data): return _unpack('>b', data.read(1)) class Int16(AbstractType): @classmethod def encode(cls, value): return _pack('>h', value) @classmethod def decode(cls, data): return _unpack('>h', data.read(2)) class Int32(AbstractType): @classmethod def encode(cls, value): return _pack('>i', value) @classmethod def decode(cls, data): return _unpack('>i', data.read(4)) class Int64(AbstractType): @classmethod def encode(cls, value): return _pack('>q', value) @classmethod def decode(cls, data): return _unpack('>q', data.read(8)) class String(AbstractType): def __init__(self, encoding='utf-8'): self.encoding = encoding def encode(self, value): if value is None: return Int16.encode(-1) value = str(value).encode(self.encoding) return Int16.encode(len(value)) + value def decode(self, data): length = Int16.decode(data) if length < 0: return None value = data.read(length) if len(value) != length: raise ValueError('Buffer underrun decoding string') return value.decode(self.encoding) class Bytes(AbstractType): @classmethod def encode(cls, value): if value is None: return Int32.encode(-1) else: return Int32.encode(len(value)) + value @classmethod def decode(cls, data): length = Int32.decode(data) if length < 0: return None value = data.read(length) if len(value) != length: raise ValueError('Buffer underrun decoding Bytes') return value class Boolean(AbstractType): @classmethod def encode(cls, value): return _pack('>?', value) @classmethod def decode(cls, data): return _unpack('>?', data.read(1)) class Schema(AbstractType): def __init__(self, *fields): if fields: self.names, self.fields = zip(*fields) else: self.names, self.fields = (), () def encode(self, item): if len(item) != len(self.fields): raise ValueError('Item field count does not match Schema') return b''.join([ field.encode(item[i]) for i, field in enumerate(self.fields) ]) def decode(self, data): return tuple([field.decode(data) for field in self.fields]) def __len__(self): return len(self.fields) def repr(self, value): key_vals = [] try: for i in range(len(self)): try: field_val = getattr(value, self.names[i]) except AttributeError: field_val = value[i] key_vals.append('%s=%s' % (self.names[i], self.fields[i].repr(field_val))) return '(' + ', '.join(key_vals) + ')' except: return repr(value) class Array(AbstractType): def __init__(self, *array_of): if len(array_of) > 1: self.array_of = Schema(*array_of) elif len(array_of) == 1 and (isinstance(array_of[0], AbstractType) or issubclass(array_of[0], AbstractType)): self.array_of = array_of[0] else: raise ValueError('Array instantiated with no array_of type') def encode(self, items): if items is None: return Int32.encode(-1) return b''.join( [Int32.encode(len(items))] + [self.array_of.encode(item) for item in items] ) def decode(self, data): length = Int32.decode(data) if length == -1: return None return [self.array_of.decode(data) for _ in range(length)] def repr(self, list_of_items): if list_of_items is None: return 'NULL' return '[' + ', '.join([self.array_of.repr(item) for item in list_of_items]) + ']' kafka-1.3.2/kafka/serializer/0000755001271300127130000000000013031057517015610 5ustar dpowers00000000000000kafka-1.3.2/kafka/serializer/__init__.py0000644001271300127130000000012713031057471017720 0ustar dpowers00000000000000from __future__ import absolute_import from .abstract import Serializer, Deserializer kafka-1.3.2/kafka/serializer/abstract.py0000644001271300127130000000074613031057471017773 0ustar dpowers00000000000000from __future__ import absolute_import import abc class Serializer(object): __meta__ = abc.ABCMeta def __init__(self, **config): pass @abc.abstractmethod def serialize(self, topic, value): pass def close(self): pass class Deserializer(object): __meta__ = abc.ABCMeta def __init__(self, **config): pass @abc.abstractmethod def deserialize(self, topic, bytes_): pass def close(self): pass kafka-1.3.2/kafka/structs.py0000644001271300127130000000622213025302127015513 0ustar dpowers00000000000000from __future__ import absolute_import from collections import namedtuple # SimpleClient Payload Structs - Deprecated # https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-MetadataAPI MetadataRequest = namedtuple("MetadataRequest", ["topics"]) MetadataResponse = namedtuple("MetadataResponse", ["brokers", "topics"]) # https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-ConsumerMetadataRequest ConsumerMetadataRequest = namedtuple("ConsumerMetadataRequest", ["groups"]) ConsumerMetadataResponse = namedtuple("ConsumerMetadataResponse", ["error", "nodeId", "host", "port"]) # https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-ProduceAPI ProduceRequestPayload = namedtuple("ProduceRequestPayload", ["topic", "partition", "messages"]) ProduceResponsePayload = namedtuple("ProduceResponsePayload", ["topic", "partition", "error", "offset"]) # https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchAPI FetchRequestPayload = namedtuple("FetchRequestPayload", ["topic", "partition", "offset", "max_bytes"]) FetchResponsePayload = namedtuple("FetchResponsePayload", ["topic", "partition", "error", "highwaterMark", "messages"]) # https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetAPI OffsetRequestPayload = namedtuple("OffsetRequestPayload", ["topic", "partition", "time", "max_offsets"]) OffsetResponsePayload = namedtuple("OffsetResponsePayload", ["topic", "partition", "error", "offsets"]) # https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-OffsetCommit/FetchAPI OffsetCommitRequestPayload = namedtuple("OffsetCommitRequestPayload", ["topic", "partition", "offset", "metadata"]) OffsetCommitResponsePayload = namedtuple("OffsetCommitResponsePayload", ["topic", "partition", "error"]) OffsetFetchRequestPayload = namedtuple("OffsetFetchRequestPayload", ["topic", "partition"]) OffsetFetchResponsePayload = namedtuple("OffsetFetchResponsePayload", ["topic", "partition", "offset", "metadata", "error"]) # Other useful structs TopicPartition = namedtuple("TopicPartition", ["topic", "partition"]) BrokerMetadata = namedtuple("BrokerMetadata", ["nodeId", "host", "port", "rack"]) PartitionMetadata = namedtuple("PartitionMetadata", ["topic", "partition", "leader", "replicas", "isr", "error"]) OffsetAndMetadata = namedtuple("OffsetAndMetadata", ["offset", "metadata"]) # Deprecated structs OffsetAndMessage = namedtuple("OffsetAndMessage", ["offset", "message"]) Message = namedtuple("Message", ["magic", "attributes", "key", "value"]) KafkaMessage = namedtuple("KafkaMessage", ["topic", "partition", "offset", "key", "value"]) # Define retry policy for async producer # Limit value: int >= 0, 0 means no retries RetryOptions = namedtuple("RetryOptions", ["limit", "backoff_ms", "retry_on_timeouts"]) # Support legacy imports from kafka.common from kafka.errors import * kafka-1.3.2/kafka/util.py0000644001271300127130000001357013025302127014765 0ustar dpowers00000000000000from __future__ import absolute_import import atexit import binascii import collections import struct import sys from threading import Thread, Event import weakref from kafka.vendor import six from kafka.errors import BufferUnderflowError def crc32(data): crc = binascii.crc32(data) # py2 and py3 behave a little differently # CRC is encoded as a signed int in kafka protocol # so we'll convert the py3 unsigned result to signed if six.PY3 and crc >= 2**31: crc -= 2**32 return crc def write_int_string(s): if s is not None and not isinstance(s, six.binary_type): raise TypeError('Expected "%s" to be bytes\n' 'data=%s' % (type(s), repr(s))) if s is None: return struct.pack('>i', -1) else: return struct.pack('>i%ds' % len(s), len(s), s) def write_short_string(s): if s is not None and not isinstance(s, six.binary_type): raise TypeError('Expected "%s" to be bytes\n' 'data=%s' % (type(s), repr(s))) if s is None: return struct.pack('>h', -1) elif len(s) > 32767 and sys.version_info < (2, 7): # Python 2.6 issues a deprecation warning instead of a struct error raise struct.error(len(s)) else: return struct.pack('>h%ds' % len(s), len(s), s) def read_short_string(data, cur): if len(data) < cur + 2: raise BufferUnderflowError("Not enough data left") (strlen,) = struct.unpack('>h', data[cur:cur + 2]) if strlen == -1: return None, cur + 2 cur += 2 if len(data) < cur + strlen: raise BufferUnderflowError("Not enough data left") out = data[cur:cur + strlen] return out, cur + strlen def read_int_string(data, cur): if len(data) < cur + 4: raise BufferUnderflowError( "Not enough data left to read string len (%d < %d)" % (len(data), cur + 4)) (strlen,) = struct.unpack('>i', data[cur:cur + 4]) if strlen == -1: return None, cur + 4 cur += 4 if len(data) < cur + strlen: raise BufferUnderflowError("Not enough data left") out = data[cur:cur + strlen] return out, cur + strlen def relative_unpack(fmt, data, cur): size = struct.calcsize(fmt) if len(data) < cur + size: raise BufferUnderflowError("Not enough data left") out = struct.unpack(fmt, data[cur:cur + size]) return out, cur + size def group_by_topic_and_partition(tuples): out = collections.defaultdict(dict) for t in tuples: assert t.topic not in out or t.partition not in out[t.topic], \ 'Duplicate {0}s for {1} {2}'.format(t.__class__.__name__, t.topic, t.partition) out[t.topic][t.partition] = t return out class ReentrantTimer(object): """ A timer that can be restarted, unlike threading.Timer (although this uses threading.Timer) Arguments: t: timer interval in milliseconds fn: a callable to invoke args: tuple of args to be passed to function kwargs: keyword arguments to be passed to function """ def __init__(self, t, fn, *args, **kwargs): if t <= 0: raise ValueError('Invalid timeout value') if not callable(fn): raise ValueError('fn must be callable') self.thread = None self.t = t / 1000.0 self.fn = fn self.args = args self.kwargs = kwargs self.active = None def _timer(self, active): # python2.6 Event.wait() always returns None # python2.7 and greater returns the flag value (true/false) # we want the flag value, so add an 'or' here for python2.6 # this is redundant for later python versions (FLAG OR FLAG == FLAG) while not (active.wait(self.t) or active.is_set()): self.fn(*self.args, **self.kwargs) def start(self): if self.thread is not None: self.stop() self.active = Event() self.thread = Thread(target=self._timer, args=(self.active,)) self.thread.daemon = True # So the app exits when main thread exits self.thread.start() def stop(self): if self.thread is None: return self.active.set() self.thread.join(self.t + 1) # noinspection PyAttributeOutsideInit self.timer = None self.fn = None def __del__(self): self.stop() class WeakMethod(object): """ Callable that weakly references a method and the object it is bound to. It is based on http://stackoverflow.com/a/24287465. Arguments: object_dot_method: A bound instance method (i.e. 'object.method'). """ def __init__(self, object_dot_method): try: self.target = weakref.ref(object_dot_method.__self__) except AttributeError: self.target = weakref.ref(object_dot_method.im_self) self._target_id = id(self.target()) try: self.method = weakref.ref(object_dot_method.__func__) except AttributeError: self.method = weakref.ref(object_dot_method.im_func) self._method_id = id(self.method()) def __call__(self, *args, **kwargs): """ Calls the method on target with args and kwargs. """ return self.method()(self.target(), *args, **kwargs) def __hash__(self): return hash(self.target) ^ hash(self.method) def __eq__(self, other): if not isinstance(other, WeakMethod): return False return self._target_id == other._target_id and self._method_id == other._method_id def try_method_on_system_exit(obj, method, *args, **kwargs): def wrapper(_obj, _meth, *args, **kwargs): try: getattr(_obj, _meth)(*args, **kwargs) except (ReferenceError, AttributeError): pass atexit.register(wrapper, weakref.proxy(obj), method, *args, **kwargs) kafka-1.3.2/kafka/vendor/0000755001271300127130000000000013031057517014734 5ustar dpowers00000000000000kafka-1.3.2/kafka/vendor/__init__.py0000644001271300127130000000000013025302127017024 0ustar dpowers00000000000000kafka-1.3.2/kafka/vendor/selectors34.py0000644001271300127130000004755113025302127017465 0ustar dpowers00000000000000# pylint: skip-file # vendored from https://github.com/berkerpeksag/selectors34 # at commit 5195dd2cbe598047ad0a2e446a829546f6ffc9eb (v1.1) # # Original author: Charles-Francois Natali (c.f.natali[at]gmail.com) # Maintainer: Berker Peksag (berker.peksag[at]gmail.com) # Also see https://pypi.python.org/pypi/selectors34 """Selectors module. This module allows high-level and efficient I/O multiplexing, built upon the `select` module primitives. The following code adapted from trollius.selectors. """ from __future__ import absolute_import from abc import ABCMeta, abstractmethod from collections import namedtuple, Mapping from errno import EINTR import math import select import sys from kafka.vendor import six def _wrap_error(exc, mapping, key): if key not in mapping: return new_err_cls = mapping[key] new_err = new_err_cls(*exc.args) # raise a new exception with the original traceback if hasattr(exc, '__traceback__'): traceback = exc.__traceback__ else: traceback = sys.exc_info()[2] six.reraise(new_err_cls, new_err, traceback) # generic events, that must be mapped to implementation-specific ones EVENT_READ = (1 << 0) EVENT_WRITE = (1 << 1) def _fileobj_to_fd(fileobj): """Return a file descriptor from a file object. Parameters: fileobj -- file object or file descriptor Returns: corresponding file descriptor Raises: ValueError if the object is invalid """ if isinstance(fileobj, six.integer_types): fd = fileobj else: try: fd = int(fileobj.fileno()) except (AttributeError, TypeError, ValueError): raise ValueError("Invalid file object: " "{0!r}".format(fileobj)) if fd < 0: raise ValueError("Invalid file descriptor: {0}".format(fd)) return fd SelectorKey = namedtuple('SelectorKey', ['fileobj', 'fd', 'events', 'data']) """Object used to associate a file object to its backing file descriptor, selected event mask and attached data.""" class _SelectorMapping(Mapping): """Mapping of file objects to selector keys.""" def __init__(self, selector): self._selector = selector def __len__(self): return len(self._selector._fd_to_key) def __getitem__(self, fileobj): try: fd = self._selector._fileobj_lookup(fileobj) return self._selector._fd_to_key[fd] except KeyError: raise KeyError("{0!r} is not registered".format(fileobj)) def __iter__(self): return iter(self._selector._fd_to_key) class BaseSelector(six.with_metaclass(ABCMeta)): """Selector abstract base class. A selector supports registering file objects to be monitored for specific I/O events. A file object is a file descriptor or any object with a `fileno()` method. An arbitrary object can be attached to the file object, which can be used for example to store context information, a callback, etc. A selector can use various implementations (select(), poll(), epoll()...) depending on the platform. The default `Selector` class uses the most efficient implementation on the current platform. """ @abstractmethod def register(self, fileobj, events, data=None): """Register a file object. Parameters: fileobj -- file object or file descriptor events -- events to monitor (bitwise mask of EVENT_READ|EVENT_WRITE) data -- attached data Returns: SelectorKey instance Raises: ValueError if events is invalid KeyError if fileobj is already registered OSError if fileobj is closed or otherwise is unacceptable to the underlying system call (if a system call is made) Note: OSError may or may not be raised """ raise NotImplementedError @abstractmethod def unregister(self, fileobj): """Unregister a file object. Parameters: fileobj -- file object or file descriptor Returns: SelectorKey instance Raises: KeyError if fileobj is not registered Note: If fileobj is registered but has since been closed this does *not* raise OSError (even if the wrapped syscall does) """ raise NotImplementedError def modify(self, fileobj, events, data=None): """Change a registered file object monitored events or attached data. Parameters: fileobj -- file object or file descriptor events -- events to monitor (bitwise mask of EVENT_READ|EVENT_WRITE) data -- attached data Returns: SelectorKey instance Raises: Anything that unregister() or register() raises """ self.unregister(fileobj) return self.register(fileobj, events, data) @abstractmethod def select(self, timeout=None): """Perform the actual selection, until some monitored file objects are ready or a timeout expires. Parameters: timeout -- if timeout > 0, this specifies the maximum wait time, in seconds if timeout <= 0, the select() call won't block, and will report the currently ready file objects if timeout is None, select() will block until a monitored file object becomes ready Returns: list of (key, events) for ready file objects `events` is a bitwise mask of EVENT_READ|EVENT_WRITE """ raise NotImplementedError def close(self): """Close the selector. This must be called to make sure that any underlying resource is freed. """ pass def get_key(self, fileobj): """Return the key associated to a registered file object. Returns: SelectorKey for this file object """ mapping = self.get_map() if mapping is None: raise RuntimeError('Selector is closed') try: return mapping[fileobj] except KeyError: raise KeyError("{0!r} is not registered".format(fileobj)) @abstractmethod def get_map(self): """Return a mapping of file objects to selector keys.""" raise NotImplementedError def __enter__(self): return self def __exit__(self, *args): self.close() class _BaseSelectorImpl(BaseSelector): """Base selector implementation.""" def __init__(self): # this maps file descriptors to keys self._fd_to_key = {} # read-only mapping returned by get_map() self._map = _SelectorMapping(self) def _fileobj_lookup(self, fileobj): """Return a file descriptor from a file object. This wraps _fileobj_to_fd() to do an exhaustive search in case the object is invalid but we still have it in our map. This is used by unregister() so we can unregister an object that was previously registered even if it is closed. It is also used by _SelectorMapping. """ try: return _fileobj_to_fd(fileobj) except ValueError: # Do an exhaustive search. for key in self._fd_to_key.values(): if key.fileobj is fileobj: return key.fd # Raise ValueError after all. raise def register(self, fileobj, events, data=None): if (not events) or (events & ~(EVENT_READ | EVENT_WRITE)): raise ValueError("Invalid events: {0!r}".format(events)) key = SelectorKey(fileobj, self._fileobj_lookup(fileobj), events, data) if key.fd in self._fd_to_key: raise KeyError("{0!r} (FD {1}) is already registered" .format(fileobj, key.fd)) self._fd_to_key[key.fd] = key return key def unregister(self, fileobj): try: key = self._fd_to_key.pop(self._fileobj_lookup(fileobj)) except KeyError: raise KeyError("{0!r} is not registered".format(fileobj)) return key def modify(self, fileobj, events, data=None): # TODO: Subclasses can probably optimize this even further. try: key = self._fd_to_key[self._fileobj_lookup(fileobj)] except KeyError: raise KeyError("{0!r} is not registered".format(fileobj)) if events != key.events: self.unregister(fileobj) key = self.register(fileobj, events, data) elif data != key.data: # Use a shortcut to update the data. key = key._replace(data=data) self._fd_to_key[key.fd] = key return key def close(self): self._fd_to_key.clear() self._map = None def get_map(self): return self._map def _key_from_fd(self, fd): """Return the key associated to a given file descriptor. Parameters: fd -- file descriptor Returns: corresponding key, or None if not found """ try: return self._fd_to_key[fd] except KeyError: return None class SelectSelector(_BaseSelectorImpl): """Select-based selector.""" def __init__(self): super(SelectSelector, self).__init__() self._readers = set() self._writers = set() def register(self, fileobj, events, data=None): key = super(SelectSelector, self).register(fileobj, events, data) if events & EVENT_READ: self._readers.add(key.fd) if events & EVENT_WRITE: self._writers.add(key.fd) return key def unregister(self, fileobj): key = super(SelectSelector, self).unregister(fileobj) self._readers.discard(key.fd) self._writers.discard(key.fd) return key if sys.platform == 'win32': def _select(self, r, w, _, timeout=None): r, w, x = select.select(r, w, w, timeout) return r, w + x, [] else: _select = select.select def select(self, timeout=None): timeout = None if timeout is None else max(timeout, 0) ready = [] try: r, w, _ = self._select(self._readers, self._writers, [], timeout) except select.error as exc: if exc.args[0] == EINTR: return ready else: raise r = set(r) w = set(w) for fd in r | w: events = 0 if fd in r: events |= EVENT_READ if fd in w: events |= EVENT_WRITE key = self._key_from_fd(fd) if key: ready.append((key, events & key.events)) return ready if hasattr(select, 'poll'): class PollSelector(_BaseSelectorImpl): """Poll-based selector.""" def __init__(self): super(PollSelector, self).__init__() self._poll = select.poll() def register(self, fileobj, events, data=None): key = super(PollSelector, self).register(fileobj, events, data) poll_events = 0 if events & EVENT_READ: poll_events |= select.POLLIN if events & EVENT_WRITE: poll_events |= select.POLLOUT self._poll.register(key.fd, poll_events) return key def unregister(self, fileobj): key = super(PollSelector, self).unregister(fileobj) self._poll.unregister(key.fd) return key def select(self, timeout=None): if timeout is None: timeout = None elif timeout <= 0: timeout = 0 else: # poll() has a resolution of 1 millisecond, round away from # zero to wait *at least* timeout seconds. timeout = int(math.ceil(timeout * 1e3)) ready = [] try: fd_event_list = self._poll.poll(timeout) except select.error as exc: if exc.args[0] == EINTR: return ready else: raise for fd, event in fd_event_list: events = 0 if event & ~select.POLLIN: events |= EVENT_WRITE if event & ~select.POLLOUT: events |= EVENT_READ key = self._key_from_fd(fd) if key: ready.append((key, events & key.events)) return ready if hasattr(select, 'epoll'): class EpollSelector(_BaseSelectorImpl): """Epoll-based selector.""" def __init__(self): super(EpollSelector, self).__init__() self._epoll = select.epoll() def fileno(self): return self._epoll.fileno() def register(self, fileobj, events, data=None): key = super(EpollSelector, self).register(fileobj, events, data) epoll_events = 0 if events & EVENT_READ: epoll_events |= select.EPOLLIN if events & EVENT_WRITE: epoll_events |= select.EPOLLOUT self._epoll.register(key.fd, epoll_events) return key def unregister(self, fileobj): key = super(EpollSelector, self).unregister(fileobj) try: self._epoll.unregister(key.fd) except IOError: # This can happen if the FD was closed since it # was registered. pass return key def select(self, timeout=None): if timeout is None: timeout = -1 elif timeout <= 0: timeout = 0 else: # epoll_wait() has a resolution of 1 millisecond, round away # from zero to wait *at least* timeout seconds. timeout = math.ceil(timeout * 1e3) * 1e-3 # epoll_wait() expects `maxevents` to be greater than zero; # we want to make sure that `select()` can be called when no # FD is registered. max_ev = max(len(self._fd_to_key), 1) ready = [] try: fd_event_list = self._epoll.poll(timeout, max_ev) except IOError as exc: if exc.errno == EINTR: return ready else: raise for fd, event in fd_event_list: events = 0 if event & ~select.EPOLLIN: events |= EVENT_WRITE if event & ~select.EPOLLOUT: events |= EVENT_READ key = self._key_from_fd(fd) if key: ready.append((key, events & key.events)) return ready def close(self): self._epoll.close() super(EpollSelector, self).close() if hasattr(select, 'devpoll'): class DevpollSelector(_BaseSelectorImpl): """Solaris /dev/poll selector.""" def __init__(self): super(DevpollSelector, self).__init__() self._devpoll = select.devpoll() def fileno(self): return self._devpoll.fileno() def register(self, fileobj, events, data=None): key = super(DevpollSelector, self).register(fileobj, events, data) poll_events = 0 if events & EVENT_READ: poll_events |= select.POLLIN if events & EVENT_WRITE: poll_events |= select.POLLOUT self._devpoll.register(key.fd, poll_events) return key def unregister(self, fileobj): key = super(DevpollSelector, self).unregister(fileobj) self._devpoll.unregister(key.fd) return key def select(self, timeout=None): if timeout is None: timeout = None elif timeout <= 0: timeout = 0 else: # devpoll() has a resolution of 1 millisecond, round away from # zero to wait *at least* timeout seconds. timeout = math.ceil(timeout * 1e3) ready = [] try: fd_event_list = self._devpoll.poll(timeout) except OSError as exc: if exc.errno == EINTR: return ready else: raise for fd, event in fd_event_list: events = 0 if event & ~select.POLLIN: events |= EVENT_WRITE if event & ~select.POLLOUT: events |= EVENT_READ key = self._key_from_fd(fd) if key: ready.append((key, events & key.events)) return ready def close(self): self._devpoll.close() super(DevpollSelector, self).close() if hasattr(select, 'kqueue'): class KqueueSelector(_BaseSelectorImpl): """Kqueue-based selector.""" def __init__(self): super(KqueueSelector, self).__init__() self._kqueue = select.kqueue() def fileno(self): return self._kqueue.fileno() def register(self, fileobj, events, data=None): key = super(KqueueSelector, self).register(fileobj, events, data) if events & EVENT_READ: kev = select.kevent(key.fd, select.KQ_FILTER_READ, select.KQ_EV_ADD) self._kqueue.control([kev], 0, 0) if events & EVENT_WRITE: kev = select.kevent(key.fd, select.KQ_FILTER_WRITE, select.KQ_EV_ADD) self._kqueue.control([kev], 0, 0) return key def unregister(self, fileobj): key = super(KqueueSelector, self).unregister(fileobj) if key.events & EVENT_READ: kev = select.kevent(key.fd, select.KQ_FILTER_READ, select.KQ_EV_DELETE) try: self._kqueue.control([kev], 0, 0) except OSError: # This can happen if the FD was closed since it # was registered. pass if key.events & EVENT_WRITE: kev = select.kevent(key.fd, select.KQ_FILTER_WRITE, select.KQ_EV_DELETE) try: self._kqueue.control([kev], 0, 0) except OSError: # See comment above. pass return key def select(self, timeout=None): timeout = None if timeout is None else max(timeout, 0) max_ev = len(self._fd_to_key) ready = [] try: kev_list = self._kqueue.control(None, max_ev, timeout) except OSError as exc: if exc.errno == EINTR: return ready else: raise for kev in kev_list: fd = kev.ident flag = kev.filter events = 0 if flag == select.KQ_FILTER_READ: events |= EVENT_READ if flag == select.KQ_FILTER_WRITE: events |= EVENT_WRITE key = self._key_from_fd(fd) if key: ready.append((key, events & key.events)) return ready def close(self): self._kqueue.close() super(KqueueSelector, self).close() # Choose the best implementation, roughly: # epoll|kqueue|devpoll > poll > select. # select() also can't accept a FD > FD_SETSIZE (usually around 1024) if 'KqueueSelector' in globals(): DefaultSelector = KqueueSelector elif 'EpollSelector' in globals(): DefaultSelector = EpollSelector elif 'DevpollSelector' in globals(): DefaultSelector = DevpollSelector elif 'PollSelector' in globals(): DefaultSelector = PollSelector else: DefaultSelector = SelectSelector kafka-1.3.2/kafka/vendor/six.py0000644001271300127130000007264613025302127016121 0ustar dpowers00000000000000# pylint: skip-file """Utilities for writing code that runs on Python 2 and 3""" # Copyright (c) 2010-2015 Benjamin Peterson # # Permission is hereby granted, free of charge, to any person obtaining a copy # of this software and associated documentation files (the "Software"), to deal # in the Software without restriction, including without limitation the rights # to use, copy, modify, merge, publish, distribute, sublicense, and/or sell # copies of the Software, and to permit persons to whom the Software is # furnished to do so, subject to the following conditions: # # The above copyright notice and this permission notice shall be included in all # copies or substantial portions of the Software. # # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, # FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE # AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER # LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, # OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE # SOFTWARE. from __future__ import absolute_import import functools import itertools import operator import sys import types __author__ = "Benjamin Peterson " __version__ = "1.10.0" # Useful for very coarse version differentiation. PY2 = sys.version_info[0] == 2 PY3 = sys.version_info[0] == 3 PY34 = sys.version_info[0:2] >= (3, 4) if PY3: string_types = str, integer_types = int, class_types = type, text_type = str binary_type = bytes MAXSIZE = sys.maxsize else: string_types = basestring, integer_types = (int, long) class_types = (type, types.ClassType) text_type = unicode binary_type = str if sys.platform.startswith("java"): # Jython always uses 32 bits. MAXSIZE = int((1 << 31) - 1) else: # It's possible to have sizeof(long) != sizeof(Py_ssize_t). class X(object): def __len__(self): return 1 << 31 try: len(X()) except OverflowError: # 32-bit MAXSIZE = int((1 << 31) - 1) else: # 64-bit MAXSIZE = int((1 << 63) - 1) del X def _add_doc(func, doc): """Add documentation to a function.""" func.__doc__ = doc def _import_module(name): """Import module, returning the module after the last dot.""" __import__(name) return sys.modules[name] class _LazyDescr(object): def __init__(self, name): self.name = name def __get__(self, obj, tp): result = self._resolve() setattr(obj, self.name, result) # Invokes __set__. try: # This is a bit ugly, but it avoids running this again by # removing this descriptor. delattr(obj.__class__, self.name) except AttributeError: pass return result class MovedModule(_LazyDescr): def __init__(self, name, old, new=None): super(MovedModule, self).__init__(name) if PY3: if new is None: new = name self.mod = new else: self.mod = old def _resolve(self): return _import_module(self.mod) def __getattr__(self, attr): _module = self._resolve() value = getattr(_module, attr) setattr(self, attr, value) return value class _LazyModule(types.ModuleType): def __init__(self, name): super(_LazyModule, self).__init__(name) self.__doc__ = self.__class__.__doc__ def __dir__(self): attrs = ["__doc__", "__name__"] attrs += [attr.name for attr in self._moved_attributes] return attrs # Subclasses should override this _moved_attributes = [] class MovedAttribute(_LazyDescr): def __init__(self, name, old_mod, new_mod, old_attr=None, new_attr=None): super(MovedAttribute, self).__init__(name) if PY3: if new_mod is None: new_mod = name self.mod = new_mod if new_attr is None: if old_attr is None: new_attr = name else: new_attr = old_attr self.attr = new_attr else: self.mod = old_mod if old_attr is None: old_attr = name self.attr = old_attr def _resolve(self): module = _import_module(self.mod) return getattr(module, self.attr) class _SixMetaPathImporter(object): """ A meta path importer to import six.moves and its submodules. This class implements a PEP302 finder and loader. It should be compatible with Python 2.5 and all existing versions of Python3 """ def __init__(self, six_module_name): self.name = six_module_name self.known_modules = {} def _add_module(self, mod, *fullnames): for fullname in fullnames: self.known_modules[self.name + "." + fullname] = mod def _get_module(self, fullname): return self.known_modules[self.name + "." + fullname] def find_module(self, fullname, path=None): if fullname in self.known_modules: return self return None def __get_module(self, fullname): try: return self.known_modules[fullname] except KeyError: raise ImportError("This loader does not know module " + fullname) def load_module(self, fullname): try: # in case of a reload return sys.modules[fullname] except KeyError: pass mod = self.__get_module(fullname) if isinstance(mod, MovedModule): mod = mod._resolve() else: mod.__loader__ = self sys.modules[fullname] = mod return mod def is_package(self, fullname): """ Return true, if the named module is a package. We need this method to get correct spec objects with Python 3.4 (see PEP451) """ return hasattr(self.__get_module(fullname), "__path__") def get_code(self, fullname): """Return None Required, if is_package is implemented""" self.__get_module(fullname) # eventually raises ImportError return None get_source = get_code # same as get_code _importer = _SixMetaPathImporter(__name__) class _MovedItems(_LazyModule): """Lazy loading of moved objects""" __path__ = [] # mark as package _moved_attributes = [ MovedAttribute("cStringIO", "cStringIO", "io", "StringIO"), MovedAttribute("filter", "itertools", "builtins", "ifilter", "filter"), MovedAttribute("filterfalse", "itertools", "itertools", "ifilterfalse", "filterfalse"), MovedAttribute("input", "__builtin__", "builtins", "raw_input", "input"), MovedAttribute("intern", "__builtin__", "sys"), MovedAttribute("map", "itertools", "builtins", "imap", "map"), MovedAttribute("getcwd", "os", "os", "getcwdu", "getcwd"), MovedAttribute("getcwdb", "os", "os", "getcwd", "getcwdb"), MovedAttribute("range", "__builtin__", "builtins", "xrange", "range"), MovedAttribute("reload_module", "__builtin__", "importlib" if PY34 else "imp", "reload"), MovedAttribute("reduce", "__builtin__", "functools"), MovedAttribute("shlex_quote", "pipes", "shlex", "quote"), MovedAttribute("StringIO", "StringIO", "io"), MovedAttribute("UserDict", "UserDict", "collections"), MovedAttribute("UserList", "UserList", "collections"), MovedAttribute("UserString", "UserString", "collections"), MovedAttribute("xrange", "__builtin__", "builtins", "xrange", "range"), MovedAttribute("zip", "itertools", "builtins", "izip", "zip"), MovedAttribute("zip_longest", "itertools", "itertools", "izip_longest", "zip_longest"), MovedModule("builtins", "__builtin__"), MovedModule("configparser", "ConfigParser"), MovedModule("copyreg", "copy_reg"), MovedModule("dbm_gnu", "gdbm", "dbm.gnu"), MovedModule("_dummy_thread", "dummy_thread", "_dummy_thread"), MovedModule("http_cookiejar", "cookielib", "http.cookiejar"), MovedModule("http_cookies", "Cookie", "http.cookies"), MovedModule("html_entities", "htmlentitydefs", "html.entities"), MovedModule("html_parser", "HTMLParser", "html.parser"), MovedModule("http_client", "httplib", "http.client"), MovedModule("email_mime_multipart", "email.MIMEMultipart", "email.mime.multipart"), MovedModule("email_mime_nonmultipart", "email.MIMENonMultipart", "email.mime.nonmultipart"), MovedModule("email_mime_text", "email.MIMEText", "email.mime.text"), MovedModule("email_mime_base", "email.MIMEBase", "email.mime.base"), MovedModule("BaseHTTPServer", "BaseHTTPServer", "http.server"), MovedModule("CGIHTTPServer", "CGIHTTPServer", "http.server"), MovedModule("SimpleHTTPServer", "SimpleHTTPServer", "http.server"), MovedModule("cPickle", "cPickle", "pickle"), MovedModule("queue", "Queue"), MovedModule("reprlib", "repr"), MovedModule("socketserver", "SocketServer"), MovedModule("_thread", "thread", "_thread"), MovedModule("tkinter", "Tkinter"), MovedModule("tkinter_dialog", "Dialog", "tkinter.dialog"), MovedModule("tkinter_filedialog", "FileDialog", "tkinter.filedialog"), MovedModule("tkinter_scrolledtext", "ScrolledText", "tkinter.scrolledtext"), MovedModule("tkinter_simpledialog", "SimpleDialog", "tkinter.simpledialog"), MovedModule("tkinter_tix", "Tix", "tkinter.tix"), MovedModule("tkinter_ttk", "ttk", "tkinter.ttk"), MovedModule("tkinter_constants", "Tkconstants", "tkinter.constants"), MovedModule("tkinter_dnd", "Tkdnd", "tkinter.dnd"), MovedModule("tkinter_colorchooser", "tkColorChooser", "tkinter.colorchooser"), MovedModule("tkinter_commondialog", "tkCommonDialog", "tkinter.commondialog"), MovedModule("tkinter_tkfiledialog", "tkFileDialog", "tkinter.filedialog"), MovedModule("tkinter_font", "tkFont", "tkinter.font"), MovedModule("tkinter_messagebox", "tkMessageBox", "tkinter.messagebox"), MovedModule("tkinter_tksimpledialog", "tkSimpleDialog", "tkinter.simpledialog"), MovedModule("urllib_parse", __name__ + ".moves.urllib_parse", "urllib.parse"), MovedModule("urllib_error", __name__ + ".moves.urllib_error", "urllib.error"), MovedModule("urllib", __name__ + ".moves.urllib", __name__ + ".moves.urllib"), MovedModule("urllib_robotparser", "robotparser", "urllib.robotparser"), MovedModule("xmlrpc_client", "xmlrpclib", "xmlrpc.client"), MovedModule("xmlrpc_server", "SimpleXMLRPCServer", "xmlrpc.server"), ] # Add windows specific modules. if sys.platform == "win32": _moved_attributes += [ MovedModule("winreg", "_winreg"), ] for attr in _moved_attributes: setattr(_MovedItems, attr.name, attr) if isinstance(attr, MovedModule): _importer._add_module(attr, "moves." + attr.name) del attr _MovedItems._moved_attributes = _moved_attributes moves = _MovedItems(__name__ + ".moves") _importer._add_module(moves, "moves") class Module_six_moves_urllib_parse(_LazyModule): """Lazy loading of moved objects in six.moves.urllib_parse""" _urllib_parse_moved_attributes = [ MovedAttribute("ParseResult", "urlparse", "urllib.parse"), MovedAttribute("SplitResult", "urlparse", "urllib.parse"), MovedAttribute("parse_qs", "urlparse", "urllib.parse"), MovedAttribute("parse_qsl", "urlparse", "urllib.parse"), MovedAttribute("urldefrag", "urlparse", "urllib.parse"), MovedAttribute("urljoin", "urlparse", "urllib.parse"), MovedAttribute("urlparse", "urlparse", "urllib.parse"), MovedAttribute("urlsplit", "urlparse", "urllib.parse"), MovedAttribute("urlunparse", "urlparse", "urllib.parse"), MovedAttribute("urlunsplit", "urlparse", "urllib.parse"), MovedAttribute("quote", "urllib", "urllib.parse"), MovedAttribute("quote_plus", "urllib", "urllib.parse"), MovedAttribute("unquote", "urllib", "urllib.parse"), MovedAttribute("unquote_plus", "urllib", "urllib.parse"), MovedAttribute("urlencode", "urllib", "urllib.parse"), MovedAttribute("splitquery", "urllib", "urllib.parse"), MovedAttribute("splittag", "urllib", "urllib.parse"), MovedAttribute("splituser", "urllib", "urllib.parse"), MovedAttribute("uses_fragment", "urlparse", "urllib.parse"), MovedAttribute("uses_netloc", "urlparse", "urllib.parse"), MovedAttribute("uses_params", "urlparse", "urllib.parse"), MovedAttribute("uses_query", "urlparse", "urllib.parse"), MovedAttribute("uses_relative", "urlparse", "urllib.parse"), ] for attr in _urllib_parse_moved_attributes: setattr(Module_six_moves_urllib_parse, attr.name, attr) del attr Module_six_moves_urllib_parse._moved_attributes = _urllib_parse_moved_attributes _importer._add_module(Module_six_moves_urllib_parse(__name__ + ".moves.urllib_parse"), "moves.urllib_parse", "moves.urllib.parse") class Module_six_moves_urllib_error(_LazyModule): """Lazy loading of moved objects in six.moves.urllib_error""" _urllib_error_moved_attributes = [ MovedAttribute("URLError", "urllib2", "urllib.error"), MovedAttribute("HTTPError", "urllib2", "urllib.error"), MovedAttribute("ContentTooShortError", "urllib", "urllib.error"), ] for attr in _urllib_error_moved_attributes: setattr(Module_six_moves_urllib_error, attr.name, attr) del attr Module_six_moves_urllib_error._moved_attributes = _urllib_error_moved_attributes _importer._add_module(Module_six_moves_urllib_error(__name__ + ".moves.urllib.error"), "moves.urllib_error", "moves.urllib.error") class Module_six_moves_urllib_request(_LazyModule): """Lazy loading of moved objects in six.moves.urllib_request""" _urllib_request_moved_attributes = [ MovedAttribute("urlopen", "urllib2", "urllib.request"), MovedAttribute("install_opener", "urllib2", "urllib.request"), MovedAttribute("build_opener", "urllib2", "urllib.request"), MovedAttribute("pathname2url", "urllib", "urllib.request"), MovedAttribute("url2pathname", "urllib", "urllib.request"), MovedAttribute("getproxies", "urllib", "urllib.request"), MovedAttribute("Request", "urllib2", "urllib.request"), MovedAttribute("OpenerDirector", "urllib2", "urllib.request"), MovedAttribute("HTTPDefaultErrorHandler", "urllib2", "urllib.request"), MovedAttribute("HTTPRedirectHandler", "urllib2", "urllib.request"), MovedAttribute("HTTPCookieProcessor", "urllib2", "urllib.request"), MovedAttribute("ProxyHandler", "urllib2", "urllib.request"), MovedAttribute("BaseHandler", "urllib2", "urllib.request"), MovedAttribute("HTTPPasswordMgr", "urllib2", "urllib.request"), MovedAttribute("HTTPPasswordMgrWithDefaultRealm", "urllib2", "urllib.request"), MovedAttribute("AbstractBasicAuthHandler", "urllib2", "urllib.request"), MovedAttribute("HTTPBasicAuthHandler", "urllib2", "urllib.request"), MovedAttribute("ProxyBasicAuthHandler", "urllib2", "urllib.request"), MovedAttribute("AbstractDigestAuthHandler", "urllib2", "urllib.request"), MovedAttribute("HTTPDigestAuthHandler", "urllib2", "urllib.request"), MovedAttribute("ProxyDigestAuthHandler", "urllib2", "urllib.request"), MovedAttribute("HTTPHandler", "urllib2", "urllib.request"), MovedAttribute("HTTPSHandler", "urllib2", "urllib.request"), MovedAttribute("FileHandler", "urllib2", "urllib.request"), MovedAttribute("FTPHandler", "urllib2", "urllib.request"), MovedAttribute("CacheFTPHandler", "urllib2", "urllib.request"), MovedAttribute("UnknownHandler", "urllib2", "urllib.request"), MovedAttribute("HTTPErrorProcessor", "urllib2", "urllib.request"), MovedAttribute("urlretrieve", "urllib", "urllib.request"), MovedAttribute("urlcleanup", "urllib", "urllib.request"), MovedAttribute("URLopener", "urllib", "urllib.request"), MovedAttribute("FancyURLopener", "urllib", "urllib.request"), MovedAttribute("proxy_bypass", "urllib", "urllib.request"), ] for attr in _urllib_request_moved_attributes: setattr(Module_six_moves_urllib_request, attr.name, attr) del attr Module_six_moves_urllib_request._moved_attributes = _urllib_request_moved_attributes _importer._add_module(Module_six_moves_urllib_request(__name__ + ".moves.urllib.request"), "moves.urllib_request", "moves.urllib.request") class Module_six_moves_urllib_response(_LazyModule): """Lazy loading of moved objects in six.moves.urllib_response""" _urllib_response_moved_attributes = [ MovedAttribute("addbase", "urllib", "urllib.response"), MovedAttribute("addclosehook", "urllib", "urllib.response"), MovedAttribute("addinfo", "urllib", "urllib.response"), MovedAttribute("addinfourl", "urllib", "urllib.response"), ] for attr in _urllib_response_moved_attributes: setattr(Module_six_moves_urllib_response, attr.name, attr) del attr Module_six_moves_urllib_response._moved_attributes = _urllib_response_moved_attributes _importer._add_module(Module_six_moves_urllib_response(__name__ + ".moves.urllib.response"), "moves.urllib_response", "moves.urllib.response") class Module_six_moves_urllib_robotparser(_LazyModule): """Lazy loading of moved objects in six.moves.urllib_robotparser""" _urllib_robotparser_moved_attributes = [ MovedAttribute("RobotFileParser", "robotparser", "urllib.robotparser"), ] for attr in _urllib_robotparser_moved_attributes: setattr(Module_six_moves_urllib_robotparser, attr.name, attr) del attr Module_six_moves_urllib_robotparser._moved_attributes = _urllib_robotparser_moved_attributes _importer._add_module(Module_six_moves_urllib_robotparser(__name__ + ".moves.urllib.robotparser"), "moves.urllib_robotparser", "moves.urllib.robotparser") class Module_six_moves_urllib(types.ModuleType): """Create a six.moves.urllib namespace that resembles the Python 3 namespace""" __path__ = [] # mark as package parse = _importer._get_module("moves.urllib_parse") error = _importer._get_module("moves.urllib_error") request = _importer._get_module("moves.urllib_request") response = _importer._get_module("moves.urllib_response") robotparser = _importer._get_module("moves.urllib_robotparser") def __dir__(self): return ['parse', 'error', 'request', 'response', 'robotparser'] _importer._add_module(Module_six_moves_urllib(__name__ + ".moves.urllib"), "moves.urllib") def add_move(move): """Add an item to six.moves.""" setattr(_MovedItems, move.name, move) def remove_move(name): """Remove item from six.moves.""" try: delattr(_MovedItems, name) except AttributeError: try: del moves.__dict__[name] except KeyError: raise AttributeError("no such move, %r" % (name,)) if PY3: _meth_func = "__func__" _meth_self = "__self__" _func_closure = "__closure__" _func_code = "__code__" _func_defaults = "__defaults__" _func_globals = "__globals__" else: _meth_func = "im_func" _meth_self = "im_self" _func_closure = "func_closure" _func_code = "func_code" _func_defaults = "func_defaults" _func_globals = "func_globals" try: advance_iterator = next except NameError: def advance_iterator(it): return it.next() next = advance_iterator try: callable = callable except NameError: def callable(obj): return any("__call__" in klass.__dict__ for klass in type(obj).__mro__) if PY3: def get_unbound_function(unbound): return unbound create_bound_method = types.MethodType def create_unbound_method(func, cls): return func Iterator = object else: def get_unbound_function(unbound): return unbound.im_func def create_bound_method(func, obj): return types.MethodType(func, obj, obj.__class__) def create_unbound_method(func, cls): return types.MethodType(func, None, cls) class Iterator(object): def next(self): return type(self).__next__(self) callable = callable _add_doc(get_unbound_function, """Get the function out of a possibly unbound function""") get_method_function = operator.attrgetter(_meth_func) get_method_self = operator.attrgetter(_meth_self) get_function_closure = operator.attrgetter(_func_closure) get_function_code = operator.attrgetter(_func_code) get_function_defaults = operator.attrgetter(_func_defaults) get_function_globals = operator.attrgetter(_func_globals) if PY3: def iterkeys(d, **kw): return iter(d.keys(**kw)) def itervalues(d, **kw): return iter(d.values(**kw)) def iteritems(d, **kw): return iter(d.items(**kw)) def iterlists(d, **kw): return iter(d.lists(**kw)) viewkeys = operator.methodcaller("keys") viewvalues = operator.methodcaller("values") viewitems = operator.methodcaller("items") else: def iterkeys(d, **kw): return d.iterkeys(**kw) def itervalues(d, **kw): return d.itervalues(**kw) def iteritems(d, **kw): return d.iteritems(**kw) def iterlists(d, **kw): return d.iterlists(**kw) viewkeys = operator.methodcaller("viewkeys") viewvalues = operator.methodcaller("viewvalues") viewitems = operator.methodcaller("viewitems") _add_doc(iterkeys, "Return an iterator over the keys of a dictionary.") _add_doc(itervalues, "Return an iterator over the values of a dictionary.") _add_doc(iteritems, "Return an iterator over the (key, value) pairs of a dictionary.") _add_doc(iterlists, "Return an iterator over the (key, [values]) pairs of a dictionary.") if PY3: def b(s): return s.encode("latin-1") def u(s): return s unichr = chr import struct int2byte = struct.Struct(">B").pack del struct byte2int = operator.itemgetter(0) indexbytes = operator.getitem iterbytes = iter import io StringIO = io.StringIO BytesIO = io.BytesIO _assertCountEqual = "assertCountEqual" if sys.version_info[1] <= 1: _assertRaisesRegex = "assertRaisesRegexp" _assertRegex = "assertRegexpMatches" else: _assertRaisesRegex = "assertRaisesRegex" _assertRegex = "assertRegex" else: def b(s): return s # Workaround for standalone backslash def u(s): return unicode(s.replace(r'\\', r'\\\\'), "unicode_escape") unichr = unichr int2byte = chr def byte2int(bs): return ord(bs[0]) def indexbytes(buf, i): return ord(buf[i]) iterbytes = functools.partial(itertools.imap, ord) import StringIO StringIO = BytesIO = StringIO.StringIO _assertCountEqual = "assertItemsEqual" _assertRaisesRegex = "assertRaisesRegexp" _assertRegex = "assertRegexpMatches" _add_doc(b, """Byte literal""") _add_doc(u, """Text literal""") def assertCountEqual(self, *args, **kwargs): return getattr(self, _assertCountEqual)(*args, **kwargs) def assertRaisesRegex(self, *args, **kwargs): return getattr(self, _assertRaisesRegex)(*args, **kwargs) def assertRegex(self, *args, **kwargs): return getattr(self, _assertRegex)(*args, **kwargs) if PY3: exec_ = getattr(moves.builtins, "exec") def reraise(tp, value, tb=None): if value is None: value = tp() if value.__traceback__ is not tb: raise value.with_traceback(tb) raise value else: def exec_(_code_, _globs_=None, _locs_=None): """Execute code in a namespace.""" if _globs_ is None: frame = sys._getframe(1) _globs_ = frame.f_globals if _locs_ is None: _locs_ = frame.f_locals del frame elif _locs_ is None: _locs_ = _globs_ exec("""exec _code_ in _globs_, _locs_""") exec_("""def reraise(tp, value, tb=None): raise tp, value, tb """) if sys.version_info[:2] == (3, 2): exec_("""def raise_from(value, from_value): if from_value is None: raise value raise value from from_value """) elif sys.version_info[:2] > (3, 2): exec_("""def raise_from(value, from_value): raise value from from_value """) else: def raise_from(value, from_value): raise value print_ = getattr(moves.builtins, "print", None) if print_ is None: def print_(*args, **kwargs): """The new-style print function for Python 2.4 and 2.5.""" fp = kwargs.pop("file", sys.stdout) if fp is None: return def write(data): if not isinstance(data, basestring): data = str(data) # If the file has an encoding, encode unicode with it. if (isinstance(fp, file) and isinstance(data, unicode) and fp.encoding is not None): errors = getattr(fp, "errors", None) if errors is None: errors = "strict" data = data.encode(fp.encoding, errors) fp.write(data) want_unicode = False sep = kwargs.pop("sep", None) if sep is not None: if isinstance(sep, unicode): want_unicode = True elif not isinstance(sep, str): raise TypeError("sep must be None or a string") end = kwargs.pop("end", None) if end is not None: if isinstance(end, unicode): want_unicode = True elif not isinstance(end, str): raise TypeError("end must be None or a string") if kwargs: raise TypeError("invalid keyword arguments to print()") if not want_unicode: for arg in args: if isinstance(arg, unicode): want_unicode = True break if want_unicode: newline = unicode("\n") space = unicode(" ") else: newline = "\n" space = " " if sep is None: sep = space if end is None: end = newline for i, arg in enumerate(args): if i: write(sep) write(arg) write(end) if sys.version_info[:2] < (3, 3): _print = print_ def print_(*args, **kwargs): fp = kwargs.get("file", sys.stdout) flush = kwargs.pop("flush", False) _print(*args, **kwargs) if flush and fp is not None: fp.flush() _add_doc(reraise, """Reraise an exception.""") if sys.version_info[0:2] < (3, 4): def wraps(wrapped, assigned=functools.WRAPPER_ASSIGNMENTS, updated=functools.WRAPPER_UPDATES): def wrapper(f): f = functools.wraps(wrapped, assigned, updated)(f) f.__wrapped__ = wrapped return f return wrapper else: wraps = functools.wraps def with_metaclass(meta, *bases): """Create a base class with a metaclass.""" # This requires a bit of explanation: the basic idea is to make a dummy # metaclass for one level of class instantiation that replaces itself with # the actual metaclass. class metaclass(meta): def __new__(cls, name, this_bases, d): return meta(name, bases, d) return type.__new__(metaclass, 'temporary_class', (), {}) def add_metaclass(metaclass): """Class decorator for creating a class with a metaclass.""" def wrapper(cls): orig_vars = cls.__dict__.copy() slots = orig_vars.get('__slots__') if slots is not None: if isinstance(slots, str): slots = [slots] for slots_var in slots: orig_vars.pop(slots_var) orig_vars.pop('__dict__', None) orig_vars.pop('__weakref__', None) return metaclass(cls.__name__, cls.__bases__, orig_vars) return wrapper def python_2_unicode_compatible(klass): """ A decorator that defines __unicode__ and __str__ methods under Python 2. Under Python 3 it does nothing. To support Python 2 and 3 with a single code base, define a __str__ method returning text and apply this decorator to the class. """ if PY2: if '__str__' not in klass.__dict__: raise ValueError("@python_2_unicode_compatible cannot be applied " "to %s because it doesn't define __str__()." % klass.__name__) klass.__unicode__ = klass.__str__ klass.__str__ = lambda self: self.__unicode__().encode('utf-8') return klass # Complete the moves implementation. # This code is at the end of this module to speed up module loading. # Turn this module into a package. __path__ = [] # required for PEP 302 and PEP 451 __package__ = __name__ # see PEP 366 @ReservedAssignment if globals().get("__spec__") is not None: __spec__.submodule_search_locations = [] # PEP 451 @UndefinedVariable # Remove other six meta path importers, since they cause problems. This can # happen if six is removed from sys.modules and then reloaded. (Setuptools does # this for some reason.) if sys.meta_path: for i, importer in enumerate(sys.meta_path): # Here's some real nastiness: Another "instance" of the six module might # be floating around. Therefore, we can't use isinstance() to check for # the six meta path importer, since the other six instance will have # inserted an importer with different class. if (type(importer).__name__ == "_SixMetaPathImporter" and importer.name == __name__): del sys.meta_path[i] break del i, importer # Finally, add the importer to the meta path import hook. sys.meta_path.append(_importer) kafka-1.3.2/kafka/vendor/socketpair.py0000644001271300127130000000410513025302127017443 0ustar dpowers00000000000000# pylint: skip-file # vendored from https://github.com/mhils/backports.socketpair from __future__ import absolute_import import sys import socket import errno _LOCALHOST = '127.0.0.1' _LOCALHOST_V6 = '::1' if not hasattr(socket, "socketpair"): # Origin: https://gist.github.com/4325783, by Geert Jansen. Public domain. def socketpair(family=socket.AF_INET, type=socket.SOCK_STREAM, proto=0): if family == socket.AF_INET: host = _LOCALHOST elif family == socket.AF_INET6: host = _LOCALHOST_V6 else: raise ValueError("Only AF_INET and AF_INET6 socket address families " "are supported") if type != socket.SOCK_STREAM: raise ValueError("Only SOCK_STREAM socket type is supported") if proto != 0: raise ValueError("Only protocol zero is supported") # We create a connected TCP socket. Note the trick with # setblocking(False) that prevents us from having to create a thread. lsock = socket.socket(family, type, proto) try: lsock.bind((host, 0)) lsock.listen(min(socket.SOMAXCONN, 128)) # On IPv6, ignore flow_info and scope_id addr, port = lsock.getsockname()[:2] csock = socket.socket(family, type, proto) try: csock.setblocking(False) if sys.version_info >= (3, 0): try: csock.connect((addr, port)) except (BlockingIOError, InterruptedError): pass else: try: csock.connect((addr, port)) except socket.error as e: if e.errno != errno.WSAEWOULDBLOCK: raise csock.setblocking(True) ssock, _ = lsock.accept() except: csock.close() raise finally: lsock.close() return (ssock, csock) socket.socketpair = socketpair kafka-1.3.2/kafka/version.py0000644001271300127130000000002613031057471015473 0ustar dpowers00000000000000__version__ = '1.3.2' kafka-1.3.2/kafka.egg-info/0000755001271300127130000000000013031057517015131 5ustar dpowers00000000000000kafka-1.3.2/kafka.egg-info/dependency_links.txt0000644001271300127130000000000113031057516021176 0ustar dpowers00000000000000 kafka-1.3.2/kafka.egg-info/PKG-INFO0000644001271300127130000001641213031057516016231 0ustar dpowers00000000000000Metadata-Version: 1.1 Name: kafka Version: 1.3.2 Summary: Pure Python client for Apache Kafka Home-page: https://github.com/dpkp/kafka-python Author: Dana Powers Author-email: dana.powers@gmail.com License: Apache License 2.0 Description: Kafka Python client ------------------------ .. image:: https://img.shields.io/badge/kafka-0.10%2C%200.9%2C%200.8.2%2C%200.8.1%2C%200.8-brightgreen.svg :target: https://kafka-python.readthedocs.org/compatibility.html .. image:: https://img.shields.io/pypi/pyversions/kafka.svg :target: https://pypi.python.org/pypi/kafka .. image:: https://coveralls.io/repos/dpkp/kafka-python/badge.svg?branch=master&service=github :target: https://coveralls.io/github/dpkp/kafka-python?branch=master .. image:: https://travis-ci.org/dpkp/kafka-python.svg?branch=master :target: https://travis-ci.org/dpkp/kafka-python .. image:: https://img.shields.io/badge/license-Apache%202-blue.svg :target: https://github.com/dpkp/kafka-python/blob/master/LICENSE Python client for the Apache Kafka distributed stream processing system. kafka-python is designed to function much like the official java client, with a sprinkling of pythonic interfaces (e.g., consumer iterators). kafka-python is best used with newer brokers (0.10 or 0.9), but is backwards-compatible with older versions (to 0.8.0). Some features will only be enabled on newer brokers, however; for example, fully coordinated consumer groups -- i.e., dynamic partition assignment to multiple consumers in the same group -- requires use of 0.9+ kafka brokers. Supporting this feature for earlier broker releases would require writing and maintaining custom leadership election and membership / health check code (perhaps using zookeeper or consul). For older brokers, you can achieve something similar by manually assigning different partitions to each consumer instance with config management tools like chef, ansible, etc. This approach will work fine, though it does not support rebalancing on failures. See for more details. Please note that the master branch may contain unreleased features. For release documentation, please see readthedocs and/or python's inline help. >>> pip install kafka KafkaConsumer ************* KafkaConsumer is a high-level message consumer, intended to operate as similarly as possible to the official java client. Full support for coordinated consumer groups requires use of kafka brokers that support the Group APIs: kafka v0.9+. See for API and configuration details. The consumer iterator returns ConsumerRecords, which are simple namedtuples that expose basic message attributes: topic, partition, offset, key, and value: >>> from kafka import KafkaConsumer >>> consumer = KafkaConsumer('my_favorite_topic') >>> for msg in consumer: ... print (msg) >>> # manually assign the partition list for the consumer >>> from kafka import TopicPartition >>> consumer = KafkaConsumer(bootstrap_servers='localhost:1234') >>> consumer.assign([TopicPartition('foobar', 2)]) >>> msg = next(consumer) >>> # Deserialize msgpack-encoded values >>> consumer = KafkaConsumer(value_deserializer=msgpack.loads) >>> consumer.subscribe(['msgpackfoo']) >>> for msg in consumer: ... assert isinstance(msg.value, dict) KafkaProducer ************* KafkaProducer is a high-level, asynchronous message producer. The class is intended to operate as similarly as possible to the official java client. See for more details. >>> from kafka import KafkaProducer >>> producer = KafkaProducer(bootstrap_servers='localhost:1234') >>> for _ in range(100): ... producer.send('foobar', b'some_message_bytes') >>> # Block until all pending messages are sent >>> producer.flush() >>> # Block until a single message is sent (or timeout) >>> producer.send('foobar', b'another_message').get(timeout=60) >>> # Use a key for hashed-partitioning >>> producer.send('foobar', key=b'foo', value=b'bar') >>> # Serialize json messages >>> import json >>> producer = KafkaProducer(value_serializer=lambda v: json.dumps(v).encode('utf-8')) >>> producer.send('fizzbuzz', {'foo': 'bar'}) >>> # Serialize string keys >>> producer = KafkaProducer(key_serializer=str.encode) >>> producer.send('flipflap', key='ping', value=b'1234') >>> # Compress messages >>> producer = KafkaProducer(compression_type='gzip') >>> for i in range(1000): ... producer.send('foobar', b'msg %d' % i) Compression *********** kafka-python supports gzip compression/decompression natively. To produce or consume lz4 compressed messages, you must install lz4tools and xxhash (modules may not work on python2.6). To enable snappy compression/decompression install python-snappy (also requires snappy library). See for more information. Protocol ******** A secondary goal of kafka-python is to provide an easy-to-use protocol layer for interacting with kafka brokers via the python repl. This is useful for testing, probing, and general experimentation. The protocol support is leveraged to enable a KafkaClient.check_version() method that probes a kafka broker and attempts to identify which version it is running (0.8.0 to 0.10). Low-level ********* Legacy support is maintained for low-level consumer and producer classes, SimpleConsumer and SimpleProducer. See for API details. Keywords: apache kafka Platform: UNKNOWN Classifier: Development Status :: 5 - Production/Stable Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: Apache Software License Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2 Classifier: Programming Language :: Python :: 2.6 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: 3.4 Classifier: Programming Language :: Python :: 3.5 Classifier: Programming Language :: Python :: Implementation :: PyPy Classifier: Topic :: Software Development :: Libraries :: Python Modules kafka-1.3.2/kafka.egg-info/SOURCES.txt0000644001271300127130000000610013031057517017012 0ustar dpowers00000000000000AUTHORS.md CHANGES.md LICENSE MANIFEST.in README.rst setup.cfg setup.py kafka/__init__.py kafka/client.py kafka/client_async.py kafka/cluster.py kafka/codec.py kafka/common.py kafka/conn.py kafka/context.py kafka/errors.py kafka/future.py kafka/structs.py kafka/util.py kafka/version.py kafka.egg-info/PKG-INFO kafka.egg-info/SOURCES.txt kafka.egg-info/dependency_links.txt kafka.egg-info/top_level.txt kafka/consumer/__init__.py kafka/consumer/base.py kafka/consumer/fetcher.py kafka/consumer/group.py kafka/consumer/multiprocess.py kafka/consumer/simple.py kafka/consumer/subscription_state.py kafka/coordinator/__init__.py kafka/coordinator/base.py kafka/coordinator/consumer.py kafka/coordinator/heartbeat.py kafka/coordinator/protocol.py kafka/coordinator/assignors/__init__.py kafka/coordinator/assignors/abstract.py kafka/coordinator/assignors/range.py kafka/coordinator/assignors/roundrobin.py kafka/metrics/__init__.py kafka/metrics/compound_stat.py kafka/metrics/dict_reporter.py kafka/metrics/kafka_metric.py kafka/metrics/measurable.py kafka/metrics/measurable_stat.py kafka/metrics/metric_config.py kafka/metrics/metric_name.py kafka/metrics/metrics.py kafka/metrics/metrics_reporter.py kafka/metrics/quota.py kafka/metrics/stat.py kafka/metrics/stats/__init__.py kafka/metrics/stats/avg.py kafka/metrics/stats/count.py kafka/metrics/stats/histogram.py kafka/metrics/stats/max_stat.py kafka/metrics/stats/min_stat.py kafka/metrics/stats/percentile.py kafka/metrics/stats/percentiles.py kafka/metrics/stats/rate.py kafka/metrics/stats/sampled_stat.py kafka/metrics/stats/sensor.py kafka/metrics/stats/total.py kafka/partitioner/__init__.py kafka/partitioner/base.py kafka/partitioner/default.py kafka/partitioner/hashed.py kafka/partitioner/roundrobin.py kafka/producer/__init__.py kafka/producer/base.py kafka/producer/buffer.py kafka/producer/future.py kafka/producer/kafka.py kafka/producer/keyed.py kafka/producer/record_accumulator.py kafka/producer/sender.py kafka/producer/simple.py kafka/protocol/__init__.py kafka/protocol/abstract.py kafka/protocol/admin.py kafka/protocol/api.py kafka/protocol/commit.py kafka/protocol/fetch.py kafka/protocol/group.py kafka/protocol/legacy.py kafka/protocol/message.py kafka/protocol/metadata.py kafka/protocol/offset.py kafka/protocol/pickle.py kafka/protocol/produce.py kafka/protocol/struct.py kafka/protocol/types.py kafka/serializer/__init__.py kafka/serializer/abstract.py kafka/vendor/__init__.py kafka/vendor/selectors34.py kafka/vendor/six.py kafka/vendor/socketpair.py test/test_assignors.py test/test_buffer.py test/test_client.py test/test_client_async.py test/test_client_integration.py test/test_codec.py test/test_conn.py test/test_consumer.py test/test_consumer_group.py test/test_consumer_integration.py test/test_context.py test/test_coordinator.py test/test_failover_integration.py test/test_fetcher.py test/test_metrics.py test/test_package.py test/test_partitioner.py test/test_producer.py test/test_producer_integration.py test/test_producer_legacy.py test/test_protocol.py test/test_protocol_legacy.py test/test_sender.py test/test_util.py test/testutil.pykafka-1.3.2/kafka.egg-info/top_level.txt0000644001271300127130000000000613031057516017656 0ustar dpowers00000000000000kafka kafka-1.3.2/LICENSE0000644001271300127130000002611712650776450013410 0ustar dpowers00000000000000 Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright 2015 David Arthur Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. kafka-1.3.2/MANIFEST.in0000644001271300127130000000014612650776450014133 0ustar dpowers00000000000000recursive-include kafka *.py include README.rst include LICENSE include AUTHORS.md include CHANGES.md kafka-1.3.2/PKG-INFO0000644001271300127130000001641213031057517013463 0ustar dpowers00000000000000Metadata-Version: 1.1 Name: kafka Version: 1.3.2 Summary: Pure Python client for Apache Kafka Home-page: https://github.com/dpkp/kafka-python Author: Dana Powers Author-email: dana.powers@gmail.com License: Apache License 2.0 Description: Kafka Python client ------------------------ .. image:: https://img.shields.io/badge/kafka-0.10%2C%200.9%2C%200.8.2%2C%200.8.1%2C%200.8-brightgreen.svg :target: https://kafka-python.readthedocs.org/compatibility.html .. image:: https://img.shields.io/pypi/pyversions/kafka.svg :target: https://pypi.python.org/pypi/kafka .. image:: https://coveralls.io/repos/dpkp/kafka-python/badge.svg?branch=master&service=github :target: https://coveralls.io/github/dpkp/kafka-python?branch=master .. image:: https://travis-ci.org/dpkp/kafka-python.svg?branch=master :target: https://travis-ci.org/dpkp/kafka-python .. image:: https://img.shields.io/badge/license-Apache%202-blue.svg :target: https://github.com/dpkp/kafka-python/blob/master/LICENSE Python client for the Apache Kafka distributed stream processing system. kafka-python is designed to function much like the official java client, with a sprinkling of pythonic interfaces (e.g., consumer iterators). kafka-python is best used with newer brokers (0.10 or 0.9), but is backwards-compatible with older versions (to 0.8.0). Some features will only be enabled on newer brokers, however; for example, fully coordinated consumer groups -- i.e., dynamic partition assignment to multiple consumers in the same group -- requires use of 0.9+ kafka brokers. Supporting this feature for earlier broker releases would require writing and maintaining custom leadership election and membership / health check code (perhaps using zookeeper or consul). For older brokers, you can achieve something similar by manually assigning different partitions to each consumer instance with config management tools like chef, ansible, etc. This approach will work fine, though it does not support rebalancing on failures. See for more details. Please note that the master branch may contain unreleased features. For release documentation, please see readthedocs and/or python's inline help. >>> pip install kafka KafkaConsumer ************* KafkaConsumer is a high-level message consumer, intended to operate as similarly as possible to the official java client. Full support for coordinated consumer groups requires use of kafka brokers that support the Group APIs: kafka v0.9+. See for API and configuration details. The consumer iterator returns ConsumerRecords, which are simple namedtuples that expose basic message attributes: topic, partition, offset, key, and value: >>> from kafka import KafkaConsumer >>> consumer = KafkaConsumer('my_favorite_topic') >>> for msg in consumer: ... print (msg) >>> # manually assign the partition list for the consumer >>> from kafka import TopicPartition >>> consumer = KafkaConsumer(bootstrap_servers='localhost:1234') >>> consumer.assign([TopicPartition('foobar', 2)]) >>> msg = next(consumer) >>> # Deserialize msgpack-encoded values >>> consumer = KafkaConsumer(value_deserializer=msgpack.loads) >>> consumer.subscribe(['msgpackfoo']) >>> for msg in consumer: ... assert isinstance(msg.value, dict) KafkaProducer ************* KafkaProducer is a high-level, asynchronous message producer. The class is intended to operate as similarly as possible to the official java client. See for more details. >>> from kafka import KafkaProducer >>> producer = KafkaProducer(bootstrap_servers='localhost:1234') >>> for _ in range(100): ... producer.send('foobar', b'some_message_bytes') >>> # Block until all pending messages are sent >>> producer.flush() >>> # Block until a single message is sent (or timeout) >>> producer.send('foobar', b'another_message').get(timeout=60) >>> # Use a key for hashed-partitioning >>> producer.send('foobar', key=b'foo', value=b'bar') >>> # Serialize json messages >>> import json >>> producer = KafkaProducer(value_serializer=lambda v: json.dumps(v).encode('utf-8')) >>> producer.send('fizzbuzz', {'foo': 'bar'}) >>> # Serialize string keys >>> producer = KafkaProducer(key_serializer=str.encode) >>> producer.send('flipflap', key='ping', value=b'1234') >>> # Compress messages >>> producer = KafkaProducer(compression_type='gzip') >>> for i in range(1000): ... producer.send('foobar', b'msg %d' % i) Compression *********** kafka-python supports gzip compression/decompression natively. To produce or consume lz4 compressed messages, you must install lz4tools and xxhash (modules may not work on python2.6). To enable snappy compression/decompression install python-snappy (also requires snappy library). See for more information. Protocol ******** A secondary goal of kafka-python is to provide an easy-to-use protocol layer for interacting with kafka brokers via the python repl. This is useful for testing, probing, and general experimentation. The protocol support is leveraged to enable a KafkaClient.check_version() method that probes a kafka broker and attempts to identify which version it is running (0.8.0 to 0.10). Low-level ********* Legacy support is maintained for low-level consumer and producer classes, SimpleConsumer and SimpleProducer. See for API details. Keywords: apache kafka Platform: UNKNOWN Classifier: Development Status :: 5 - Production/Stable Classifier: Intended Audience :: Developers Classifier: License :: OSI Approved :: Apache Software License Classifier: Programming Language :: Python Classifier: Programming Language :: Python :: 2 Classifier: Programming Language :: Python :: 2.6 Classifier: Programming Language :: Python :: 2.7 Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.3 Classifier: Programming Language :: Python :: 3.4 Classifier: Programming Language :: Python :: 3.5 Classifier: Programming Language :: Python :: Implementation :: PyPy Classifier: Topic :: Software Development :: Libraries :: Python Modules kafka-1.3.2/README.rst0000644001271300127130000001245313031057472014056 0ustar dpowers00000000000000Kafka Python client ------------------------ .. image:: https://img.shields.io/badge/kafka-0.10%2C%200.9%2C%200.8.2%2C%200.8.1%2C%200.8-brightgreen.svg :target: https://kafka-python.readthedocs.org/compatibility.html .. image:: https://img.shields.io/pypi/pyversions/kafka.svg :target: https://pypi.python.org/pypi/kafka .. image:: https://coveralls.io/repos/dpkp/kafka-python/badge.svg?branch=master&service=github :target: https://coveralls.io/github/dpkp/kafka-python?branch=master .. image:: https://travis-ci.org/dpkp/kafka-python.svg?branch=master :target: https://travis-ci.org/dpkp/kafka-python .. image:: https://img.shields.io/badge/license-Apache%202-blue.svg :target: https://github.com/dpkp/kafka-python/blob/master/LICENSE Python client for the Apache Kafka distributed stream processing system. kafka-python is designed to function much like the official java client, with a sprinkling of pythonic interfaces (e.g., consumer iterators). kafka-python is best used with newer brokers (0.10 or 0.9), but is backwards-compatible with older versions (to 0.8.0). Some features will only be enabled on newer brokers, however; for example, fully coordinated consumer groups -- i.e., dynamic partition assignment to multiple consumers in the same group -- requires use of 0.9+ kafka brokers. Supporting this feature for earlier broker releases would require writing and maintaining custom leadership election and membership / health check code (perhaps using zookeeper or consul). For older brokers, you can achieve something similar by manually assigning different partitions to each consumer instance with config management tools like chef, ansible, etc. This approach will work fine, though it does not support rebalancing on failures. See for more details. Please note that the master branch may contain unreleased features. For release documentation, please see readthedocs and/or python's inline help. >>> pip install kafka KafkaConsumer ************* KafkaConsumer is a high-level message consumer, intended to operate as similarly as possible to the official java client. Full support for coordinated consumer groups requires use of kafka brokers that support the Group APIs: kafka v0.9+. See for API and configuration details. The consumer iterator returns ConsumerRecords, which are simple namedtuples that expose basic message attributes: topic, partition, offset, key, and value: >>> from kafka import KafkaConsumer >>> consumer = KafkaConsumer('my_favorite_topic') >>> for msg in consumer: ... print (msg) >>> # manually assign the partition list for the consumer >>> from kafka import TopicPartition >>> consumer = KafkaConsumer(bootstrap_servers='localhost:1234') >>> consumer.assign([TopicPartition('foobar', 2)]) >>> msg = next(consumer) >>> # Deserialize msgpack-encoded values >>> consumer = KafkaConsumer(value_deserializer=msgpack.loads) >>> consumer.subscribe(['msgpackfoo']) >>> for msg in consumer: ... assert isinstance(msg.value, dict) KafkaProducer ************* KafkaProducer is a high-level, asynchronous message producer. The class is intended to operate as similarly as possible to the official java client. See for more details. >>> from kafka import KafkaProducer >>> producer = KafkaProducer(bootstrap_servers='localhost:1234') >>> for _ in range(100): ... producer.send('foobar', b'some_message_bytes') >>> # Block until all pending messages are sent >>> producer.flush() >>> # Block until a single message is sent (or timeout) >>> producer.send('foobar', b'another_message').get(timeout=60) >>> # Use a key for hashed-partitioning >>> producer.send('foobar', key=b'foo', value=b'bar') >>> # Serialize json messages >>> import json >>> producer = KafkaProducer(value_serializer=lambda v: json.dumps(v).encode('utf-8')) >>> producer.send('fizzbuzz', {'foo': 'bar'}) >>> # Serialize string keys >>> producer = KafkaProducer(key_serializer=str.encode) >>> producer.send('flipflap', key='ping', value=b'1234') >>> # Compress messages >>> producer = KafkaProducer(compression_type='gzip') >>> for i in range(1000): ... producer.send('foobar', b'msg %d' % i) Compression *********** kafka-python supports gzip compression/decompression natively. To produce or consume lz4 compressed messages, you must install lz4tools and xxhash (modules may not work on python2.6). To enable snappy compression/decompression install python-snappy (also requires snappy library). See for more information. Protocol ******** A secondary goal of kafka-python is to provide an easy-to-use protocol layer for interacting with kafka brokers via the python repl. This is useful for testing, probing, and general experimentation. The protocol support is leveraged to enable a KafkaClient.check_version() method that probes a kafka broker and attempts to identify which version it is running (0.8.0 to 0.10). Low-level ********* Legacy support is maintained for low-level consumer and producer classes, SimpleConsumer and SimpleProducer. See for API details. kafka-1.3.2/setup.cfg0000644001271300127130000000013013031057517014175 0ustar dpowers00000000000000[bdist_wheel] universal = 1 [egg_info] tag_build = tag_date = 0 tag_svn_revision = 0 kafka-1.3.2/setup.py0000644001271300127130000000340313031057472014074 0ustar dpowers00000000000000import sys import os from setuptools import setup, Command, find_packages # Pull version from source without importing # since we can't import something we haven't built yet :) exec(open('kafka/version.py').read()) class Tox(Command): user_options = [] def initialize_options(self): pass def finalize_options(self): pass @classmethod def run(cls): import tox sys.exit(tox.cmdline([])) test_require = ['tox', 'mock'] if sys.version_info < (2, 7): test_require.append('unittest2') here = os.path.abspath(os.path.dirname(__file__)) with open(os.path.join(here, 'README.rst')) as f: README = f.read() setup( name="kafka", version=__version__, tests_require=test_require, cmdclass={"test": Tox}, packages=find_packages(exclude=['test']), author="Dana Powers", author_email="dana.powers@gmail.com", url="https://github.com/dpkp/kafka-python", license="Apache License 2.0", description="Pure Python client for Apache Kafka", long_description=README, keywords="apache kafka", classifiers=[ "Development Status :: 5 - Production/Stable", "Intended Audience :: Developers", "License :: OSI Approved :: Apache Software License", "Programming Language :: Python", "Programming Language :: Python :: 2", "Programming Language :: Python :: 2.6", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.3", "Programming Language :: Python :: 3.4", "Programming Language :: Python :: 3.5", "Programming Language :: Python :: Implementation :: PyPy", "Topic :: Software Development :: Libraries :: Python Modules", ] ) kafka-1.3.2/test/0000755001271300127130000000000013031057517013341 5ustar dpowers00000000000000kafka-1.3.2/test/test_assignors.py0000644001271300127130000000344612700557363016777 0ustar dpowers00000000000000# pylint: skip-file from __future__ import absolute_import import pytest from kafka.coordinator.assignors.range import RangePartitionAssignor from kafka.coordinator.assignors.roundrobin import RoundRobinPartitionAssignor from kafka.coordinator.protocol import ( ConsumerProtocolMemberMetadata, ConsumerProtocolMemberAssignment) @pytest.fixture def cluster(mocker): cluster = mocker.MagicMock() cluster.partitions_for_topic.return_value = set([0, 1, 2]) return cluster def test_assignor_roundrobin(cluster): assignor = RoundRobinPartitionAssignor member_metadata = { 'C0': assignor.metadata(set(['t0', 't1'])), 'C1': assignor.metadata(set(['t0', 't1'])), } ret = assignor.assign(cluster, member_metadata) expected = { 'C0': ConsumerProtocolMemberAssignment( assignor.version, [('t0', [0, 2]), ('t1', [1])], b''), 'C1': ConsumerProtocolMemberAssignment( assignor.version, [('t0', [1]), ('t1', [0, 2])], b'') } assert ret == expected assert set(ret) == set(expected) for member in ret: assert ret[member].encode() == expected[member].encode() def test_assignor_range(cluster): assignor = RangePartitionAssignor member_metadata = { 'C0': assignor.metadata(set(['t0', 't1'])), 'C1': assignor.metadata(set(['t0', 't1'])), } ret = assignor.assign(cluster, member_metadata) expected = { 'C0': ConsumerProtocolMemberAssignment( assignor.version, [('t0', [0, 1]), ('t1', [0, 1])], b''), 'C1': ConsumerProtocolMemberAssignment( assignor.version, [('t0', [2]), ('t1', [2])], b'') } assert ret == expected assert set(ret) == set(expected) for member in ret: assert ret[member].encode() == expected[member].encode() kafka-1.3.2/test/test_buffer.py0000644001271300127130000000346213025302127016221 0ustar dpowers00000000000000# pylint: skip-file from __future__ import absolute_import import io import pytest from kafka.producer.buffer import MessageSetBuffer from kafka.protocol.message import Message, MessageSet def test_buffer_close(): records = MessageSetBuffer(io.BytesIO(), 100000) orig_msg = Message(b'foobar') records.append(1234, orig_msg) records.close() msgset = MessageSet.decode(records.buffer()) assert len(msgset) == 1 (offset, size, msg) = msgset[0] assert offset == 1234 assert msg == orig_msg # Closing again should work fine records.close() msgset = MessageSet.decode(records.buffer()) assert len(msgset) == 1 (offset, size, msg) = msgset[0] assert offset == 1234 assert msg == orig_msg @pytest.mark.parametrize('compression', [ 'gzip', 'snappy', pytest.mark.skipif("sys.version_info < (2,7)")('lz4'), # lz4tools does not work on py26 ]) def test_compressed_buffer_close(compression): records = MessageSetBuffer(io.BytesIO(), 100000, compression_type=compression) orig_msg = Message(b'foobar') records.append(1234, orig_msg) records.close() msgset = MessageSet.decode(records.buffer()) assert len(msgset) == 1 (offset, size, msg) = msgset[0] assert offset == 0 assert msg.is_compressed() msgset = msg.decompress() (offset, size, msg) = msgset[0] assert not msg.is_compressed() assert offset == 1234 assert msg == orig_msg # Closing again should work fine records.close() msgset = MessageSet.decode(records.buffer()) assert len(msgset) == 1 (offset, size, msg) = msgset[0] assert offset == 0 assert msg.is_compressed() msgset = msg.decompress() (offset, size, msg) = msgset[0] assert not msg.is_compressed() assert offset == 1234 assert msg == orig_msg kafka-1.3.2/test/test_client.py0000644001271300127130000003635113025302127016231 0ustar dpowers00000000000000import socket from mock import ANY, MagicMock, patch from operator import itemgetter import six from . import unittest from kafka import SimpleClient from kafka.errors import ( KafkaUnavailableError, LeaderNotAvailableError, KafkaTimeoutError, UnknownTopicOrPartitionError, ConnectionError, FailedPayloadsError) from kafka.future import Future from kafka.protocol import KafkaProtocol, create_message from kafka.protocol.metadata import MetadataResponse from kafka.structs import ProduceRequestPayload, BrokerMetadata, TopicPartition NO_ERROR = 0 UNKNOWN_TOPIC_OR_PARTITION = 3 NO_LEADER = 5 def mock_conn(conn, success=True): mocked = MagicMock() mocked.connected.return_value = True if success: mocked.send.return_value = Future().success(True) else: mocked.send.return_value = Future().failure(Exception()) conn.return_value = mocked class TestSimpleClient(unittest.TestCase): def test_init_with_list(self): with patch.object(SimpleClient, 'load_metadata_for_topics'): client = SimpleClient(hosts=['kafka01:9092', 'kafka02:9092', 'kafka03:9092']) self.assertEqual( sorted([('kafka01', 9092, socket.AF_UNSPEC), ('kafka02', 9092, socket.AF_UNSPEC), ('kafka03', 9092, socket.AF_UNSPEC)]), sorted(client.hosts)) def test_init_with_csv(self): with patch.object(SimpleClient, 'load_metadata_for_topics'): client = SimpleClient(hosts='kafka01:9092,kafka02:9092,kafka03:9092') self.assertEqual( sorted([('kafka01', 9092, socket.AF_UNSPEC), ('kafka02', 9092, socket.AF_UNSPEC), ('kafka03', 9092, socket.AF_UNSPEC)]), sorted(client.hosts)) def test_init_with_unicode_csv(self): with patch.object(SimpleClient, 'load_metadata_for_topics'): client = SimpleClient(hosts=u'kafka01:9092,kafka02:9092,kafka03:9092') self.assertEqual( sorted([('kafka01', 9092, socket.AF_UNSPEC), ('kafka02', 9092, socket.AF_UNSPEC), ('kafka03', 9092, socket.AF_UNSPEC)]), sorted(client.hosts)) @patch.object(SimpleClient, '_get_conn') @patch.object(SimpleClient, 'load_metadata_for_topics') def test_send_broker_unaware_request_fail(self, load_metadata, conn): mocked_conns = { ('kafka01', 9092): MagicMock(), ('kafka02', 9092): MagicMock() } for val in mocked_conns.values(): mock_conn(val, success=False) def mock_get_conn(host, port, afi): return mocked_conns[(host, port)] conn.side_effect = mock_get_conn client = SimpleClient(hosts=['kafka01:9092', 'kafka02:9092']) req = KafkaProtocol.encode_metadata_request() with self.assertRaises(KafkaUnavailableError): client._send_broker_unaware_request(payloads=['fake request'], encoder_fn=MagicMock(return_value='fake encoded message'), decoder_fn=lambda x: x) for key, conn in six.iteritems(mocked_conns): conn.send.assert_called_with('fake encoded message') def test_send_broker_unaware_request(self): mocked_conns = { ('kafka01', 9092): MagicMock(), ('kafka02', 9092): MagicMock(), ('kafka03', 9092): MagicMock() } # inject BrokerConnection side effects mock_conn(mocked_conns[('kafka01', 9092)], success=False) mock_conn(mocked_conns[('kafka03', 9092)], success=False) future = Future() mocked_conns[('kafka02', 9092)].send.return_value = future mocked_conns[('kafka02', 9092)].recv.side_effect = lambda: future.success('valid response') def mock_get_conn(host, port, afi): return mocked_conns[(host, port)] # patch to avoid making requests before we want it with patch.object(SimpleClient, 'load_metadata_for_topics'): with patch.object(SimpleClient, '_get_conn', side_effect=mock_get_conn): client = SimpleClient(hosts='kafka01:9092,kafka02:9092') resp = client._send_broker_unaware_request(payloads=['fake request'], encoder_fn=MagicMock(), decoder_fn=lambda x: x) self.assertEqual('valid response', resp) mocked_conns[('kafka02', 9092)].recv.assert_called_once_with() @patch('kafka.SimpleClient._get_conn') @patch('kafka.client.KafkaProtocol') def test_load_metadata(self, protocol, conn): mock_conn(conn) brokers = [ BrokerMetadata(0, 'broker_1', 4567, None), BrokerMetadata(1, 'broker_2', 5678, None) ] resp0_brokers = list(map(itemgetter(0, 1, 2), brokers)) topics = [ (NO_ERROR, 'topic_1', [ (NO_ERROR, 0, 1, [1, 2], [1, 2]) ]), (NO_ERROR, 'topic_noleader', [ (NO_LEADER, 0, -1, [], []), (NO_LEADER, 1, -1, [], []), ]), (NO_LEADER, 'topic_no_partitions', []), (UNKNOWN_TOPIC_OR_PARTITION, 'topic_unknown', []), (NO_ERROR, 'topic_3', [ (NO_ERROR, 0, 0, [0, 1], [0, 1]), (NO_ERROR, 1, 1, [1, 0], [1, 0]), (NO_ERROR, 2, 0, [0, 1], [0, 1]) ]) ] protocol.decode_metadata_response.return_value = MetadataResponse[0](resp0_brokers, topics) # client loads metadata at init client = SimpleClient(hosts=['broker_1:4567']) self.assertDictEqual({ TopicPartition('topic_1', 0): brokers[1], TopicPartition('topic_noleader', 0): None, TopicPartition('topic_noleader', 1): None, TopicPartition('topic_3', 0): brokers[0], TopicPartition('topic_3', 1): brokers[1], TopicPartition('topic_3', 2): brokers[0]}, client.topics_to_brokers) # if we ask for metadata explicitly, it should raise errors with self.assertRaises(LeaderNotAvailableError): client.load_metadata_for_topics('topic_no_partitions') with self.assertRaises(UnknownTopicOrPartitionError): client.load_metadata_for_topics('topic_unknown') # This should not raise client.load_metadata_for_topics('topic_no_leader') @patch('kafka.SimpleClient._get_conn') @patch('kafka.client.KafkaProtocol') def test_has_metadata_for_topic(self, protocol, conn): mock_conn(conn) brokers = [ BrokerMetadata(0, 'broker_1', 4567, None), BrokerMetadata(1, 'broker_2', 5678, None) ] resp0_brokers = list(map(itemgetter(0, 1, 2), brokers)) topics = [ (NO_LEADER, 'topic_still_creating', []), (UNKNOWN_TOPIC_OR_PARTITION, 'topic_doesnt_exist', []), (NO_ERROR, 'topic_noleaders', [ (NO_LEADER, 0, -1, [], []), (NO_LEADER, 1, -1, [], []), ]), ] protocol.decode_metadata_response.return_value = MetadataResponse[0](resp0_brokers, topics) client = SimpleClient(hosts=['broker_1:4567']) # Topics with no partitions return False self.assertFalse(client.has_metadata_for_topic('topic_still_creating')) self.assertFalse(client.has_metadata_for_topic('topic_doesnt_exist')) # Topic with partition metadata, but no leaders return True self.assertTrue(client.has_metadata_for_topic('topic_noleaders')) @patch('kafka.SimpleClient._get_conn') @patch('kafka.client.KafkaProtocol.decode_metadata_response') def test_ensure_topic_exists(self, decode_metadata_response, conn): mock_conn(conn) brokers = [ BrokerMetadata(0, 'broker_1', 4567, None), BrokerMetadata(1, 'broker_2', 5678, None) ] resp0_brokers = list(map(itemgetter(0, 1, 2), brokers)) topics = [ (NO_LEADER, 'topic_still_creating', []), (UNKNOWN_TOPIC_OR_PARTITION, 'topic_doesnt_exist', []), (NO_ERROR, 'topic_noleaders', [ (NO_LEADER, 0, -1, [], []), (NO_LEADER, 1, -1, [], []), ]), ] decode_metadata_response.return_value = MetadataResponse[0](resp0_brokers, topics) client = SimpleClient(hosts=['broker_1:4567']) with self.assertRaises(UnknownTopicOrPartitionError): client.ensure_topic_exists('topic_doesnt_exist', timeout=1) with self.assertRaises(KafkaTimeoutError): client.ensure_topic_exists('topic_still_creating', timeout=1) # This should not raise client.ensure_topic_exists('topic_noleaders', timeout=1) @patch('kafka.SimpleClient._get_conn') @patch('kafka.client.KafkaProtocol') def test_get_leader_for_partitions_reloads_metadata(self, protocol, conn): "Get leader for partitions reload metadata if it is not available" mock_conn(conn) brokers = [ BrokerMetadata(0, 'broker_1', 4567, None), BrokerMetadata(1, 'broker_2', 5678, None) ] resp0_brokers = list(map(itemgetter(0, 1, 2), brokers)) topics = [ (NO_LEADER, 'topic_no_partitions', []) ] protocol.decode_metadata_response.return_value = MetadataResponse[0](resp0_brokers, topics) client = SimpleClient(hosts=['broker_1:4567']) # topic metadata is loaded but empty self.assertDictEqual({}, client.topics_to_brokers) topics = [ (NO_ERROR, 'topic_one_partition', [ (NO_ERROR, 0, 0, [0, 1], [0, 1]) ]) ] protocol.decode_metadata_response.return_value = MetadataResponse[0](resp0_brokers, topics) # calling _get_leader_for_partition (from any broker aware request) # will try loading metadata again for the same topic leader = client._get_leader_for_partition('topic_one_partition', 0) self.assertEqual(brokers[0], leader) self.assertDictEqual({ TopicPartition('topic_one_partition', 0): brokers[0]}, client.topics_to_brokers) @patch('kafka.SimpleClient._get_conn') @patch('kafka.client.KafkaProtocol') def test_get_leader_for_unassigned_partitions(self, protocol, conn): mock_conn(conn) brokers = [ BrokerMetadata(0, 'broker_1', 4567, None), BrokerMetadata(1, 'broker_2', 5678, None) ] resp0_brokers = list(map(itemgetter(0, 1, 2), brokers)) topics = [ (NO_LEADER, 'topic_no_partitions', []), (UNKNOWN_TOPIC_OR_PARTITION, 'topic_unknown', []), ] protocol.decode_metadata_response.return_value = MetadataResponse[0](resp0_brokers, topics) client = SimpleClient(hosts=['broker_1:4567']) self.assertDictEqual({}, client.topics_to_brokers) with self.assertRaises(LeaderNotAvailableError): client._get_leader_for_partition('topic_no_partitions', 0) with self.assertRaises(UnknownTopicOrPartitionError): client._get_leader_for_partition('topic_unknown', 0) @patch('kafka.SimpleClient._get_conn') @patch('kafka.client.KafkaProtocol') def test_get_leader_exceptions_when_noleader(self, protocol, conn): mock_conn(conn) brokers = [ BrokerMetadata(0, 'broker_1', 4567, None), BrokerMetadata(1, 'broker_2', 5678, None) ] resp0_brokers = list(map(itemgetter(0, 1, 2), brokers)) topics = [ (NO_ERROR, 'topic_noleader', [ (NO_LEADER, 0, -1, [], []), (NO_LEADER, 1, -1, [], []), ]), ] protocol.decode_metadata_response.return_value = MetadataResponse[0](resp0_brokers, topics) client = SimpleClient(hosts=['broker_1:4567']) self.assertDictEqual( { TopicPartition('topic_noleader', 0): None, TopicPartition('topic_noleader', 1): None }, client.topics_to_brokers) # No leader partitions -- raise LeaderNotAvailableError with self.assertRaises(LeaderNotAvailableError): self.assertIsNone(client._get_leader_for_partition('topic_noleader', 0)) with self.assertRaises(LeaderNotAvailableError): self.assertIsNone(client._get_leader_for_partition('topic_noleader', 1)) # Unknown partitions -- raise UnknownTopicOrPartitionError with self.assertRaises(UnknownTopicOrPartitionError): self.assertIsNone(client._get_leader_for_partition('topic_noleader', 2)) topics = [ (NO_ERROR, 'topic_noleader', [ (NO_ERROR, 0, 0, [0, 1], [0, 1]), (NO_ERROR, 1, 1, [1, 0], [1, 0]) ]), ] protocol.decode_metadata_response.return_value = MetadataResponse[0](resp0_brokers, topics) self.assertEqual(brokers[0], client._get_leader_for_partition('topic_noleader', 0)) self.assertEqual(brokers[1], client._get_leader_for_partition('topic_noleader', 1)) @patch.object(SimpleClient, '_get_conn') @patch('kafka.client.KafkaProtocol') def test_send_produce_request_raises_when_noleader(self, protocol, conn): mock_conn(conn) brokers = [ BrokerMetadata(0, 'broker_1', 4567, None), BrokerMetadata(1, 'broker_2', 5678, None) ] resp0_brokers = list(map(itemgetter(0, 1, 2), brokers)) topics = [ (NO_ERROR, 'topic_noleader', [ (NO_LEADER, 0, -1, [], []), (NO_LEADER, 1, -1, [], []), ]), ] protocol.decode_metadata_response.return_value = MetadataResponse[0](resp0_brokers, topics) client = SimpleClient(hosts=['broker_1:4567']) requests = [ProduceRequestPayload( "topic_noleader", 0, [create_message("a"), create_message("b")])] with self.assertRaises(FailedPayloadsError): client.send_produce_request(requests) @patch('kafka.SimpleClient._get_conn') @patch('kafka.client.KafkaProtocol') def test_send_produce_request_raises_when_topic_unknown(self, protocol, conn): mock_conn(conn) brokers = [ BrokerMetadata(0, 'broker_1', 4567, None), BrokerMetadata(1, 'broker_2', 5678, None) ] resp0_brokers = list(map(itemgetter(0, 1, 2), brokers)) topics = [ (UNKNOWN_TOPIC_OR_PARTITION, 'topic_doesnt_exist', []), ] protocol.decode_metadata_response.return_value = MetadataResponse[0](resp0_brokers, topics) client = SimpleClient(hosts=['broker_1:4567']) requests = [ProduceRequestPayload( "topic_doesnt_exist", 0, [create_message("a"), create_message("b")])] with self.assertRaises(FailedPayloadsError): client.send_produce_request(requests) def test_correlation_rollover(self): with patch.object(SimpleClient, 'load_metadata_for_topics'): big_num = 2**31 - 3 client = SimpleClient(hosts=[], correlation_id=big_num) self.assertEqual(big_num + 1, client._next_id()) self.assertEqual(big_num + 2, client._next_id()) self.assertEqual(0, client._next_id()) kafka-1.3.2/test/test_client_async.py0000644001271300127130000002672713031057471017442 0ustar dpowers00000000000000# selectors in stdlib as of py3.4 try: import selectors # pylint: disable=import-error except ImportError: # vendored backport module import kafka.vendor.selectors34 as selectors import socket import time import pytest from kafka.client_async import KafkaClient from kafka.conn import ConnectionStates import kafka.errors as Errors from kafka.future import Future from kafka.protocol.metadata import MetadataResponse, MetadataRequest from kafka.protocol.produce import ProduceRequest from kafka.structs import BrokerMetadata from kafka.cluster import ClusterMetadata from kafka.future import Future @pytest.fixture def cli(conn): return KafkaClient(api_version=(0, 9)) @pytest.mark.parametrize("bootstrap,expected_hosts", [ (None, [('localhost', 9092, socket.AF_UNSPEC)]), ('foobar:1234', [('foobar', 1234, socket.AF_UNSPEC)]), ('fizzbuzz', [('fizzbuzz', 9092, socket.AF_UNSPEC)]), ('foo:12,bar:34', [('foo', 12, socket.AF_UNSPEC), ('bar', 34, socket.AF_UNSPEC)]), (['fizz:56', 'buzz'], [('fizz', 56, socket.AF_UNSPEC), ('buzz', 9092, socket.AF_UNSPEC)]), ]) def test_bootstrap_servers(mocker, bootstrap, expected_hosts): mocker.patch.object(KafkaClient, '_bootstrap') if bootstrap is None: KafkaClient(api_version=(0, 9)) # pass api_version to skip auto version checks else: KafkaClient(bootstrap_servers=bootstrap, api_version=(0, 9)) # host order is randomized internally, so resort before testing (hosts,), _ = KafkaClient._bootstrap.call_args # pylint: disable=no-member assert sorted(hosts) == sorted(expected_hosts) def test_bootstrap_success(conn): conn.state = ConnectionStates.CONNECTED cli = KafkaClient(api_version=(0, 9)) args, kwargs = conn.call_args assert args == ('localhost', 9092, socket.AF_UNSPEC) kwargs.pop('state_change_callback') kwargs.pop('node_id') assert kwargs == cli.config conn.connect.assert_called_with() conn.send.assert_called_once_with(MetadataRequest[0]([])) assert cli._bootstrap_fails == 0 assert cli.cluster.brokers() == set([BrokerMetadata(0, 'foo', 12, None), BrokerMetadata(1, 'bar', 34, None)]) def test_bootstrap_failure(conn): conn.state = ConnectionStates.DISCONNECTED cli = KafkaClient(api_version=(0, 9)) args, kwargs = conn.call_args assert args == ('localhost', 9092, socket.AF_UNSPEC) kwargs.pop('state_change_callback') kwargs.pop('node_id') assert kwargs == cli.config conn.connect.assert_called_with() conn.close.assert_called_with() assert cli._bootstrap_fails == 1 assert cli.cluster.brokers() == set() def test_can_connect(cli, conn): # Node is not in broker metadata - cant connect assert not cli._can_connect(2) # Node is in broker metadata but not in _conns assert 0 not in cli._conns assert cli._can_connect(0) # Node is connected, can't reconnect assert cli._maybe_connect(0) is True assert not cli._can_connect(0) # Node is disconnected, can connect cli._conns[0].state = ConnectionStates.DISCONNECTED assert cli._can_connect(0) # Node is disconnected, but blacked out conn.blacked_out.return_value = True assert not cli._can_connect(0) def test_maybe_connect(cli, conn): try: # Node not in metadata, raises AssertionError cli._maybe_connect(2) except AssertionError: pass else: assert False, 'Exception not raised' # New node_id creates a conn object assert 0 not in cli._conns conn.state = ConnectionStates.DISCONNECTED conn.connect.side_effect = lambda: conn._set_conn_state(ConnectionStates.CONNECTING) assert cli._maybe_connect(0) is False assert cli._conns[0] is conn def test_conn_state_change(mocker, cli, conn): sel = mocker.patch.object(cli, '_selector') node_id = 0 conn.state = ConnectionStates.CONNECTING cli._conn_state_change(node_id, conn) assert node_id in cli._connecting sel.register.assert_called_with(conn._sock, selectors.EVENT_WRITE) conn.state = ConnectionStates.CONNECTED cli._conn_state_change(node_id, conn) assert node_id not in cli._connecting sel.unregister.assert_called_with(conn._sock) sel.register.assert_called_with(conn._sock, selectors.EVENT_READ, conn) # Failure to connect should trigger metadata update assert cli.cluster._need_update is False conn.state = ConnectionStates.DISCONNECTING cli._conn_state_change(node_id, conn) assert node_id not in cli._connecting assert cli.cluster._need_update is True sel.unregister.assert_called_with(conn._sock) conn.state = ConnectionStates.CONNECTING cli._conn_state_change(node_id, conn) assert node_id in cli._connecting conn.state = ConnectionStates.DISCONNECTING cli._conn_state_change(node_id, conn) assert node_id not in cli._connecting def test_ready(mocker, cli, conn): maybe_connect = mocker.patch.object(cli, '_maybe_connect') node_id = 1 cli.ready(node_id) maybe_connect.assert_called_with(node_id) def test_is_ready(mocker, cli, conn): cli._maybe_connect(0) cli._maybe_connect(1) # metadata refresh blocks ready nodes assert cli.is_ready(0) assert cli.is_ready(1) cli._metadata_refresh_in_progress = True assert not cli.is_ready(0) assert not cli.is_ready(1) # requesting metadata update also blocks ready nodes cli._metadata_refresh_in_progress = False assert cli.is_ready(0) assert cli.is_ready(1) cli.cluster.request_update() cli.cluster.config['retry_backoff_ms'] = 0 assert not cli._metadata_refresh_in_progress assert not cli.is_ready(0) assert not cli.is_ready(1) cli.cluster._need_update = False # if connection can't send more, not ready assert cli.is_ready(0) conn.can_send_more.return_value = False assert not cli.is_ready(0) conn.can_send_more.return_value = True # disconnected nodes, not ready assert cli.is_ready(0) conn.state = ConnectionStates.DISCONNECTED assert not cli.is_ready(0) def test_close(mocker, cli, conn): mocker.patch.object(cli, '_selector') # bootstrap connection should have been closed assert conn.close.call_count == 1 # Unknown node - silent cli.close(2) # Single node close cli._maybe_connect(0) assert conn.close.call_count == 1 cli.close(0) assert conn.close.call_count == 2 # All node close cli._maybe_connect(1) cli.close() assert conn.close.call_count == 4 def test_is_disconnected(cli, conn): # False if not connected yet conn.state = ConnectionStates.DISCONNECTED assert not cli.is_disconnected(0) cli._maybe_connect(0) assert cli.is_disconnected(0) conn.state = ConnectionStates.CONNECTING assert not cli.is_disconnected(0) conn.state = ConnectionStates.CONNECTED assert not cli.is_disconnected(0) def test_send(cli, conn): # Send to unknown node => raises AssertionError try: cli.send(2, None) assert False, 'Exception not raised' except AssertionError: pass # Send to disconnected node => NodeNotReady conn.state = ConnectionStates.DISCONNECTED f = cli.send(0, None) assert f.failed() assert isinstance(f.exception, Errors.NodeNotReadyError) conn.state = ConnectionStates.CONNECTED cli._maybe_connect(0) # ProduceRequest w/ 0 required_acks -> no response request = ProduceRequest[0](0, 0, []) ret = cli.send(0, request) assert conn.send.called_with(request, expect_response=False) assert isinstance(ret, Future) request = MetadataRequest[0]([]) cli.send(0, request) assert conn.send.called_with(request, expect_response=True) def test_poll(mocker): mocker.patch.object(KafkaClient, '_bootstrap') metadata = mocker.patch.object(KafkaClient, '_maybe_refresh_metadata') _poll = mocker.patch.object(KafkaClient, '_poll') cli = KafkaClient(api_version=(0, 9)) tasks = mocker.patch.object(cli._delayed_tasks, 'next_at') # metadata timeout wins metadata.return_value = 1000 tasks.return_value = 2 cli.poll() _poll.assert_called_with(1.0, sleep=True) # user timeout wins cli.poll(250) _poll.assert_called_with(0.25, sleep=True) # tasks timeout wins tasks.return_value = 0 cli.poll(250) _poll.assert_called_with(0, sleep=True) # default is request_timeout_ms metadata.return_value = 1000000 tasks.return_value = 10000 cli.poll() _poll.assert_called_with(cli.config['request_timeout_ms'] / 1000.0, sleep=True) def test__poll(): pass def test_in_flight_request_count(): pass def test_least_loaded_node(): pass def test_set_topics(mocker): request_update = mocker.patch.object(ClusterMetadata, 'request_update') request_update.side_effect = lambda: Future() cli = KafkaClient(api_version=(0, 10)) # replace 'empty' with 'non empty' request_update.reset_mock() fut = cli.set_topics(['t1', 't2']) assert not fut.is_done request_update.assert_called_with() # replace 'non empty' with 'same' request_update.reset_mock() fut = cli.set_topics(['t1', 't2']) assert fut.is_done assert fut.value == set(['t1', 't2']) request_update.assert_not_called() # replace 'non empty' with 'empty' request_update.reset_mock() fut = cli.set_topics([]) assert fut.is_done assert fut.value == set() request_update.assert_not_called() @pytest.fixture def client(mocker): mocker.patch.object(KafkaClient, '_bootstrap') _poll = mocker.patch.object(KafkaClient, '_poll') cli = KafkaClient(request_timeout_ms=9999999, retry_backoff_ms=2222, api_version=(0, 9)) tasks = mocker.patch.object(cli._delayed_tasks, 'next_at') tasks.return_value = 9999999 ttl = mocker.patch.object(cli.cluster, 'ttl') ttl.return_value = 0 return cli def test_maybe_refresh_metadata_ttl(mocker, client): client.cluster.ttl.return_value = 1234 client.poll(timeout_ms=9999999, sleep=True) client._poll.assert_called_with(1.234, sleep=True) def test_maybe_refresh_metadata_backoff(mocker, client): now = time.time() t = mocker.patch('time.time') t.return_value = now client._last_no_node_available_ms = now * 1000 client.poll(timeout_ms=9999999, sleep=True) client._poll.assert_called_with(2.222, sleep=True) def test_maybe_refresh_metadata_in_progress(mocker, client): client._metadata_refresh_in_progress = True client.poll(timeout_ms=9999999, sleep=True) client._poll.assert_called_with(9999.999, sleep=True) def test_maybe_refresh_metadata_update(mocker, client): mocker.patch.object(client, 'least_loaded_node', return_value='foobar') mocker.patch.object(client, '_can_send_request', return_value=True) send = mocker.patch.object(client, 'send') client.poll(timeout_ms=9999999, sleep=True) client._poll.assert_called_with(0, sleep=True) assert client._metadata_refresh_in_progress request = MetadataRequest[0]([]) send.assert_called_with('foobar', request) def test_maybe_refresh_metadata_failure(mocker, client): mocker.patch.object(client, 'least_loaded_node', return_value='foobar') now = time.time() t = mocker.patch('time.time') t.return_value = now client.poll(timeout_ms=9999999, sleep=True) client._poll.assert_called_with(0, sleep=True) assert client._last_no_node_available_ms == now * 1000 assert not client._metadata_refresh_in_progress def test_schedule(): pass def test_unschedule(): pass kafka-1.3.2/test/test_client_integration.py0000644001271300127130000000624312702214455020640 0ustar dpowers00000000000000import os from kafka.errors import KafkaTimeoutError from kafka.protocol import create_message from kafka.structs import ( FetchRequestPayload, OffsetCommitRequestPayload, OffsetFetchRequestPayload, ProduceRequestPayload) from test.fixtures import ZookeeperFixture, KafkaFixture from test.testutil import KafkaIntegrationTestCase, kafka_versions class TestKafkaClientIntegration(KafkaIntegrationTestCase): @classmethod def setUpClass(cls): # noqa if not os.environ.get('KAFKA_VERSION'): return cls.zk = ZookeeperFixture.instance() cls.server = KafkaFixture.instance(0, cls.zk.host, cls.zk.port) @classmethod def tearDownClass(cls): # noqa if not os.environ.get('KAFKA_VERSION'): return cls.server.close() cls.zk.close() def test_consume_none(self): fetch = FetchRequestPayload(self.topic, 0, 0, 1024) fetch_resp, = self.client.send_fetch_request([fetch]) self.assertEqual(fetch_resp.error, 0) self.assertEqual(fetch_resp.topic, self.topic) self.assertEqual(fetch_resp.partition, 0) messages = list(fetch_resp.messages) self.assertEqual(len(messages), 0) def test_ensure_topic_exists(self): # assume that self.topic was created by setUp # if so, this should succeed self.client.ensure_topic_exists(self.topic, timeout=1) # ensure_topic_exists should fail with KafkaTimeoutError with self.assertRaises(KafkaTimeoutError): self.client.ensure_topic_exists('this_topic_doesnt_exist', timeout=0) def test_send_produce_request_maintains_request_response_order(self): self.client.ensure_topic_exists('foo') self.client.ensure_topic_exists('bar') requests = [ ProduceRequestPayload( 'foo', 0, [create_message(b'a'), create_message(b'b')]), ProduceRequestPayload( 'bar', 1, [create_message(b'a'), create_message(b'b')]), ProduceRequestPayload( 'foo', 1, [create_message(b'a'), create_message(b'b')]), ProduceRequestPayload( 'bar', 0, [create_message(b'a'), create_message(b'b')]), ] responses = self.client.send_produce_request(requests) while len(responses): request = requests.pop() response = responses.pop() self.assertEqual(request.topic, response.topic) self.assertEqual(request.partition, response.partition) #################### # Offset Tests # #################### @kafka_versions('>=0.8.1') def test_commit_fetch_offsets(self): req = OffsetCommitRequestPayload(self.topic, 0, 42, 'metadata') (resp,) = self.client.send_offset_commit_request('group', [req]) self.assertEqual(resp.error, 0) req = OffsetFetchRequestPayload(self.topic, 0) (resp,) = self.client.send_offset_fetch_request('group', [req]) self.assertEqual(resp.error, 0) self.assertEqual(resp.offset, 42) self.assertEqual(resp.metadata, '') # Metadata isn't stored for now kafka-1.3.2/test/test_codec.py0000644001271300127130000000725412737112516016042 0ustar dpowers00000000000000import struct import pytest from six.moves import xrange from kafka.codec import ( has_snappy, has_gzip, has_lz4, gzip_encode, gzip_decode, snappy_encode, snappy_decode, lz4_encode, lz4_decode, lz4_encode_old_kafka, lz4_decode_old_kafka, ) from test.testutil import random_string def test_gzip(): for i in xrange(1000): b1 = random_string(100).encode('utf-8') b2 = gzip_decode(gzip_encode(b1)) assert b1 == b2 @pytest.mark.skipif(not has_snappy(), reason="Snappy not available") def test_snappy(): for i in xrange(1000): b1 = random_string(100).encode('utf-8') b2 = snappy_decode(snappy_encode(b1)) assert b1 == b2 @pytest.mark.skipif(not has_snappy(), reason="Snappy not available") def test_snappy_detect_xerial(): import kafka as kafka1 _detect_xerial_stream = kafka1.codec._detect_xerial_stream header = b'\x82SNAPPY\x00\x00\x00\x00\x01\x00\x00\x00\x01Some extra bytes' false_header = b'\x01SNAPPY\x00\x00\x00\x01\x00\x00\x00\x01' default_snappy = snappy_encode(b'foobar' * 50) random_snappy = snappy_encode(b'SNAPPY' * 50, xerial_compatible=False) short_data = b'\x01\x02\x03\x04' assert _detect_xerial_stream(header) is True assert _detect_xerial_stream(b'') is False assert _detect_xerial_stream(b'\x00') is False assert _detect_xerial_stream(false_header) is False assert _detect_xerial_stream(default_snappy) is True assert _detect_xerial_stream(random_snappy) is False assert _detect_xerial_stream(short_data) is False @pytest.mark.skipif(not has_snappy(), reason="Snappy not available") def test_snappy_decode_xerial(): header = b'\x82SNAPPY\x00\x00\x00\x00\x01\x00\x00\x00\x01' random_snappy = snappy_encode(b'SNAPPY' * 50, xerial_compatible=False) block_len = len(random_snappy) random_snappy2 = snappy_encode(b'XERIAL' * 50, xerial_compatible=False) block_len2 = len(random_snappy2) to_test = header \ + struct.pack('!i', block_len) + random_snappy \ + struct.pack('!i', block_len2) + random_snappy2 \ assert snappy_decode(to_test) == (b'SNAPPY' * 50) + (b'XERIAL' * 50) @pytest.mark.skipif(not has_snappy(), reason="Snappy not available") def test_snappy_encode_xerial(): to_ensure = ( b'\x82SNAPPY\x00\x00\x00\x00\x01\x00\x00\x00\x01' b'\x00\x00\x00\x18' b'\xac\x02\x14SNAPPY\xfe\x06\x00\xfe\x06\x00\xfe\x06\x00\xfe\x06\x00\x96\x06\x00' b'\x00\x00\x00\x18' b'\xac\x02\x14XERIAL\xfe\x06\x00\xfe\x06\x00\xfe\x06\x00\xfe\x06\x00\x96\x06\x00' ) to_test = (b'SNAPPY' * 50) + (b'XERIAL' * 50) compressed = snappy_encode(to_test, xerial_compatible=True, xerial_blocksize=300) assert compressed == to_ensure @pytest.mark.skipif(not has_lz4(), reason="LZ4 not available") def test_lz4(): for i in xrange(1000): b1 = random_string(100).encode('utf-8') b2 = lz4_decode(lz4_encode(b1)) assert len(b1) == len(b2) assert b1 == b2 @pytest.mark.skipif(not has_lz4(), reason="LZ4 not available") def test_lz4_old(): for i in xrange(1000): b1 = random_string(100).encode('utf-8') b2 = lz4_decode_old_kafka(lz4_encode_old_kafka(b1)) assert len(b1) == len(b2) assert b1 == b2 @pytest.mark.xfail(reason="lz4tools library doesnt support incremental decompression") @pytest.mark.skipif(not has_lz4(), reason="LZ4 not available") def test_lz4_incremental(): for i in xrange(1000): # lz4 max single block size is 4MB # make sure we test with multiple-blocks b1 = random_string(100).encode('utf-8') * 50000 b2 = lz4_decode(lz4_encode(b1)) assert len(b1) == len(b2) assert b1 == b2 kafka-1.3.2/test/test_conn.py0000644001271300127130000002100613031057471015705 0ustar dpowers00000000000000# pylint: skip-file from __future__ import absolute_import from errno import EALREADY, EINPROGRESS, EISCONN, ECONNRESET import socket import time import mock import pytest from kafka.conn import BrokerConnection, ConnectionStates, collect_hosts from kafka.protocol.api import RequestHeader from kafka.protocol.metadata import MetadataRequest import kafka.common as Errors @pytest.fixture def _socket(mocker): socket = mocker.MagicMock() socket.connect_ex.return_value = 0 mocker.patch('socket.socket', return_value=socket) return socket @pytest.fixture def conn(_socket): conn = BrokerConnection('localhost', 9092, socket.AF_INET) return conn @pytest.mark.parametrize("states", [ (([EINPROGRESS, EALREADY], ConnectionStates.CONNECTING),), (([EALREADY, EALREADY], ConnectionStates.CONNECTING),), (([0], ConnectionStates.CONNECTED),), (([EINPROGRESS, EALREADY], ConnectionStates.CONNECTING), ([ECONNRESET], ConnectionStates.DISCONNECTED)), (([EINPROGRESS, EALREADY], ConnectionStates.CONNECTING), ([EALREADY], ConnectionStates.CONNECTING), ([EISCONN], ConnectionStates.CONNECTED)), ]) def test_connect(_socket, conn, states): assert conn.state is ConnectionStates.DISCONNECTED for errno, state in states: _socket.connect_ex.side_effect = errno conn.connect() assert conn.state is state def test_connect_timeout(_socket, conn): assert conn.state is ConnectionStates.DISCONNECTED # Initial connect returns EINPROGRESS # immediate inline connect returns EALREADY # second explicit connect returns EALREADY # third explicit connect returns EALREADY and times out via last_attempt _socket.connect_ex.side_effect = [EINPROGRESS, EALREADY, EALREADY, EALREADY] conn.connect() assert conn.state is ConnectionStates.CONNECTING conn.connect() assert conn.state is ConnectionStates.CONNECTING conn.last_attempt = 0 conn.connect() assert conn.state is ConnectionStates.DISCONNECTED def test_blacked_out(conn): assert conn.blacked_out() is False conn.last_attempt = time.time() assert conn.blacked_out() is True def test_connected(conn): assert conn.connected() is False conn.state = ConnectionStates.CONNECTED assert conn.connected() is True def test_connecting(conn): assert conn.connecting() is False conn.state = ConnectionStates.CONNECTING assert conn.connecting() is True conn.state = ConnectionStates.CONNECTED assert conn.connecting() is False def test_send_disconnected(conn): conn.state = ConnectionStates.DISCONNECTED f = conn.send('foobar') assert f.failed() is True assert isinstance(f.exception, Errors.ConnectionError) def test_send_connecting(conn): conn.state = ConnectionStates.CONNECTING f = conn.send('foobar') assert f.failed() is True assert isinstance(f.exception, Errors.NodeNotReadyError) def test_send_max_ifr(conn): conn.state = ConnectionStates.CONNECTED max_ifrs = conn.config['max_in_flight_requests_per_connection'] for _ in range(max_ifrs): conn.in_flight_requests.append('foo') f = conn.send('foobar') assert f.failed() is True assert isinstance(f.exception, Errors.TooManyInFlightRequests) def test_send_no_response(_socket, conn): conn.connect() assert conn.state is ConnectionStates.CONNECTED req = MetadataRequest[0]([]) header = RequestHeader(req, client_id=conn.config['client_id']) payload_bytes = len(header.encode()) + len(req.encode()) third = payload_bytes // 3 remainder = payload_bytes % 3 _socket.send.side_effect = [4, third, third, third, remainder] assert len(conn.in_flight_requests) == 0 f = conn.send(req, expect_response=False) assert f.succeeded() is True assert f.value is None assert len(conn.in_flight_requests) == 0 def test_send_response(_socket, conn): conn.connect() assert conn.state is ConnectionStates.CONNECTED req = MetadataRequest[0]([]) header = RequestHeader(req, client_id=conn.config['client_id']) payload_bytes = len(header.encode()) + len(req.encode()) third = payload_bytes // 3 remainder = payload_bytes % 3 _socket.send.side_effect = [4, third, third, third, remainder] assert len(conn.in_flight_requests) == 0 f = conn.send(req) assert f.is_done is False assert len(conn.in_flight_requests) == 1 def test_send_error(_socket, conn): conn.connect() assert conn.state is ConnectionStates.CONNECTED req = MetadataRequest[0]([]) try: _socket.send.side_effect = ConnectionError except NameError: _socket.send.side_effect = socket.error f = conn.send(req) assert f.failed() is True assert isinstance(f.exception, Errors.ConnectionError) assert _socket.close.call_count == 1 assert conn.state is ConnectionStates.DISCONNECTED def test_can_send_more(conn): assert conn.can_send_more() is True max_ifrs = conn.config['max_in_flight_requests_per_connection'] for _ in range(max_ifrs): assert conn.can_send_more() is True conn.in_flight_requests.append('foo') assert conn.can_send_more() is False def test_recv_disconnected(): sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.bind(('127.0.0.1', 0)) port = sock.getsockname()[1] sock.listen(5) conn = BrokerConnection('127.0.0.1', port, socket.AF_INET) timeout = time.time() + 1 while time.time() < timeout: conn.connect() if conn.connected(): break else: assert False, 'Connection attempt to local socket timed-out ?' conn.send(MetadataRequest[0]([])) # Disconnect server socket sock.close() # Attempt to receive should mark connection as disconnected assert conn.connected() conn.recv() assert conn.disconnected() def test_recv_disconnected_too(_socket, conn): conn.connect() assert conn.connected() req = MetadataRequest[0]([]) header = RequestHeader(req, client_id=conn.config['client_id']) payload_bytes = len(header.encode()) + len(req.encode()) _socket.send.side_effect = [4, payload_bytes] conn.send(req) # Empty data on recv means the socket is disconnected _socket.recv.return_value = b'' # Attempt to receive should mark connection as disconnected assert conn.connected() conn.recv() assert conn.disconnected() def test_recv(_socket, conn): pass # TODO def test_close(conn): pass # TODO def test_collect_hosts__happy_path(): hosts = "127.0.0.1:1234,127.0.0.1" results = collect_hosts(hosts) assert set(results) == set([ ('127.0.0.1', 1234, socket.AF_INET), ('127.0.0.1', 9092, socket.AF_INET), ]) def test_collect_hosts__ipv6(): hosts = "[localhost]:1234,[2001:1000:2000::1],[2001:1000:2000::1]:1234" results = collect_hosts(hosts) assert set(results) == set([ ('localhost', 1234, socket.AF_INET6), ('2001:1000:2000::1', 9092, socket.AF_INET6), ('2001:1000:2000::1', 1234, socket.AF_INET6), ]) def test_collect_hosts__string_list(): hosts = [ 'localhost:1234', 'localhost', '[localhost]', '2001::1', '[2001::1]', '[2001::1]:1234', ] results = collect_hosts(hosts) assert set(results) == set([ ('localhost', 1234, socket.AF_UNSPEC), ('localhost', 9092, socket.AF_UNSPEC), ('localhost', 9092, socket.AF_INET6), ('2001::1', 9092, socket.AF_INET6), ('2001::1', 9092, socket.AF_INET6), ('2001::1', 1234, socket.AF_INET6), ]) def test_collect_hosts__with_spaces(): hosts = "localhost:1234, localhost" results = collect_hosts(hosts) assert set(results) == set([ ('localhost', 1234, socket.AF_UNSPEC), ('localhost', 9092, socket.AF_UNSPEC), ]) def test_lookup_on_connect(): hostname = 'example.org' port = 9092 conn = BrokerConnection(hostname, port, socket.AF_UNSPEC) assert conn.host == conn.hostname == hostname ip1 = '127.0.0.1' mock_return1 = [ (2, 2, 17, '', (ip1, 9092)), ] with mock.patch("socket.getaddrinfo", return_value=mock_return1) as m: conn.connect() m.assert_called_once_with(hostname, port, 0, 1) conn.close() assert conn.host == ip1 ip2 = '127.0.0.2' mock_return2 = [ (2, 2, 17, '', (ip2, 9092)), ] with mock.patch("socket.getaddrinfo", return_value=mock_return2) as m: conn.connect() m.assert_called_once_with(hostname, port, 0, 1) conn.close() assert conn.host == ip2 kafka-1.3.2/test/test_consumer.py0000644001271300127130000001302012702214455016601 0ustar dpowers00000000000000import sys from mock import MagicMock, patch from . import unittest from kafka import SimpleConsumer, KafkaConsumer, MultiProcessConsumer from kafka.errors import ( FailedPayloadsError, KafkaConfigurationError, NotLeaderForPartitionError, UnknownTopicOrPartitionError) from kafka.structs import ( FetchResponsePayload, OffsetAndMessage, OffsetFetchResponsePayload) class TestKafkaConsumer(unittest.TestCase): def test_non_integer_partitions(self): with self.assertRaises(AssertionError): SimpleConsumer(MagicMock(), 'group', 'topic', partitions = [ '0' ]) class TestMultiProcessConsumer(unittest.TestCase): @unittest.skipIf(sys.platform.startswith('win'), 'test mocking fails on windows') def test_partition_list(self): client = MagicMock() partitions = (0,) with patch.object(MultiProcessConsumer, 'fetch_last_known_offsets') as fetch_last_known_offsets: MultiProcessConsumer(client, 'testing-group', 'testing-topic', partitions=partitions) self.assertEqual(fetch_last_known_offsets.call_args[0], (partitions,) ) self.assertEqual(client.get_partition_ids_for_topic.call_count, 0) # pylint: disable=no-member class TestSimpleConsumer(unittest.TestCase): def test_simple_consumer_failed_payloads(self): client = MagicMock() consumer = SimpleConsumer(client, group=None, topic='topic', partitions=[0, 1], auto_commit=False) def failed_payloads(payload): return FailedPayloadsError(payload) client.send_fetch_request.side_effect = self.fail_requests_factory(failed_payloads) # This should not raise an exception consumer.get_messages(5) def test_simple_consumer_leader_change(self): client = MagicMock() consumer = SimpleConsumer(client, group=None, topic='topic', partitions=[0, 1], auto_commit=False) # Mock so that only the first request gets a valid response def not_leader(request): return FetchResponsePayload(request.topic, request.partition, NotLeaderForPartitionError.errno, -1, ()) client.send_fetch_request.side_effect = self.fail_requests_factory(not_leader) # This should not raise an exception consumer.get_messages(20) # client should have updated metadata self.assertGreaterEqual(client.reset_topic_metadata.call_count, 1) self.assertGreaterEqual(client.load_metadata_for_topics.call_count, 1) def test_simple_consumer_unknown_topic_partition(self): client = MagicMock() consumer = SimpleConsumer(client, group=None, topic='topic', partitions=[0, 1], auto_commit=False) # Mock so that only the first request gets a valid response def unknown_topic_partition(request): return FetchResponsePayload(request.topic, request.partition, UnknownTopicOrPartitionError.errno, -1, ()) client.send_fetch_request.side_effect = self.fail_requests_factory(unknown_topic_partition) # This should not raise an exception with self.assertRaises(UnknownTopicOrPartitionError): consumer.get_messages(20) def test_simple_consumer_commit_does_not_raise(self): client = MagicMock() client.get_partition_ids_for_topic.return_value = [0, 1] def mock_offset_fetch_request(group, payloads, **kwargs): return [OffsetFetchResponsePayload(p.topic, p.partition, 0, b'', 0) for p in payloads] client.send_offset_fetch_request.side_effect = mock_offset_fetch_request def mock_offset_commit_request(group, payloads, **kwargs): raise FailedPayloadsError(payloads[0]) client.send_offset_commit_request.side_effect = mock_offset_commit_request consumer = SimpleConsumer(client, group='foobar', topic='topic', partitions=[0, 1], auto_commit=False) # Mock internal commit check consumer.count_since_commit = 10 # This should not raise an exception self.assertFalse(consumer.commit(partitions=[0, 1])) def test_simple_consumer_reset_partition_offset(self): client = MagicMock() def mock_offset_request(payloads, **kwargs): raise FailedPayloadsError(payloads[0]) client.send_offset_request.side_effect = mock_offset_request consumer = SimpleConsumer(client, group='foobar', topic='topic', partitions=[0, 1], auto_commit=False) # This should not raise an exception self.assertEqual(consumer.reset_partition_offset(0), None) @staticmethod def fail_requests_factory(error_factory): # Mock so that only the first request gets a valid response def fail_requests(payloads, **kwargs): responses = [ FetchResponsePayload(payloads[0].topic, payloads[0].partition, 0, 0, [OffsetAndMessage( payloads[0].offset + i, "msg %d" % (payloads[0].offset + i)) for i in range(10)]), ] for failure in payloads[1:]: responses.append(error_factory(failure)) return responses return fail_requests kafka-1.3.2/test/test_consumer_group.py0000644001271300127130000001122413031057471020020 0ustar dpowers00000000000000import collections import logging import threading import time import pytest import six from kafka import SimpleClient from kafka.conn import ConnectionStates from kafka.consumer.group import KafkaConsumer from kafka.structs import TopicPartition from test.conftest import version from test.testutil import random_string def get_connect_str(kafka_broker): return 'localhost:' + str(kafka_broker.port) @pytest.fixture def simple_client(kafka_broker): return SimpleClient(get_connect_str(kafka_broker)) @pytest.fixture def topic(simple_client): topic = random_string(5) simple_client.ensure_topic_exists(topic) return topic @pytest.mark.skipif(not version(), reason="No KAFKA_VERSION set") def test_consumer(kafka_broker, version): # 0.8.2 brokers need a topic to function well if version >= (0, 8, 2) and version < (0, 9): topic(simple_client(kafka_broker)) consumer = KafkaConsumer(bootstrap_servers=get_connect_str(kafka_broker)) consumer.poll(500) assert len(consumer._client._conns) > 0 node_id = list(consumer._client._conns.keys())[0] assert consumer._client._conns[node_id].state is ConnectionStates.CONNECTED @pytest.mark.skipif(version() < (0, 9), reason='Unsupported Kafka Version') @pytest.mark.skipif(not version(), reason="No KAFKA_VERSION set") def test_group(kafka_broker, topic): num_partitions = 4 connect_str = get_connect_str(kafka_broker) consumers = {} stop = {} threads = {} messages = collections.defaultdict(list) def consumer_thread(i): assert i not in consumers assert i not in stop stop[i] = threading.Event() consumers[i] = KafkaConsumer(topic, bootstrap_servers=connect_str, heartbeat_interval_ms=500) while not stop[i].is_set(): for tp, records in six.itervalues(consumers[i].poll(100)): messages[i][tp].extend(records) consumers[i].close() del consumers[i] del stop[i] num_consumers = 4 for i in range(num_consumers): t = threading.Thread(target=consumer_thread, args=(i,)) t.start() threads[i] = t try: timeout = time.time() + 35 while True: for c in range(num_consumers): # Verify all consumers have been created if c not in consumers: break # Verify all consumers have an assignment elif not consumers[c].assignment(): break # If all consumers exist and have an assignment else: # Verify all consumers are in the same generation # then log state and break while loop generations = set([consumer._coordinator.generation for consumer in list(consumers.values())]) # New generation assignment is not complete until # coordinator.rejoining = False rejoining = any([consumer._coordinator.rejoining for consumer in list(consumers.values())]) if not rejoining and len(generations) == 1: for c, consumer in list(consumers.items()): logging.info("[%s] %s %s: %s", c, consumer._coordinator.generation, consumer._coordinator.member_id, consumer.assignment()) break assert time.time() < timeout, "timeout waiting for assignments" group_assignment = set() for c in range(num_consumers): assert len(consumers[c].assignment()) != 0 assert set.isdisjoint(consumers[c].assignment(), group_assignment) group_assignment.update(consumers[c].assignment()) assert group_assignment == set([ TopicPartition(topic, partition) for partition in range(num_partitions)]) finally: for c in range(num_consumers): stop[c].set() threads[c].join() @pytest.mark.skipif(not version(), reason="No KAFKA_VERSION set") def test_paused(kafka_broker, topic): consumer = KafkaConsumer(bootstrap_servers=get_connect_str(kafka_broker)) topics = [TopicPartition(topic, 1)] consumer.assign(topics) assert set(topics) == consumer.assignment() assert set() == consumer.paused() consumer.pause(topics[0]) assert set([topics[0]]) == consumer.paused() consumer.resume(topics[0]) assert set() == consumer.paused() consumer.unsubscribe() assert set() == consumer.paused() kafka-1.3.2/test/test_consumer_integration.py0000644001271300127130000005175113031057471021220 0ustar dpowers00000000000000import logging import os from six.moves import xrange from . import unittest from kafka import ( KafkaConsumer, MultiProcessConsumer, SimpleConsumer, create_message, create_gzip_message ) from kafka.consumer.base import MAX_FETCH_BUFFER_SIZE_BYTES from kafka.errors import ConsumerFetchSizeTooSmall, OffsetOutOfRangeError from kafka.structs import ProduceRequestPayload, TopicPartition from test.fixtures import ZookeeperFixture, KafkaFixture from test.testutil import ( KafkaIntegrationTestCase, kafka_versions, random_string, Timer ) class TestConsumerIntegration(KafkaIntegrationTestCase): @classmethod def setUpClass(cls): if not os.environ.get('KAFKA_VERSION'): return cls.zk = ZookeeperFixture.instance() chroot = random_string(10) cls.server1 = KafkaFixture.instance(0, cls.zk.host, cls.zk.port, zk_chroot=chroot) cls.server2 = KafkaFixture.instance(1, cls.zk.host, cls.zk.port, zk_chroot=chroot) cls.server = cls.server1 # Bootstrapping server @classmethod def tearDownClass(cls): if not os.environ.get('KAFKA_VERSION'): return cls.server1.close() cls.server2.close() cls.zk.close() def send_messages(self, partition, messages): messages = [ create_message(self.msg(str(msg))) for msg in messages ] produce = ProduceRequestPayload(self.topic, partition, messages = messages) resp, = self.client.send_produce_request([produce]) self.assertEqual(resp.error, 0) return [ x.value for x in messages ] def send_gzip_message(self, partition, messages): message = create_gzip_message([(self.msg(str(msg)), None) for msg in messages]) produce = ProduceRequestPayload(self.topic, partition, messages = [message]) resp, = self.client.send_produce_request([produce]) self.assertEqual(resp.error, 0) def assert_message_count(self, messages, num_messages): # Make sure we got them all self.assertEqual(len(messages), num_messages) # Make sure there are no duplicates self.assertEqual(len(set(messages)), num_messages) def consumer(self, **kwargs): if os.environ['KAFKA_VERSION'] == "0.8.0": # Kafka 0.8.0 simply doesn't support offset requests, so hard code it being off kwargs['group'] = None kwargs['auto_commit'] = False else: kwargs.setdefault('group', None) kwargs.setdefault('auto_commit', False) consumer_class = kwargs.pop('consumer', SimpleConsumer) group = kwargs.pop('group', None) topic = kwargs.pop('topic', self.topic) if consumer_class in [SimpleConsumer, MultiProcessConsumer]: kwargs.setdefault('iter_timeout', 0) return consumer_class(self.client, group, topic, **kwargs) def kafka_consumer(self, **configs): brokers = '%s:%d' % (self.server.host, self.server.port) consumer = KafkaConsumer(self.topic, bootstrap_servers=brokers, **configs) return consumer def test_simple_consumer(self): self.send_messages(0, range(0, 100)) self.send_messages(1, range(100, 200)) # Start a consumer consumer = self.consumer() self.assert_message_count([ message for message in consumer ], 200) consumer.stop() def test_simple_consumer_gzip(self): self.send_gzip_message(0, range(0, 100)) self.send_gzip_message(1, range(100, 200)) # Start a consumer consumer = self.consumer() self.assert_message_count([ message for message in consumer ], 200) consumer.stop() def test_simple_consumer_smallest_offset_reset(self): self.send_messages(0, range(0, 100)) self.send_messages(1, range(100, 200)) consumer = self.consumer(auto_offset_reset='smallest') # Move fetch offset ahead of 300 message (out of range) consumer.seek(300, 2) # Since auto_offset_reset is set to smallest we should read all 200 # messages from beginning. self.assert_message_count([message for message in consumer], 200) def test_simple_consumer_largest_offset_reset(self): self.send_messages(0, range(0, 100)) self.send_messages(1, range(100, 200)) # Default largest consumer = self.consumer() # Move fetch offset ahead of 300 message (out of range) consumer.seek(300, 2) # Since auto_offset_reset is set to largest we should not read any # messages. self.assert_message_count([message for message in consumer], 0) # Send 200 new messages to the queue self.send_messages(0, range(200, 300)) self.send_messages(1, range(300, 400)) # Since the offset is set to largest we should read all the new messages. self.assert_message_count([message for message in consumer], 200) def test_simple_consumer_no_reset(self): self.send_messages(0, range(0, 100)) self.send_messages(1, range(100, 200)) # Default largest consumer = self.consumer(auto_offset_reset=None) # Move fetch offset ahead of 300 message (out of range) consumer.seek(300, 2) with self.assertRaises(OffsetOutOfRangeError): consumer.get_message() @kafka_versions('>=0.8.1') def test_simple_consumer_load_initial_offsets(self): self.send_messages(0, range(0, 100)) self.send_messages(1, range(100, 200)) # Create 1st consumer and change offsets consumer = self.consumer(group='test_simple_consumer_load_initial_offsets') self.assertEqual(consumer.offsets, {0: 0, 1: 0}) consumer.offsets.update({0:51, 1:101}) # Update counter after manual offsets update consumer.count_since_commit += 1 consumer.commit() # Create 2nd consumer and check initial offsets consumer = self.consumer(group='test_simple_consumer_load_initial_offsets', auto_commit=False) self.assertEqual(consumer.offsets, {0: 51, 1: 101}) def test_simple_consumer__seek(self): self.send_messages(0, range(0, 100)) self.send_messages(1, range(100, 200)) consumer = self.consumer() # Rewind 10 messages from the end consumer.seek(-10, 2) self.assert_message_count([ message for message in consumer ], 10) # Rewind 13 messages from the end consumer.seek(-13, 2) self.assert_message_count([ message for message in consumer ], 13) # Set absolute offset consumer.seek(100) self.assert_message_count([ message for message in consumer ], 0) consumer.seek(100, partition=0) self.assert_message_count([ message for message in consumer ], 0) consumer.seek(101, partition=1) self.assert_message_count([ message for message in consumer ], 0) consumer.seek(90, partition=0) self.assert_message_count([ message for message in consumer ], 10) consumer.seek(20, partition=1) self.assert_message_count([ message for message in consumer ], 80) consumer.seek(0, partition=1) self.assert_message_count([ message for message in consumer ], 100) consumer.stop() def test_simple_consumer_blocking(self): consumer = self.consumer() # Ask for 5 messages, nothing in queue, block 1 second with Timer() as t: messages = consumer.get_messages(block=True, timeout=1) self.assert_message_count(messages, 0) self.assertGreaterEqual(t.interval, 1) self.send_messages(0, range(0, 5)) self.send_messages(1, range(5, 10)) # Ask for 5 messages, 10 in queue. Get 5 back, no blocking with Timer() as t: messages = consumer.get_messages(count=5, block=True, timeout=3) self.assert_message_count(messages, 5) self.assertLess(t.interval, 3) # Ask for 10 messages, get 5 back, block 1 second with Timer() as t: messages = consumer.get_messages(count=10, block=True, timeout=1) self.assert_message_count(messages, 5) self.assertGreaterEqual(t.interval, 1) # Ask for 10 messages, 5 in queue, ask to block for 1 message or 1 # second, get 5 back, no blocking self.send_messages(0, range(0, 3)) self.send_messages(1, range(3, 5)) with Timer() as t: messages = consumer.get_messages(count=10, block=1, timeout=1) self.assert_message_count(messages, 5) self.assertLessEqual(t.interval, 1) consumer.stop() def test_simple_consumer_pending(self): # make sure that we start with no pending messages consumer = self.consumer() self.assertEquals(consumer.pending(), 0) self.assertEquals(consumer.pending(partitions=[0]), 0) self.assertEquals(consumer.pending(partitions=[1]), 0) # Produce 10 messages to partitions 0 and 1 self.send_messages(0, range(0, 10)) self.send_messages(1, range(10, 20)) consumer = self.consumer() self.assertEqual(consumer.pending(), 20) self.assertEqual(consumer.pending(partitions=[0]), 10) self.assertEqual(consumer.pending(partitions=[1]), 10) # move to last message, so one partition should have 1 pending # message and other 0 consumer.seek(-1, 2) self.assertEqual(consumer.pending(), 1) pending_part1 = consumer.pending(partitions=[0]) pending_part2 = consumer.pending(partitions=[1]) self.assertEquals(set([0, 1]), set([pending_part1, pending_part2])) consumer.stop() @unittest.skip('MultiProcessConsumer deprecated and these tests are flaky') def test_multi_process_consumer(self): # Produce 100 messages to partitions 0 and 1 self.send_messages(0, range(0, 100)) self.send_messages(1, range(100, 200)) consumer = self.consumer(consumer = MultiProcessConsumer) self.assert_message_count([ message for message in consumer ], 200) consumer.stop() @unittest.skip('MultiProcessConsumer deprecated and these tests are flaky') def test_multi_process_consumer_blocking(self): consumer = self.consumer(consumer = MultiProcessConsumer) # Ask for 5 messages, No messages in queue, block 1 second with Timer() as t: messages = consumer.get_messages(block=True, timeout=1) self.assert_message_count(messages, 0) self.assertGreaterEqual(t.interval, 1) # Send 10 messages self.send_messages(0, range(0, 10)) # Ask for 5 messages, 10 messages in queue, block 0 seconds with Timer() as t: messages = consumer.get_messages(count=5, block=True, timeout=5) self.assert_message_count(messages, 5) self.assertLessEqual(t.interval, 1) # Ask for 10 messages, 5 in queue, block 1 second with Timer() as t: messages = consumer.get_messages(count=10, block=True, timeout=1) self.assert_message_count(messages, 5) self.assertGreaterEqual(t.interval, 1) # Ask for 10 messages, 5 in queue, ask to block for 1 message or 1 # second, get at least one back, no blocking self.send_messages(0, range(0, 5)) with Timer() as t: messages = consumer.get_messages(count=10, block=1, timeout=1) received_message_count = len(messages) self.assertGreaterEqual(received_message_count, 1) self.assert_message_count(messages, received_message_count) self.assertLessEqual(t.interval, 1) consumer.stop() @unittest.skip('MultiProcessConsumer deprecated and these tests are flaky') def test_multi_proc_pending(self): self.send_messages(0, range(0, 10)) self.send_messages(1, range(10, 20)) # set group to None and auto_commit to False to avoid interactions w/ # offset commit/fetch apis consumer = MultiProcessConsumer(self.client, None, self.topic, auto_commit=False, iter_timeout=0) self.assertEqual(consumer.pending(), 20) self.assertEqual(consumer.pending(partitions=[0]), 10) self.assertEqual(consumer.pending(partitions=[1]), 10) consumer.stop() @unittest.skip('MultiProcessConsumer deprecated and these tests are flaky') @kafka_versions('>=0.8.1') def test_multi_process_consumer_load_initial_offsets(self): self.send_messages(0, range(0, 10)) self.send_messages(1, range(10, 20)) # Create 1st consumer and change offsets consumer = self.consumer(group='test_multi_process_consumer_load_initial_offsets') self.assertEqual(consumer.offsets, {0: 0, 1: 0}) consumer.offsets.update({0:5, 1:15}) # Update counter after manual offsets update consumer.count_since_commit += 1 consumer.commit() # Create 2nd consumer and check initial offsets consumer = self.consumer(consumer = MultiProcessConsumer, group='test_multi_process_consumer_load_initial_offsets', auto_commit=False) self.assertEqual(consumer.offsets, {0: 5, 1: 15}) def test_large_messages(self): # Produce 10 "normal" size messages small_messages = self.send_messages(0, [ str(x) for x in range(10) ]) # Produce 10 messages that are large (bigger than default fetch size) large_messages = self.send_messages(0, [ random_string(5000) for x in range(10) ]) # Consumer should still get all of them consumer = self.consumer() expected_messages = set(small_messages + large_messages) actual_messages = set([ x.message.value for x in consumer ]) self.assertEqual(expected_messages, actual_messages) consumer.stop() def test_huge_messages(self): huge_message, = self.send_messages(0, [ create_message(random_string(MAX_FETCH_BUFFER_SIZE_BYTES + 10)), ]) # Create a consumer with the default buffer size consumer = self.consumer() # This consumer failes to get the message with self.assertRaises(ConsumerFetchSizeTooSmall): consumer.get_message(False, 0.1) consumer.stop() # Create a consumer with no fetch size limit big_consumer = self.consumer( max_buffer_size = None, partitions = [0], ) # Seek to the last message big_consumer.seek(-1, 2) # Consume giant message successfully message = big_consumer.get_message(block=False, timeout=10) self.assertIsNotNone(message) self.assertEqual(message.message.value, huge_message) big_consumer.stop() @kafka_versions('>=0.8.1') def test_offset_behavior__resuming_behavior(self): self.send_messages(0, range(0, 100)) self.send_messages(1, range(100, 200)) # Start a consumer consumer1 = self.consumer( group='test_offset_behavior__resuming_behavior', auto_commit=True, auto_commit_every_t = None, auto_commit_every_n = 20, ) # Grab the first 195 messages output_msgs1 = [ consumer1.get_message().message.value for _ in xrange(195) ] self.assert_message_count(output_msgs1, 195) # The total offset across both partitions should be at 180 consumer2 = self.consumer( group='test_offset_behavior__resuming_behavior', auto_commit=True, auto_commit_every_t = None, auto_commit_every_n = 20, ) # 181-200 self.assert_message_count([ message for message in consumer2 ], 20) consumer1.stop() consumer2.stop() @unittest.skip('MultiProcessConsumer deprecated and these tests are flaky') @kafka_versions('>=0.8.1') def test_multi_process_offset_behavior__resuming_behavior(self): self.send_messages(0, range(0, 100)) self.send_messages(1, range(100, 200)) # Start a consumer consumer1 = self.consumer( consumer=MultiProcessConsumer, group='test_multi_process_offset_behavior__resuming_behavior', auto_commit=True, auto_commit_every_t = None, auto_commit_every_n = 20, ) # Grab the first 195 messages output_msgs1 = [] idx = 0 for message in consumer1: output_msgs1.append(message.message.value) idx += 1 if idx >= 195: break self.assert_message_count(output_msgs1, 195) # The total offset across both partitions should be at 180 consumer2 = self.consumer( consumer=MultiProcessConsumer, group='test_multi_process_offset_behavior__resuming_behavior', auto_commit=True, auto_commit_every_t = None, auto_commit_every_n = 20, ) # 181-200 self.assert_message_count([ message for message in consumer2 ], 20) consumer1.stop() consumer2.stop() # TODO: Make this a unit test -- should not require integration def test_fetch_buffer_size(self): # Test parameters (see issue 135 / PR 136) TEST_MESSAGE_SIZE=1048 INIT_BUFFER_SIZE=1024 MAX_BUFFER_SIZE=2048 assert TEST_MESSAGE_SIZE > INIT_BUFFER_SIZE assert TEST_MESSAGE_SIZE < MAX_BUFFER_SIZE assert MAX_BUFFER_SIZE == 2 * INIT_BUFFER_SIZE self.send_messages(0, [ "x" * 1048 ]) self.send_messages(1, [ "x" * 1048 ]) consumer = self.consumer(buffer_size=1024, max_buffer_size=2048) messages = [ message for message in consumer ] self.assertEqual(len(messages), 2) def test_kafka_consumer(self): self.send_messages(0, range(0, 100)) self.send_messages(1, range(100, 200)) # Start a consumer consumer = self.kafka_consumer(auto_offset_reset='earliest') n = 0 messages = {0: set(), 1: set()} for m in consumer: logging.debug("Consumed message %s" % repr(m)) n += 1 messages[m.partition].add(m.offset) if n >= 200: break self.assertEqual(len(messages[0]), 100) self.assertEqual(len(messages[1]), 100) def test_kafka_consumer__blocking(self): TIMEOUT_MS = 500 consumer = self.kafka_consumer(auto_offset_reset='earliest', enable_auto_commit=False, consumer_timeout_ms=TIMEOUT_MS) # Manual assignment avoids overhead of consumer group mgmt consumer.unsubscribe() consumer.assign([TopicPartition(self.topic, 0)]) # Ask for 5 messages, nothing in queue, block 500ms with Timer() as t: with self.assertRaises(StopIteration): msg = next(consumer) self.assertGreaterEqual(t.interval, TIMEOUT_MS / 1000.0 ) self.send_messages(0, range(0, 10)) # Ask for 5 messages, 10 in queue. Get 5 back, no blocking messages = set() with Timer() as t: for i in range(5): msg = next(consumer) messages.add((msg.partition, msg.offset)) self.assertEqual(len(messages), 5) self.assertLess(t.interval, TIMEOUT_MS / 1000.0 ) # Ask for 10 messages, get 5 back, block 500ms messages = set() with Timer() as t: with self.assertRaises(StopIteration): for i in range(10): msg = next(consumer) messages.add((msg.partition, msg.offset)) self.assertEqual(len(messages), 5) self.assertGreaterEqual(t.interval, TIMEOUT_MS / 1000.0 ) @kafka_versions('>=0.8.1') def test_kafka_consumer__offset_commit_resume(self): GROUP_ID = random_string(10) self.send_messages(0, range(0, 100)) self.send_messages(1, range(100, 200)) # Start a consumer consumer1 = self.kafka_consumer( group_id=GROUP_ID, enable_auto_commit=True, auto_commit_interval_ms=100, auto_offset_reset='earliest', ) # Grab the first 180 messages output_msgs1 = [] for _ in xrange(180): m = next(consumer1) output_msgs1.append(m) self.assert_message_count(output_msgs1, 180) consumer1.close() # The total offset across both partitions should be at 180 consumer2 = self.kafka_consumer( group_id=GROUP_ID, enable_auto_commit=True, auto_commit_interval_ms=100, auto_offset_reset='earliest', ) # 181-200 output_msgs2 = [] for _ in xrange(20): m = next(consumer2) output_msgs2.append(m) self.assert_message_count(output_msgs2, 20) self.assertEqual(len(set(output_msgs1) | set(output_msgs2)), 200) kafka-1.3.2/test/test_context.py0000644001271300127130000000763412702214455016450 0ustar dpowers00000000000000""" OffsetCommitContext tests. """ from . import unittest from mock import MagicMock, patch from kafka.context import OffsetCommitContext from kafka.errors import OffsetOutOfRangeError class TestOffsetCommitContext(unittest.TestCase): """ OffsetCommitContext tests. """ def setUp(self): self.client = MagicMock() self.consumer = MagicMock() self.topic = "topic" self.group = "group" self.partition = 0 self.consumer.topic = self.topic self.consumer.group = self.group self.consumer.client = self.client self.consumer.offsets = {self.partition: 0} self.context = OffsetCommitContext(self.consumer) def test_noop(self): """ Should revert consumer after context exit with no mark() call. """ with self.context: # advance offset self.consumer.offsets = {self.partition: 1} # offset restored self.assertEqual(self.consumer.offsets, {self.partition: 0}) # and seek called with relative zero delta self.assertEqual(self.consumer.seek.call_count, 1) self.assertEqual(self.consumer.seek.call_args[0], (0, 1)) def test_mark(self): """ Should remain at marked location ater context exit. """ with self.context as context: context.mark(self.partition, 0) # advance offset self.consumer.offsets = {self.partition: 1} # offset sent to client self.assertEqual(self.client.send_offset_commit_request.call_count, 1) # offset remains advanced self.assertEqual(self.consumer.offsets, {self.partition: 1}) # and seek called with relative zero delta self.assertEqual(self.consumer.seek.call_count, 1) self.assertEqual(self.consumer.seek.call_args[0], (0, 1)) def test_mark_multiple(self): """ Should remain at highest marked location after context exit. """ with self.context as context: context.mark(self.partition, 0) context.mark(self.partition, 1) context.mark(self.partition, 2) # advance offset self.consumer.offsets = {self.partition: 3} # offset sent to client self.assertEqual(self.client.send_offset_commit_request.call_count, 1) # offset remains advanced self.assertEqual(self.consumer.offsets, {self.partition: 3}) # and seek called with relative zero delta self.assertEqual(self.consumer.seek.call_count, 1) self.assertEqual(self.consumer.seek.call_args[0], (0, 1)) def test_rollback(self): """ Should rollback to initial offsets on context exit with exception. """ with self.assertRaises(Exception): with self.context as context: context.mark(self.partition, 0) # advance offset self.consumer.offsets = {self.partition: 1} raise Exception("Intentional failure") # offset rolled back (ignoring mark) self.assertEqual(self.consumer.offsets, {self.partition: 0}) # and seek called with relative zero delta self.assertEqual(self.consumer.seek.call_count, 1) self.assertEqual(self.consumer.seek.call_args[0], (0, 1)) def test_out_of_range(self): """ Should reset to beginning of valid offsets on `OffsetOutOfRangeError` """ def _seek(offset, whence): # seek must be called with 0, 0 to find the beginning of the range self.assertEqual(offset, 0) self.assertEqual(whence, 0) # set offsets to something different self.consumer.offsets = {self.partition: 100} with patch.object(self.consumer, "seek", _seek): with self.context: raise OffsetOutOfRangeError() self.assertEqual(self.consumer.offsets, {self.partition: 100}) kafka-1.3.2/test/test_coordinator.py0000644001271300127130000006042013025302127017270 0ustar dpowers00000000000000# pylint: skip-file from __future__ import absolute_import import time import pytest from kafka.client_async import KafkaClient from kafka.structs import TopicPartition, OffsetAndMetadata from kafka.consumer.subscription_state import ( SubscriptionState, ConsumerRebalanceListener) from kafka.coordinator.assignors.range import RangePartitionAssignor from kafka.coordinator.assignors.roundrobin import RoundRobinPartitionAssignor from kafka.coordinator.consumer import ConsumerCoordinator from kafka.coordinator.protocol import ( ConsumerProtocolMemberMetadata, ConsumerProtocolMemberAssignment) import kafka.errors as Errors from kafka.future import Future from kafka.metrics import Metrics from kafka.protocol.commit import ( OffsetCommitRequest, OffsetCommitResponse, OffsetFetchRequest, OffsetFetchResponse) from kafka.protocol.metadata import MetadataResponse from kafka.util import WeakMethod @pytest.fixture def client(conn): return KafkaClient(api_version=(0, 9)) @pytest.fixture def coordinator(client): return ConsumerCoordinator(client, SubscriptionState(), Metrics()) def test_init(client, coordinator): # metadata update on init assert client.cluster._need_update is True assert WeakMethod(coordinator._handle_metadata_update) in client.cluster._listeners @pytest.mark.parametrize("api_version", [(0, 8, 0), (0, 8, 1), (0, 8, 2), (0, 9)]) def test_autocommit_enable_api_version(client, api_version): coordinator = ConsumerCoordinator(client, SubscriptionState(), Metrics(), enable_auto_commit=True, group_id='foobar', api_version=api_version) if api_version < (0, 8, 1): assert coordinator._auto_commit_task is None assert coordinator.config['enable_auto_commit'] is False else: assert coordinator._auto_commit_task is not None assert coordinator.config['enable_auto_commit'] is True def test_protocol_type(coordinator): assert coordinator.protocol_type() is 'consumer' def test_group_protocols(coordinator): # Requires a subscription try: coordinator.group_protocols() except AssertionError: pass else: assert False, 'Exception not raised when expected' coordinator._subscription.subscribe(topics=['foobar']) assert coordinator.group_protocols() == [ ('range', ConsumerProtocolMemberMetadata( RangePartitionAssignor.version, ['foobar'], b'')), ('roundrobin', ConsumerProtocolMemberMetadata( RoundRobinPartitionAssignor.version, ['foobar'], b'')), ] @pytest.mark.parametrize('api_version', [(0, 8, 0), (0, 8, 1), (0, 8, 2), (0, 9)]) def test_pattern_subscription(coordinator, api_version): coordinator.config['api_version'] = api_version coordinator._subscription.subscribe(pattern='foo') assert coordinator._subscription.subscription == set([]) assert coordinator._subscription_metadata_changed({}) is False assert coordinator._subscription.needs_partition_assignment is False cluster = coordinator._client.cluster cluster.update_metadata(MetadataResponse[0]( # brokers [(0, 'foo', 12), (1, 'bar', 34)], # topics [(0, 'fizz', []), (0, 'foo1', [(0, 0, 0, [], [])]), (0, 'foo2', [(0, 0, 1, [], [])])])) assert coordinator._subscription.subscription == set(['foo1', 'foo2']) # 0.9 consumers should trigger dynamic partition assignment if api_version >= (0, 9): assert coordinator._subscription.needs_partition_assignment is True assert coordinator._subscription.assignment == {} # earlier consumers get all partitions assigned locally else: assert coordinator._subscription.needs_partition_assignment is False assert set(coordinator._subscription.assignment.keys()) == set([ TopicPartition('foo1', 0), TopicPartition('foo2', 0)]) def test_lookup_assignor(coordinator): assert coordinator._lookup_assignor('roundrobin') is RoundRobinPartitionAssignor assert coordinator._lookup_assignor('range') is RangePartitionAssignor assert coordinator._lookup_assignor('foobar') is None def test_join_complete(mocker, coordinator): coordinator._subscription.subscribe(topics=['foobar']) assignor = RoundRobinPartitionAssignor() coordinator.config['assignors'] = (assignor,) mocker.spy(assignor, 'on_assignment') assert assignor.on_assignment.call_count == 0 assignment = ConsumerProtocolMemberAssignment(0, [('foobar', [0, 1])], b'') coordinator._on_join_complete( 0, 'member-foo', 'roundrobin', assignment.encode()) assert assignor.on_assignment.call_count == 1 assignor.on_assignment.assert_called_with(assignment) def test_subscription_listener(mocker, coordinator): listener = mocker.MagicMock(spec=ConsumerRebalanceListener) coordinator._subscription.subscribe( topics=['foobar'], listener=listener) coordinator._on_join_prepare(0, 'member-foo') assert listener.on_partitions_revoked.call_count == 1 listener.on_partitions_revoked.assert_called_with(set([])) assignment = ConsumerProtocolMemberAssignment(0, [('foobar', [0, 1])], b'') coordinator._on_join_complete( 0, 'member-foo', 'roundrobin', assignment.encode()) assert listener.on_partitions_assigned.call_count == 1 listener.on_partitions_assigned.assert_called_with(set([ TopicPartition('foobar', 0), TopicPartition('foobar', 1)])) def test_subscription_listener_failure(mocker, coordinator): listener = mocker.MagicMock(spec=ConsumerRebalanceListener) coordinator._subscription.subscribe( topics=['foobar'], listener=listener) # exception raised in listener should not be re-raised by coordinator listener.on_partitions_revoked.side_effect = Exception('crash') coordinator._on_join_prepare(0, 'member-foo') assert listener.on_partitions_revoked.call_count == 1 assignment = ConsumerProtocolMemberAssignment(0, [('foobar', [0, 1])], b'') coordinator._on_join_complete( 0, 'member-foo', 'roundrobin', assignment.encode()) assert listener.on_partitions_assigned.call_count == 1 def test_perform_assignment(mocker, coordinator): member_metadata = { 'member-foo': ConsumerProtocolMemberMetadata(0, ['foo1'], b''), 'member-bar': ConsumerProtocolMemberMetadata(0, ['foo1'], b'') } assignments = { 'member-foo': ConsumerProtocolMemberAssignment( 0, [('foo1', [0])], b''), 'member-bar': ConsumerProtocolMemberAssignment( 0, [('foo1', [1])], b'') } mocker.patch.object(RoundRobinPartitionAssignor, 'assign') RoundRobinPartitionAssignor.assign.return_value = assignments ret = coordinator._perform_assignment( 'member-foo', 'roundrobin', [(member, metadata.encode()) for member, metadata in member_metadata.items()]) assert RoundRobinPartitionAssignor.assign.call_count == 1 RoundRobinPartitionAssignor.assign.assert_called_with( coordinator._client.cluster, member_metadata) assert ret == assignments def test_on_join_prepare(coordinator): coordinator._subscription.subscribe(topics=['foobar']) coordinator._on_join_prepare(0, 'member-foo') assert coordinator._subscription.needs_partition_assignment is True def test_need_rejoin(coordinator): # No subscription - no rejoin assert coordinator.need_rejoin() is False coordinator._subscription.subscribe(topics=['foobar']) assert coordinator.need_rejoin() is True coordinator._subscription.needs_partition_assignment = False coordinator.rejoin_needed = False assert coordinator.need_rejoin() is False coordinator._subscription.needs_partition_assignment = True assert coordinator.need_rejoin() is True def test_refresh_committed_offsets_if_needed(mocker, coordinator): mocker.patch.object(ConsumerCoordinator, 'fetch_committed_offsets', return_value = { TopicPartition('foobar', 0): OffsetAndMetadata(123, b''), TopicPartition('foobar', 1): OffsetAndMetadata(234, b'')}) coordinator._subscription.assign_from_user([TopicPartition('foobar', 0)]) assert coordinator._subscription.needs_fetch_committed_offsets is True coordinator.refresh_committed_offsets_if_needed() assignment = coordinator._subscription.assignment assert assignment[TopicPartition('foobar', 0)].committed == 123 assert TopicPartition('foobar', 1) not in assignment assert coordinator._subscription.needs_fetch_committed_offsets is False def test_fetch_committed_offsets(mocker, coordinator): # No partitions, no IO polling mocker.patch.object(coordinator._client, 'poll') assert coordinator.fetch_committed_offsets([]) == {} assert coordinator._client.poll.call_count == 0 # general case -- send offset fetch request, get successful future mocker.patch.object(coordinator, 'ensure_coordinator_known') mocker.patch.object(coordinator, '_send_offset_fetch_request', return_value=Future().success('foobar')) partitions = [TopicPartition('foobar', 0)] ret = coordinator.fetch_committed_offsets(partitions) assert ret == 'foobar' coordinator._send_offset_fetch_request.assert_called_with(partitions) assert coordinator._client.poll.call_count == 1 # Failed future is raised if not retriable coordinator._send_offset_fetch_request.return_value = Future().failure(AssertionError) coordinator._client.poll.reset_mock() try: coordinator.fetch_committed_offsets(partitions) except AssertionError: pass else: assert False, 'Exception not raised when expected' assert coordinator._client.poll.call_count == 1 coordinator._client.poll.reset_mock() coordinator._send_offset_fetch_request.side_effect = [ Future().failure(Errors.RequestTimedOutError), Future().success('fizzbuzz')] ret = coordinator.fetch_committed_offsets(partitions) assert ret == 'fizzbuzz' assert coordinator._client.poll.call_count == 2 # call + retry def test_close(mocker, coordinator): mocker.patch.object(coordinator, '_maybe_auto_commit_offsets_sync') mocker.patch.object(coordinator, '_handle_leave_group_response') mocker.patch.object(coordinator, 'coordinator_unknown', return_value=False) coordinator.coordinator_id = 0 coordinator.generation = 1 cli = coordinator._client mocker.patch.object(cli, 'unschedule') mocker.patch.object(cli, 'send', return_value=Future().success('foobar')) mocker.patch.object(cli, 'poll') coordinator.close() assert coordinator._maybe_auto_commit_offsets_sync.call_count == 1 cli.unschedule.assert_called_with(coordinator.heartbeat_task) coordinator._handle_leave_group_response.assert_called_with('foobar') assert coordinator.generation == -1 assert coordinator.member_id == '' assert coordinator.rejoin_needed is True @pytest.fixture def offsets(): return { TopicPartition('foobar', 0): OffsetAndMetadata(123, b''), TopicPartition('foobar', 1): OffsetAndMetadata(234, b''), } def test_commit_offsets_async(mocker, coordinator, offsets): mocker.patch.object(coordinator._client, 'poll') mocker.patch.object(coordinator, 'ensure_coordinator_known') mocker.patch.object(coordinator, '_send_offset_commit_request', return_value=Future().success('fizzbuzz')) ret = coordinator.commit_offsets_async(offsets) assert isinstance(ret, Future) assert coordinator._send_offset_commit_request.call_count == 1 def test_commit_offsets_sync(mocker, coordinator, offsets): mocker.patch.object(coordinator, 'ensure_coordinator_known') mocker.patch.object(coordinator, '_send_offset_commit_request', return_value=Future().success('fizzbuzz')) cli = coordinator._client mocker.patch.object(cli, 'poll') # No offsets, no calls assert coordinator.commit_offsets_sync({}) is None assert coordinator._send_offset_commit_request.call_count == 0 assert cli.poll.call_count == 0 ret = coordinator.commit_offsets_sync(offsets) assert coordinator._send_offset_commit_request.call_count == 1 assert cli.poll.call_count == 1 assert ret == 'fizzbuzz' # Failed future is raised if not retriable coordinator._send_offset_commit_request.return_value = Future().failure(AssertionError) coordinator._client.poll.reset_mock() try: coordinator.commit_offsets_sync(offsets) except AssertionError: pass else: assert False, 'Exception not raised when expected' assert coordinator._client.poll.call_count == 1 coordinator._client.poll.reset_mock() coordinator._send_offset_commit_request.side_effect = [ Future().failure(Errors.RequestTimedOutError), Future().success('fizzbuzz')] ret = coordinator.commit_offsets_sync(offsets) assert ret == 'fizzbuzz' assert coordinator._client.poll.call_count == 2 # call + retry @pytest.mark.parametrize( 'api_version,group_id,enable,error,has_auto_commit,commit_offsets,warn,exc', [ ((0, 8, 0), 'foobar', True, None, False, False, True, False), ((0, 8, 1), 'foobar', True, None, True, True, False, False), ((0, 8, 2), 'foobar', True, None, True, True, False, False), ((0, 9), 'foobar', False, None, False, False, False, False), ((0, 9), 'foobar', True, Errors.UnknownMemberIdError(), True, True, True, False), ((0, 9), 'foobar', True, Errors.IllegalGenerationError(), True, True, True, False), ((0, 9), 'foobar', True, Errors.RebalanceInProgressError(), True, True, True, False), ((0, 9), 'foobar', True, Exception(), True, True, False, True), ((0, 9), 'foobar', True, None, True, True, False, False), ((0, 9), None, True, None, False, False, True, False), ]) def test_maybe_auto_commit_offsets_sync(mocker, api_version, group_id, enable, error, has_auto_commit, commit_offsets, warn, exc): mock_warn = mocker.patch('kafka.coordinator.consumer.log.warning') mock_exc = mocker.patch('kafka.coordinator.consumer.log.exception') client = KafkaClient(api_version=api_version) coordinator = ConsumerCoordinator(client, SubscriptionState(), Metrics(), api_version=api_version, enable_auto_commit=enable, group_id=group_id) commit_sync = mocker.patch.object(coordinator, 'commit_offsets_sync', side_effect=error) if has_auto_commit: assert coordinator._auto_commit_task is not None else: assert coordinator._auto_commit_task is None assert coordinator._maybe_auto_commit_offsets_sync() is None if has_auto_commit: assert coordinator._auto_commit_task is not None assert commit_sync.call_count == (1 if commit_offsets else 0) assert mock_warn.call_count == (1 if warn else 0) assert mock_exc.call_count == (1 if exc else 0) @pytest.fixture def patched_coord(mocker, coordinator): coordinator._subscription.subscribe(topics=['foobar']) coordinator._subscription.needs_partition_assignment = False mocker.patch.object(coordinator, 'coordinator_unknown', return_value=False) coordinator.coordinator_id = 0 coordinator.generation = 0 mocker.patch.object(coordinator, 'need_rejoin', return_value=False) mocker.patch.object(coordinator._client, 'least_loaded_node', return_value=1) mocker.patch.object(coordinator._client, 'ready', return_value=True) mocker.patch.object(coordinator._client, 'send') mocker.patch.object(coordinator._client, 'schedule') mocker.spy(coordinator, '_failed_request') mocker.spy(coordinator, '_handle_offset_commit_response') mocker.spy(coordinator, '_handle_offset_fetch_response') mocker.spy(coordinator.heartbeat_task, '_handle_heartbeat_success') mocker.spy(coordinator.heartbeat_task, '_handle_heartbeat_failure') return coordinator def test_send_offset_commit_request_fail(patched_coord, offsets): patched_coord.coordinator_unknown.return_value = True patched_coord.coordinator_id = None # No offsets ret = patched_coord._send_offset_commit_request({}) assert isinstance(ret, Future) assert ret.succeeded() # No coordinator ret = patched_coord._send_offset_commit_request(offsets) assert ret.failed() assert isinstance(ret.exception, Errors.GroupCoordinatorNotAvailableError) @pytest.mark.parametrize('api_version,req_type', [ ((0, 8, 1), OffsetCommitRequest[0]), ((0, 8, 2), OffsetCommitRequest[1]), ((0, 9), OffsetCommitRequest[2])]) def test_send_offset_commit_request_versions(patched_coord, offsets, api_version, req_type): expect_node = 0 patched_coord.config['api_version'] = api_version patched_coord._send_offset_commit_request(offsets) (node, request), _ = patched_coord._client.send.call_args assert node == expect_node, 'Unexpected coordinator node' assert isinstance(request, req_type) def test_send_offset_commit_request_failure(patched_coord, offsets): _f = Future() patched_coord._client.send.return_value = _f future = patched_coord._send_offset_commit_request(offsets) (node, request), _ = patched_coord._client.send.call_args error = Exception() _f.failure(error) patched_coord._failed_request.assert_called_with(0, request, future, error) assert future.failed() assert future.exception is error def test_send_offset_commit_request_success(mocker, patched_coord, offsets): _f = Future() patched_coord._client.send.return_value = _f future = patched_coord._send_offset_commit_request(offsets) (node, request), _ = patched_coord._client.send.call_args response = OffsetCommitResponse[0]([('foobar', [(0, 0), (1, 0)])]) _f.success(response) patched_coord._handle_offset_commit_response.assert_called_with( offsets, future, mocker.ANY, response) @pytest.mark.parametrize('response,error,dead,reassign', [ (OffsetCommitResponse[0]([('foobar', [(0, 30), (1, 30)])]), Errors.GroupAuthorizationFailedError, False, False), (OffsetCommitResponse[0]([('foobar', [(0, 12), (1, 12)])]), Errors.OffsetMetadataTooLargeError, False, False), (OffsetCommitResponse[0]([('foobar', [(0, 28), (1, 28)])]), Errors.InvalidCommitOffsetSizeError, False, False), (OffsetCommitResponse[0]([('foobar', [(0, 14), (1, 14)])]), Errors.GroupLoadInProgressError, False, False), (OffsetCommitResponse[0]([('foobar', [(0, 15), (1, 15)])]), Errors.GroupCoordinatorNotAvailableError, True, False), (OffsetCommitResponse[0]([('foobar', [(0, 16), (1, 16)])]), Errors.NotCoordinatorForGroupError, True, False), (OffsetCommitResponse[0]([('foobar', [(0, 7), (1, 7)])]), Errors.RequestTimedOutError, True, False), (OffsetCommitResponse[0]([('foobar', [(0, 25), (1, 25)])]), Errors.CommitFailedError, False, True), (OffsetCommitResponse[0]([('foobar', [(0, 22), (1, 22)])]), Errors.CommitFailedError, False, True), (OffsetCommitResponse[0]([('foobar', [(0, 27), (1, 27)])]), Errors.CommitFailedError, False, True), (OffsetCommitResponse[0]([('foobar', [(0, 17), (1, 17)])]), Errors.InvalidTopicError, False, False), (OffsetCommitResponse[0]([('foobar', [(0, 29), (1, 29)])]), Errors.TopicAuthorizationFailedError, False, False), ]) def test_handle_offset_commit_response(mocker, patched_coord, offsets, response, error, dead, reassign): future = Future() patched_coord._handle_offset_commit_response(offsets, future, time.time(), response) assert isinstance(future.exception, error) assert patched_coord.coordinator_id is (None if dead else 0) assert patched_coord._subscription.needs_partition_assignment is reassign @pytest.fixture def partitions(): return [TopicPartition('foobar', 0), TopicPartition('foobar', 1)] def test_send_offset_fetch_request_fail(patched_coord, partitions): patched_coord.coordinator_unknown.return_value = True patched_coord.coordinator_id = None # No partitions ret = patched_coord._send_offset_fetch_request([]) assert isinstance(ret, Future) assert ret.succeeded() assert ret.value == {} # No coordinator ret = patched_coord._send_offset_fetch_request(partitions) assert ret.failed() assert isinstance(ret.exception, Errors.GroupCoordinatorNotAvailableError) @pytest.mark.parametrize('api_version,req_type', [ ((0, 8, 1), OffsetFetchRequest[0]), ((0, 8, 2), OffsetFetchRequest[1]), ((0, 9), OffsetFetchRequest[1])]) def test_send_offset_fetch_request_versions(patched_coord, partitions, api_version, req_type): # assuming fixture sets coordinator=0, least_loaded_node=1 expect_node = 0 patched_coord.config['api_version'] = api_version patched_coord._send_offset_fetch_request(partitions) (node, request), _ = patched_coord._client.send.call_args assert node == expect_node, 'Unexpected coordinator node' assert isinstance(request, req_type) def test_send_offset_fetch_request_failure(patched_coord, partitions): _f = Future() patched_coord._client.send.return_value = _f future = patched_coord._send_offset_fetch_request(partitions) (node, request), _ = patched_coord._client.send.call_args error = Exception() _f.failure(error) patched_coord._failed_request.assert_called_with(0, request, future, error) assert future.failed() assert future.exception is error def test_send_offset_fetch_request_success(patched_coord, partitions): _f = Future() patched_coord._client.send.return_value = _f future = patched_coord._send_offset_fetch_request(partitions) (node, request), _ = patched_coord._client.send.call_args response = OffsetFetchResponse[0]([('foobar', [(0, 123, b'', 0), (1, 234, b'', 0)])]) _f.success(response) patched_coord._handle_offset_fetch_response.assert_called_with( future, response) @pytest.mark.parametrize('response,error,dead,reassign', [ #(OffsetFetchResponse[0]([('foobar', [(0, 123, b'', 30), (1, 234, b'', 30)])]), # Errors.GroupAuthorizationFailedError, False, False), #(OffsetFetchResponse[0]([('foobar', [(0, 123, b'', 7), (1, 234, b'', 7)])]), # Errors.RequestTimedOutError, True, False), #(OffsetFetchResponse[0]([('foobar', [(0, 123, b'', 27), (1, 234, b'', 27)])]), # Errors.RebalanceInProgressError, False, True), (OffsetFetchResponse[0]([('foobar', [(0, 123, b'', 14), (1, 234, b'', 14)])]), Errors.GroupLoadInProgressError, False, False), (OffsetFetchResponse[0]([('foobar', [(0, 123, b'', 16), (1, 234, b'', 16)])]), Errors.NotCoordinatorForGroupError, True, False), (OffsetFetchResponse[0]([('foobar', [(0, 123, b'', 25), (1, 234, b'', 25)])]), Errors.UnknownMemberIdError, False, True), (OffsetFetchResponse[0]([('foobar', [(0, 123, b'', 22), (1, 234, b'', 22)])]), Errors.IllegalGenerationError, False, True), (OffsetFetchResponse[0]([('foobar', [(0, 123, b'', 29), (1, 234, b'', 29)])]), Errors.TopicAuthorizationFailedError, False, False), (OffsetFetchResponse[0]([('foobar', [(0, 123, b'', 0), (1, 234, b'', 0)])]), None, False, False), ]) def test_handle_offset_fetch_response(patched_coord, offsets, response, error, dead, reassign): future = Future() patched_coord._handle_offset_fetch_response(future, response) if error is not None: assert isinstance(future.exception, error) else: assert future.succeeded() assert future.value == offsets assert patched_coord.coordinator_id is (None if dead else 0) assert patched_coord._subscription.needs_partition_assignment is reassign def test_heartbeat(patched_coord): patched_coord.coordinator_unknown.return_value = True patched_coord.heartbeat_task() assert patched_coord._client.schedule.call_count == 1 assert patched_coord.heartbeat_task._handle_heartbeat_failure.call_count == 1 kafka-1.3.2/test/test_failover_integration.py0000644001271300127130000002223713031057471021171 0ustar dpowers00000000000000import logging import os import time from kafka import SimpleClient, SimpleConsumer, KeyedProducer from kafka.errors import ( FailedPayloadsError, ConnectionError, RequestTimedOutError, NotLeaderForPartitionError) from kafka.producer.base import Producer from kafka.structs import TopicPartition from test.fixtures import ZookeeperFixture, KafkaFixture from test.testutil import KafkaIntegrationTestCase, random_string log = logging.getLogger(__name__) class TestFailover(KafkaIntegrationTestCase): create_client = False def setUp(self): if not os.environ.get('KAFKA_VERSION'): self.skipTest('integration test requires KAFKA_VERSION') zk_chroot = random_string(10) replicas = 3 partitions = 3 # mini zookeeper, 3 kafka brokers self.zk = ZookeeperFixture.instance() kk_args = [self.zk.host, self.zk.port] kk_kwargs = {'zk_chroot': zk_chroot, 'replicas': replicas, 'partitions': partitions} self.brokers = [KafkaFixture.instance(i, *kk_args, **kk_kwargs) for i in range(replicas)] hosts = ['%s:%d' % (b.host, b.port) for b in self.brokers] self.client = SimpleClient(hosts, timeout=2) super(TestFailover, self).setUp() def tearDown(self): super(TestFailover, self).tearDown() if not os.environ.get('KAFKA_VERSION'): return self.client.close() for broker in self.brokers: broker.close() self.zk.close() def test_switch_leader(self): topic = self.topic partition = 0 # Testing the base Producer class here so that we can easily send # messages to a specific partition, kill the leader for that partition # and check that after another broker takes leadership the producer # is able to resume sending messages # require that the server commit messages to all in-sync replicas # so that failover doesn't lose any messages on server-side # and we can assert that server-side message count equals client-side producer = Producer(self.client, async=False, req_acks=Producer.ACK_AFTER_CLUSTER_COMMIT) # Send 100 random messages to a specific partition self._send_random_messages(producer, topic, partition, 100) # kill leader for partition self._kill_leader(topic, partition) # expect failure, but don't wait more than 60 secs to recover recovered = False started = time.time() timeout = 60 while not recovered and (time.time() - started) < timeout: try: log.debug("attempting to send 'success' message after leader killed") producer.send_messages(topic, partition, b'success') log.debug("success!") recovered = True except (FailedPayloadsError, ConnectionError, RequestTimedOutError, NotLeaderForPartitionError): log.debug("caught exception sending message -- will retry") continue # Verify we successfully sent the message self.assertTrue(recovered) # send some more messages to new leader self._send_random_messages(producer, topic, partition, 100) # count number of messages # Should be equal to 100 before + 1 recovery + 100 after # at_least=True because exactly once delivery isn't really a thing self.assert_message_count(topic, 201, partitions=(partition,), at_least=True) def test_switch_leader_async(self): topic = self.topic partition = 0 # Test the base class Producer -- send_messages to a specific partition producer = Producer(self.client, async=True, batch_send_every_n=15, batch_send_every_t=3, req_acks=Producer.ACK_AFTER_CLUSTER_COMMIT, async_log_messages_on_error=False) # Send 10 random messages self._send_random_messages(producer, topic, partition, 10) self._send_random_messages(producer, topic, partition + 1, 10) # kill leader for partition self._kill_leader(topic, partition) log.debug("attempting to send 'success' message after leader killed") # in async mode, this should return immediately producer.send_messages(topic, partition, b'success') producer.send_messages(topic, partition + 1, b'success') # send to new leader self._send_random_messages(producer, topic, partition, 10) self._send_random_messages(producer, topic, partition + 1, 10) # Stop the producer and wait for it to shutdown producer.stop() started = time.time() timeout = 60 while (time.time() - started) < timeout: if not producer.thread.is_alive(): break time.sleep(0.1) else: self.fail('timeout waiting for producer queue to empty') # count number of messages # Should be equal to 10 before + 1 recovery + 10 after # at_least=True because exactly once delivery isn't really a thing self.assert_message_count(topic, 21, partitions=(partition,), at_least=True) self.assert_message_count(topic, 21, partitions=(partition + 1,), at_least=True) def test_switch_leader_keyed_producer(self): topic = self.topic producer = KeyedProducer(self.client, async=False) # Send 10 random messages for _ in range(10): key = random_string(3).encode('utf-8') msg = random_string(10).encode('utf-8') producer.send_messages(topic, key, msg) # kill leader for partition 0 self._kill_leader(topic, 0) recovered = False started = time.time() timeout = 60 while not recovered and (time.time() - started) < timeout: try: key = random_string(3).encode('utf-8') msg = random_string(10).encode('utf-8') producer.send_messages(topic, key, msg) if producer.partitioners[topic].partition(key) == 0: recovered = True except (FailedPayloadsError, ConnectionError, RequestTimedOutError, NotLeaderForPartitionError): log.debug("caught exception sending message -- will retry") continue # Verify we successfully sent the message self.assertTrue(recovered) # send some more messages just to make sure no more exceptions for _ in range(10): key = random_string(3).encode('utf-8') msg = random_string(10).encode('utf-8') producer.send_messages(topic, key, msg) def test_switch_leader_simple_consumer(self): producer = Producer(self.client, async=False) consumer = SimpleConsumer(self.client, None, self.topic, partitions=None, auto_commit=False, iter_timeout=10) self._send_random_messages(producer, self.topic, 0, 2) consumer.get_messages() self._kill_leader(self.topic, 0) consumer.get_messages() def _send_random_messages(self, producer, topic, partition, n): for j in range(n): msg = 'msg {0}: {1}'.format(j, random_string(10)) log.debug('_send_random_message %s to %s:%d', msg, topic, partition) while True: try: producer.send_messages(topic, partition, msg.encode('utf-8')) except: log.exception('failure in _send_random_messages - retrying') continue else: break def _kill_leader(self, topic, partition): leader = self.client.topics_to_brokers[TopicPartition(topic, partition)] broker = self.brokers[leader.nodeId] broker.close() return broker def assert_message_count(self, topic, check_count, timeout=10, partitions=None, at_least=False): hosts = ','.join(['%s:%d' % (broker.host, broker.port) for broker in self.brokers]) client = SimpleClient(hosts, timeout=2) consumer = SimpleConsumer(client, None, topic, partitions=partitions, auto_commit=False, iter_timeout=timeout) started_at = time.time() pending = -1 while pending < check_count and (time.time() - started_at < timeout): try: pending = consumer.pending(partitions) except FailedPayloadsError: pass time.sleep(0.5) consumer.stop() client.close() if pending < check_count: self.fail('Too few pending messages: found %d, expected %d' % (pending, check_count)) elif pending > check_count and not at_least: self.fail('Too many pending messages: found %d, expected %d' % (pending, check_count)) return True kafka-1.3.2/test/test_fetcher.py0000644001271300127130000000731113031057471016373 0ustar dpowers00000000000000# pylint: skip-file from __future__ import absolute_import import pytest from kafka.client_async import KafkaClient from kafka.consumer.fetcher import Fetcher from kafka.consumer.subscription_state import SubscriptionState import kafka.errors as Errors from kafka.future import Future from kafka.metrics import Metrics from kafka.protocol.fetch import FetchRequest from kafka.structs import TopicPartition, OffsetAndMetadata @pytest.fixture def client(mocker): return mocker.Mock(spec=KafkaClient(bootstrap_servers=[], api_version=(0, 9))) @pytest.fixture def subscription_state(): return SubscriptionState() @pytest.fixture def fetcher(client, subscription_state): subscription_state.subscribe(topics=['foobar']) assignment = [TopicPartition('foobar', i) for i in range(3)] subscription_state.assign_from_subscribed(assignment) for tp in assignment: subscription_state.seek(tp, 0) return Fetcher(client, subscription_state, Metrics()) def test_send_fetches(fetcher, mocker): fetch_requests = [ FetchRequest[0]( -1, fetcher.config['fetch_max_wait_ms'], fetcher.config['fetch_min_bytes'], [('foobar', [ (0, 0, fetcher.config['max_partition_fetch_bytes']), (1, 0, fetcher.config['max_partition_fetch_bytes']), ])]), FetchRequest[0]( -1, fetcher.config['fetch_max_wait_ms'], fetcher.config['fetch_min_bytes'], [('foobar', [ (2, 0, fetcher.config['max_partition_fetch_bytes']), ])]) ] mocker.patch.object(fetcher, '_create_fetch_requests', return_value = dict(enumerate(fetch_requests))) ret = fetcher.send_fetches() for node, request in enumerate(fetch_requests): fetcher._client.send.assert_any_call(node, request) assert len(ret) == len(fetch_requests) @pytest.mark.parametrize(("api_version", "fetch_version"), [ ((0, 10), 2), ((0, 9), 1), ((0, 8), 0) ]) def test_create_fetch_requests(fetcher, mocker, api_version, fetch_version): fetcher._client.in_flight_request_count.return_value = 0 fetcher.config['api_version'] = api_version by_node = fetcher._create_fetch_requests() requests = by_node.values() assert all([isinstance(r, FetchRequest[fetch_version]) for r in requests]) def test_update_fetch_positions(fetcher, mocker): mocker.patch.object(fetcher, '_reset_offset') partition = TopicPartition('foobar', 0) # unassigned partition fetcher.update_fetch_positions([TopicPartition('fizzbuzz', 0)]) assert fetcher._reset_offset.call_count == 0 # fetchable partition (has offset, not paused) fetcher.update_fetch_positions([partition]) assert fetcher._reset_offset.call_count == 0 # partition needs reset, no committed offset fetcher._subscriptions.need_offset_reset(partition) fetcher._subscriptions.assignment[partition].awaiting_reset = False fetcher.update_fetch_positions([partition]) fetcher._reset_offset.assert_called_with(partition) assert fetcher._subscriptions.assignment[partition].awaiting_reset is True fetcher.update_fetch_positions([partition]) fetcher._reset_offset.assert_called_with(partition) # partition needs reset, has committed offset fetcher._reset_offset.reset_mock() fetcher._subscriptions.need_offset_reset(partition) fetcher._subscriptions.assignment[partition].awaiting_reset = False fetcher._subscriptions.assignment[partition].committed = 123 mocker.patch.object(fetcher._subscriptions, 'seek') fetcher.update_fetch_positions([partition]) assert fetcher._reset_offset.call_count == 0 fetcher._subscriptions.seek.assert_called_with(partition, 123) kafka-1.3.2/test/test_metrics.py0000644001271300127130000004624612742775745016456 0ustar dpowers00000000000000import sys import time import pytest from kafka.errors import QuotaViolationError from kafka.metrics import DictReporter, MetricConfig, MetricName, Metrics, Quota from kafka.metrics.measurable import AbstractMeasurable from kafka.metrics.stats import (Avg, Count, Max, Min, Percentile, Percentiles, Rate, Total) from kafka.metrics.stats.percentiles import BucketSizing from kafka.metrics.stats.rate import TimeUnit EPS = 0.000001 @pytest.fixture def time_keeper(): return TimeKeeper() @pytest.fixture def config(): return MetricConfig() @pytest.fixture def reporter(): return DictReporter() @pytest.fixture def metrics(request, config, reporter): metrics = Metrics(config, [reporter], enable_expiration=True) request.addfinalizer(lambda: metrics.close()) return metrics def test_MetricName(): # The Java test only cover the differences between the deprecated # constructors, so I'm skipping them but doing some other basic testing. # In short, metrics should be equal IFF their name, group, and tags are # the same. Descriptions do not matter. name1 = MetricName('name', 'group', 'A metric.', {'a': 1, 'b': 2}) name2 = MetricName('name', 'group', 'A description.', {'a': 1, 'b': 2}) assert name1 == name2 name1 = MetricName('name', 'group', tags={'a': 1, 'b': 2}) name2 = MetricName('name', 'group', tags={'a': 1, 'b': 2}) assert name1 == name2 name1 = MetricName('foo', 'group') name2 = MetricName('name', 'group') assert name1 != name2 name1 = MetricName('name', 'foo') name2 = MetricName('name', 'group') assert name1 != name2 # name and group must be non-empty. Everything else is optional. with pytest.raises(Exception): MetricName('', 'group') with pytest.raises(Exception): MetricName('name', None) # tags must be a dict if supplied with pytest.raises(Exception): MetricName('name', 'group', tags=set()) # Because of the implementation of __eq__ and __hash__, the values of # a MetricName cannot be mutable. tags = {'a': 1} name = MetricName('name', 'group', 'description', tags=tags) with pytest.raises(AttributeError): name.name = 'new name' with pytest.raises(AttributeError): name.group = 'new name' with pytest.raises(AttributeError): name.tags = {} # tags is a copy, so the instance isn't altered name.tags['b'] = 2 assert name.tags == tags def test_simple_stats(mocker, time_keeper, config, metrics): mocker.patch('time.time', side_effect=time_keeper.time) measurable = ConstantMeasurable() metrics.add_metric(metrics.metric_name('direct.measurable', 'grp1', 'The fraction of time an appender waits for space allocation.'), measurable) sensor = metrics.sensor('test.sensor') sensor.add(metrics.metric_name('test.avg', 'grp1'), Avg()) sensor.add(metrics.metric_name('test.max', 'grp1'), Max()) sensor.add(metrics.metric_name('test.min', 'grp1'), Min()) sensor.add(metrics.metric_name('test.rate', 'grp1'), Rate(TimeUnit.SECONDS)) sensor.add(metrics.metric_name('test.occurences', 'grp1'),Rate(TimeUnit.SECONDS, Count())) sensor.add(metrics.metric_name('test.count', 'grp1'), Count()) percentiles = [Percentile(metrics.metric_name('test.median', 'grp1'), 50.0), Percentile(metrics.metric_name('test.perc99_9', 'grp1'), 99.9)] sensor.add_compound(Percentiles(100, BucketSizing.CONSTANT, 100, -100, percentiles=percentiles)) sensor2 = metrics.sensor('test.sensor2') sensor2.add(metrics.metric_name('s2.total', 'grp1'), Total()) sensor2.record(5.0) sum_val = 0 count = 10 for i in range(count): sensor.record(i) sum_val += i # prior to any time passing elapsed_secs = (config.time_window_ms * (config.samples - 1)) / 1000.0 assert abs(count / elapsed_secs - metrics.metrics.get(metrics.metric_name('test.occurences', 'grp1')).value()) \ < EPS, 'Occurrences(0...%d) = %f' % (count, count / elapsed_secs) # pretend 2 seconds passed... sleep_time_seconds = 2.0 time_keeper.sleep(sleep_time_seconds) elapsed_secs += sleep_time_seconds assert abs(5.0 - metrics.metrics.get(metrics.metric_name('s2.total', 'grp1')).value()) \ < EPS, 's2 reflects the constant value' assert abs(4.5 - metrics.metrics.get(metrics.metric_name('test.avg', 'grp1')).value()) \ < EPS, 'Avg(0...9) = 4.5' assert abs((count - 1) - metrics.metrics.get(metrics.metric_name('test.max', 'grp1')).value()) \ < EPS, 'Max(0...9) = 9' assert abs(0.0 - metrics.metrics.get(metrics.metric_name('test.min', 'grp1')).value()) \ < EPS, 'Min(0...9) = 0' assert abs((sum_val / elapsed_secs) - metrics.metrics.get(metrics.metric_name('test.rate', 'grp1')).value()) \ < EPS, 'Rate(0...9) = 1.40625' assert abs((count / elapsed_secs) - metrics.metrics.get(metrics.metric_name('test.occurences', 'grp1')).value()) \ < EPS, 'Occurrences(0...%d) = %f' % (count, count / elapsed_secs) assert abs(count - metrics.metrics.get(metrics.metric_name('test.count', 'grp1')).value()) \ < EPS, 'Count(0...9) = 10' def test_hierarchical_sensors(metrics): parent1 = metrics.sensor('test.parent1') parent1.add(metrics.metric_name('test.parent1.count', 'grp1'), Count()) parent2 = metrics.sensor('test.parent2') parent2.add(metrics.metric_name('test.parent2.count', 'grp1'), Count()) child1 = metrics.sensor('test.child1', parents=[parent1, parent2]) child1.add(metrics.metric_name('test.child1.count', 'grp1'), Count()) child2 = metrics.sensor('test.child2', parents=[parent1]) child2.add(metrics.metric_name('test.child2.count', 'grp1'), Count()) grandchild = metrics.sensor('test.grandchild', parents=[child1]) grandchild.add(metrics.metric_name('test.grandchild.count', 'grp1'), Count()) # increment each sensor one time parent1.record() parent2.record() child1.record() child2.record() grandchild.record() p1 = parent1.metrics[0].value() p2 = parent2.metrics[0].value() c1 = child1.metrics[0].value() c2 = child2.metrics[0].value() gc = grandchild.metrics[0].value() # each metric should have a count equal to one + its children's count assert 1.0 == gc assert 1.0 + gc == c1 assert 1.0 == c2 assert 1.0 + c1 == p2 assert 1.0 + c1 + c2 == p1 assert [child1, child2] == metrics._children_sensors.get(parent1) assert [child1] == metrics._children_sensors.get(parent2) assert metrics._children_sensors.get(grandchild) is None def test_bad_sensor_hierarchy(metrics): parent = metrics.sensor('parent') child1 = metrics.sensor('child1', parents=[parent]) child2 = metrics.sensor('child2', parents=[parent]) with pytest.raises(ValueError): metrics.sensor('gc', parents=[child1, child2]) def test_remove_sensor(metrics): size = len(metrics.metrics) parent1 = metrics.sensor('test.parent1') parent1.add(metrics.metric_name('test.parent1.count', 'grp1'), Count()) parent2 = metrics.sensor('test.parent2') parent2.add(metrics.metric_name('test.parent2.count', 'grp1'), Count()) child1 = metrics.sensor('test.child1', parents=[parent1, parent2]) child1.add(metrics.metric_name('test.child1.count', 'grp1'), Count()) child2 = metrics.sensor('test.child2', parents=[parent2]) child2.add(metrics.metric_name('test.child2.count', 'grp1'), Count()) grandchild1 = metrics.sensor('test.gchild2', parents=[child2]) grandchild1.add(metrics.metric_name('test.gchild2.count', 'grp1'), Count()) sensor = metrics.get_sensor('test.parent1') assert sensor is not None metrics.remove_sensor('test.parent1') assert metrics.get_sensor('test.parent1') is None assert metrics.metrics.get(metrics.metric_name('test.parent1.count', 'grp1')) is None assert metrics.get_sensor('test.child1') is None assert metrics._children_sensors.get(sensor) is None assert metrics.metrics.get(metrics.metric_name('test.child1.count', 'grp1')) is None sensor = metrics.get_sensor('test.gchild2') assert sensor is not None metrics.remove_sensor('test.gchild2') assert metrics.get_sensor('test.gchild2') is None assert metrics._children_sensors.get(sensor) is None assert metrics.metrics.get(metrics.metric_name('test.gchild2.count', 'grp1')) is None sensor = metrics.get_sensor('test.child2') assert sensor is not None metrics.remove_sensor('test.child2') assert metrics.get_sensor('test.child2') is None assert metrics._children_sensors.get(sensor) is None assert metrics.metrics.get(metrics.metric_name('test.child2.count', 'grp1')) is None sensor = metrics.get_sensor('test.parent2') assert sensor is not None metrics.remove_sensor('test.parent2') assert metrics.get_sensor('test.parent2') is None assert metrics._children_sensors.get(sensor) is None assert metrics.metrics.get(metrics.metric_name('test.parent2.count', 'grp1')) is None assert size == len(metrics.metrics) def test_remove_inactive_metrics(mocker, time_keeper, metrics): mocker.patch('time.time', side_effect=time_keeper.time) s1 = metrics.sensor('test.s1', None, 1) s1.add(metrics.metric_name('test.s1.count', 'grp1'), Count()) s2 = metrics.sensor('test.s2', None, 3) s2.add(metrics.metric_name('test.s2.count', 'grp1'), Count()) purger = Metrics.ExpireSensorTask purger.run(metrics) assert metrics.get_sensor('test.s1') is not None, \ 'Sensor test.s1 must be present' assert metrics.metrics.get(metrics.metric_name('test.s1.count', 'grp1')) is not None, \ 'MetricName test.s1.count must be present' assert metrics.get_sensor('test.s2') is not None, \ 'Sensor test.s2 must be present' assert metrics.metrics.get(metrics.metric_name('test.s2.count', 'grp1')) is not None, \ 'MetricName test.s2.count must be present' time_keeper.sleep(1.001) purger.run(metrics) assert metrics.get_sensor('test.s1') is None, \ 'Sensor test.s1 should have been purged' assert metrics.metrics.get(metrics.metric_name('test.s1.count', 'grp1')) is None, \ 'MetricName test.s1.count should have been purged' assert metrics.get_sensor('test.s2') is not None, \ 'Sensor test.s2 must be present' assert metrics.metrics.get(metrics.metric_name('test.s2.count', 'grp1')) is not None, \ 'MetricName test.s2.count must be present' # record a value in sensor s2. This should reset the clock for that sensor. # It should not get purged at the 3 second mark after creation s2.record() time_keeper.sleep(2) purger.run(metrics) assert metrics.get_sensor('test.s2') is not None, \ 'Sensor test.s2 must be present' assert metrics.metrics.get(metrics.metric_name('test.s2.count', 'grp1')) is not None, \ 'MetricName test.s2.count must be present' # After another 1 second sleep, the metric should be purged time_keeper.sleep(1) purger.run(metrics) assert metrics.get_sensor('test.s1') is None, \ 'Sensor test.s2 should have been purged' assert metrics.metrics.get(metrics.metric_name('test.s1.count', 'grp1')) is None, \ 'MetricName test.s2.count should have been purged' # After purging, it should be possible to recreate a metric s1 = metrics.sensor('test.s1', None, 1) s1.add(metrics.metric_name('test.s1.count', 'grp1'), Count()) assert metrics.get_sensor('test.s1') is not None, \ 'Sensor test.s1 must be present' assert metrics.metrics.get(metrics.metric_name('test.s1.count', 'grp1')) is not None, \ 'MetricName test.s1.count must be present' def test_remove_metric(metrics): size = len(metrics.metrics) metrics.add_metric(metrics.metric_name('test1', 'grp1'), Count()) metrics.add_metric(metrics.metric_name('test2', 'grp1'), Count()) assert metrics.remove_metric(metrics.metric_name('test1', 'grp1')) is not None assert metrics.metrics.get(metrics.metric_name('test1', 'grp1')) is None assert metrics.metrics.get(metrics.metric_name('test2', 'grp1')) is not None assert metrics.remove_metric(metrics.metric_name('test2', 'grp1')) is not None assert metrics.metrics.get(metrics.metric_name('test2', 'grp1')) is None assert size == len(metrics.metrics) def test_event_windowing(mocker, time_keeper): mocker.patch('time.time', side_effect=time_keeper.time) count = Count() config = MetricConfig(event_window=1, samples=2) count.record(config, 1.0, time_keeper.ms()) count.record(config, 1.0, time_keeper.ms()) assert 2.0 == count.measure(config, time_keeper.ms()) count.record(config, 1.0, time_keeper.ms()) # first event times out assert 2.0 == count.measure(config, time_keeper.ms()) def test_time_windowing(mocker, time_keeper): mocker.patch('time.time', side_effect=time_keeper.time) count = Count() config = MetricConfig(time_window_ms=1, samples=2) count.record(config, 1.0, time_keeper.ms()) time_keeper.sleep(.001) count.record(config, 1.0, time_keeper.ms()) assert 2.0 == count.measure(config, time_keeper.ms()) time_keeper.sleep(.001) count.record(config, 1.0, time_keeper.ms()) # oldest event times out assert 2.0 == count.measure(config, time_keeper.ms()) def test_old_data_has_no_effect(mocker, time_keeper): mocker.patch('time.time', side_effect=time_keeper.time) max_stat = Max() min_stat = Min() avg_stat = Avg() count_stat = Count() window_ms = 100 samples = 2 config = MetricConfig(time_window_ms=window_ms, samples=samples) max_stat.record(config, 50, time_keeper.ms()) min_stat.record(config, 50, time_keeper.ms()) avg_stat.record(config, 50, time_keeper.ms()) count_stat.record(config, 50, time_keeper.ms()) time_keeper.sleep(samples * window_ms / 1000.0) assert float('-inf') == max_stat.measure(config, time_keeper.ms()) assert float(sys.maxsize) == min_stat.measure(config, time_keeper.ms()) assert 0.0 == avg_stat.measure(config, time_keeper.ms()) assert 0 == count_stat.measure(config, time_keeper.ms()) def test_duplicate_MetricName(metrics): metrics.sensor('test').add(metrics.metric_name('test', 'grp1'), Avg()) with pytest.raises(ValueError): metrics.sensor('test2').add(metrics.metric_name('test', 'grp1'), Total()) def test_Quotas(metrics): sensor = metrics.sensor('test') sensor.add(metrics.metric_name('test1.total', 'grp1'), Total(), MetricConfig(quota=Quota.upper_bound(5.0))) sensor.add(metrics.metric_name('test2.total', 'grp1'), Total(), MetricConfig(quota=Quota.lower_bound(0.0))) sensor.record(5.0) with pytest.raises(QuotaViolationError): sensor.record(1.0) assert abs(6.0 - metrics.metrics.get(metrics.metric_name('test1.total', 'grp1')).value()) \ < EPS sensor.record(-6.0) with pytest.raises(QuotaViolationError): sensor.record(-1.0) def test_Quotas_equality(): quota1 = Quota.upper_bound(10.5) quota2 = Quota.lower_bound(10.5) assert quota1 != quota2, 'Quota with different upper values should not be equal' quota3 = Quota.lower_bound(10.5) assert quota2 == quota3, 'Quota with same upper and bound values should be equal' def test_Percentiles(metrics): buckets = 100 _percentiles = [ Percentile(metrics.metric_name('test.p25', 'grp1'), 25), Percentile(metrics.metric_name('test.p50', 'grp1'), 50), Percentile(metrics.metric_name('test.p75', 'grp1'), 75), ] percs = Percentiles(4 * buckets, BucketSizing.CONSTANT, 100.0, 0.0, percentiles=_percentiles) config = MetricConfig(event_window=50, samples=2) sensor = metrics.sensor('test', config) sensor.add_compound(percs) p25 = metrics.metrics.get(metrics.metric_name('test.p25', 'grp1')) p50 = metrics.metrics.get(metrics.metric_name('test.p50', 'grp1')) p75 = metrics.metrics.get(metrics.metric_name('test.p75', 'grp1')) # record two windows worth of sequential values for i in range(buckets): sensor.record(i) assert abs(p25.value() - 25) < 1.0 assert abs(p50.value() - 50) < 1.0 assert abs(p75.value() - 75) < 1.0 for i in range(buckets): sensor.record(0.0) assert p25.value() < 1.0 assert p50.value() < 1.0 assert p75.value() < 1.0 def test_rate_windowing(mocker, time_keeper, metrics): mocker.patch('time.time', side_effect=time_keeper.time) # Use the default time window. Set 3 samples config = MetricConfig(samples=3) sensor = metrics.sensor('test.sensor', config) sensor.add(metrics.metric_name('test.rate', 'grp1'), Rate(TimeUnit.SECONDS)) sum_val = 0 count = config.samples - 1 # Advance 1 window after every record for i in range(count): sensor.record(100) sum_val += 100 time_keeper.sleep(config.time_window_ms / 1000.0) # Sleep for half the window. time_keeper.sleep(config.time_window_ms / 2.0 / 1000.0) # prior to any time passing elapsed_secs = (config.time_window_ms * (config.samples - 1) + config.time_window_ms / 2.0) / 1000.0 kafka_metric = metrics.metrics.get(metrics.metric_name('test.rate', 'grp1')) assert abs((sum_val / elapsed_secs) - kafka_metric.value()) < EPS, \ 'Rate(0...2) = 2.666' assert abs(elapsed_secs - (kafka_metric.measurable.window_size(config, time.time() * 1000) / 1000.0)) \ < EPS, 'Elapsed Time = 75 seconds' def test_reporter(metrics): reporter = DictReporter() foo_reporter = DictReporter(prefix='foo') metrics.add_reporter(reporter) metrics.add_reporter(foo_reporter) sensor = metrics.sensor('kafka.requests') sensor.add(metrics.metric_name('pack.bean1.avg', 'grp1'), Avg()) sensor.add(metrics.metric_name('pack.bean2.total', 'grp2'), Total()) sensor2 = metrics.sensor('kafka.blah') sensor2.add(metrics.metric_name('pack.bean1.some', 'grp1'), Total()) sensor2.add(metrics.metric_name('pack.bean2.some', 'grp1', tags={'a': 42, 'b': 'bar'}), Total()) # kafka-metrics-count > count is the total number of metrics and automatic expected = { 'kafka-metrics-count': {'count': 5.0}, 'grp2': {'pack.bean2.total': 0.0}, 'grp1': {'pack.bean1.avg': 0.0, 'pack.bean1.some': 0.0}, 'grp1.a=42,b=bar': {'pack.bean2.some': 0.0}, } assert expected == reporter.snapshot() for key in list(expected.keys()): metrics = expected.pop(key) expected['foo.%s' % key] = metrics assert expected == foo_reporter.snapshot() class ConstantMeasurable(AbstractMeasurable): _value = 0.0 def measure(self, config, now): return self._value class TimeKeeper(object): """ A clock that you can manually advance by calling sleep """ def __init__(self, auto_tick_ms=0): self._millis = time.time() * 1000 self._auto_tick_ms = auto_tick_ms def time(self): return self.ms() / 1000.0 def ms(self): self.sleep(self._auto_tick_ms) return self._millis def sleep(self, seconds): self._millis += (seconds * 1000) kafka-1.3.2/test/test_package.py0000644001271300127130000000205612661137006016350 0ustar dpowers00000000000000from . import unittest class TestPackage(unittest.TestCase): def test_top_level_namespace(self): import kafka as kafka1 self.assertEqual(kafka1.KafkaConsumer.__name__, "KafkaConsumer") self.assertEqual(kafka1.consumer.__name__, "kafka.consumer") self.assertEqual(kafka1.codec.__name__, "kafka.codec") def test_submodule_namespace(self): import kafka.client as client1 self.assertEqual(client1.__name__, "kafka.client") from kafka import client as client2 self.assertEqual(client2.__name__, "kafka.client") from kafka.client import SimpleClient as SimpleClient1 self.assertEqual(SimpleClient1.__name__, "SimpleClient") from kafka.codec import gzip_encode as gzip_encode1 self.assertEqual(gzip_encode1.__name__, "gzip_encode") from kafka import SimpleClient as SimpleClient2 self.assertEqual(SimpleClient2.__name__, "SimpleClient") from kafka.codec import snappy_encode self.assertEqual(snappy_encode.__name__, "snappy_encode") kafka-1.3.2/test/test_partitioner.py0000644001271300127130000000434513031057471017317 0ustar dpowers00000000000000from __future__ import absolute_import from kafka.partitioner import DefaultPartitioner, Murmur2Partitioner, RoundRobinPartitioner from kafka.partitioner.hashed import murmur2 def test_default_partitioner(): partitioner = DefaultPartitioner() all_partitions = list(range(100)) available = all_partitions # partitioner should return the same partition for the same key p1 = partitioner(b'foo', all_partitions, available) p2 = partitioner(b'foo', all_partitions, available) assert p1 == p2 assert p1 in all_partitions # when key is None, choose one of available partitions assert partitioner(None, all_partitions, [123]) == 123 # with fallback to all_partitions assert partitioner(None, all_partitions, []) in all_partitions def test_roundrobin_partitioner(): partitioner = RoundRobinPartitioner() all_partitions = list(range(100)) available = all_partitions # partitioner should cycle between partitions i = 0 max_partition = all_partitions[len(all_partitions) - 1] while i <= max_partition: assert i == partitioner(None, all_partitions, available) i += 1 i = 0 while i <= int(max_partition / 2): assert i == partitioner(None, all_partitions, available) i += 1 # test dynamic partition re-assignment available = available[:-25] while i <= max(available): assert i == partitioner(None, all_partitions, available) i += 1 all_partitions = list(range(200)) available = all_partitions max_partition = all_partitions[len(all_partitions) - 1] while i <= max_partition: assert i == partitioner(None, all_partitions, available) i += 1 def test_murmur2_java_compatibility(): p = Murmur2Partitioner(range(1000)) # compare with output from Kafka's org.apache.kafka.clients.producer.Partitioner assert p.partition(b'') == 681 assert p.partition(b'a') == 524 assert p.partition(b'ab') == 434 assert p.partition(b'abc') == 107 assert p.partition(b'123456789') == 566 assert p.partition(b'\x00 ') == 742 def test_murmur2_not_ascii(): # Verify no regression of murmur2() bug encoding py2 bytes that don't ascii encode murmur2(b'\xa4') murmur2(b'\x81' * 1000) kafka-1.3.2/test/test_producer.py0000644001271300127130000000470213031057471016577 0ustar dpowers00000000000000import gc import platform import sys import threading import pytest from kafka import KafkaConsumer, KafkaProducer from kafka.producer.buffer import SimpleBufferPool from test.conftest import version from test.testutil import random_string def test_buffer_pool(): pool = SimpleBufferPool(1000, 1000) buf1 = pool.allocate(1000, 1000) message = ''.join(map(str, range(100))) buf1.write(message.encode('utf-8')) pool.deallocate(buf1) buf2 = pool.allocate(1000, 1000) assert buf2.read() == b'' @pytest.mark.skipif(not version(), reason="No KAFKA_VERSION set") @pytest.mark.parametrize("compression", [None, 'gzip', 'snappy', 'lz4']) def test_end_to_end(kafka_broker, compression): if compression == 'lz4': # LZ4 requires 0.8.2 if version() < (0, 8, 2): return # LZ4 python libs don't work on python2.6 elif sys.version_info < (2, 7): return connect_str = 'localhost:' + str(kafka_broker.port) producer = KafkaProducer(bootstrap_servers=connect_str, retries=5, max_block_ms=10000, compression_type=compression, value_serializer=str.encode) consumer = KafkaConsumer(bootstrap_servers=connect_str, group_id=None, consumer_timeout_ms=10000, auto_offset_reset='earliest', value_deserializer=bytes.decode) topic = random_string(5) messages = 100 futures = [] for i in range(messages): futures.append(producer.send(topic, 'msg %d' % i)) ret = [f.get(timeout=30) for f in futures] assert len(ret) == messages producer.close() consumer.subscribe([topic]) msgs = set() for i in range(messages): try: msgs.add(next(consumer).value) except StopIteration: break assert msgs == set(['msg %d' % i for i in range(messages)]) @pytest.mark.skipif(platform.python_implementation() != 'CPython', reason='Test relies on CPython-specific gc policies') def test_kafka_producer_gc_cleanup(): threads = threading.active_count() producer = KafkaProducer(api_version='0.9') # set api_version explicitly to avoid auto-detection assert threading.active_count() == threads + 1 del(producer) gc.collect() assert threading.active_count() == threads kafka-1.3.2/test/test_producer_integration.py0000644001271300127130000004466612713763022021221 0ustar dpowers00000000000000import os import time import uuid from six.moves import range from kafka import ( SimpleProducer, KeyedProducer, create_message, create_gzip_message, create_snappy_message, RoundRobinPartitioner, HashedPartitioner ) from kafka.codec import has_snappy from kafka.errors import UnknownTopicOrPartitionError, LeaderNotAvailableError from kafka.producer.base import Producer from kafka.structs import FetchRequestPayload, ProduceRequestPayload from test.fixtures import ZookeeperFixture, KafkaFixture from test.testutil import KafkaIntegrationTestCase, kafka_versions class TestKafkaProducerIntegration(KafkaIntegrationTestCase): @classmethod def setUpClass(cls): # noqa if not os.environ.get('KAFKA_VERSION'): return cls.zk = ZookeeperFixture.instance() cls.server = KafkaFixture.instance(0, cls.zk.host, cls.zk.port) @classmethod def tearDownClass(cls): # noqa if not os.environ.get('KAFKA_VERSION'): return cls.server.close() cls.zk.close() def test_produce_many_simple(self): start_offset = self.current_offset(self.topic, 0) self.assert_produce_request( [create_message(("Test message %d" % i).encode('utf-8')) for i in range(100)], start_offset, 100, ) self.assert_produce_request( [create_message(("Test message %d" % i).encode('utf-8')) for i in range(100)], start_offset+100, 100, ) def test_produce_10k_simple(self): start_offset = self.current_offset(self.topic, 0) self.assert_produce_request( [create_message(("Test message %d" % i).encode('utf-8')) for i in range(10000)], start_offset, 10000, ) def test_produce_many_gzip(self): start_offset = self.current_offset(self.topic, 0) message1 = create_gzip_message([ (("Gzipped 1 %d" % i).encode('utf-8'), None) for i in range(100)]) message2 = create_gzip_message([ (("Gzipped 2 %d" % i).encode('utf-8'), None) for i in range(100)]) self.assert_produce_request( [ message1, message2 ], start_offset, 200, ) def test_produce_many_snappy(self): self.skipTest("All snappy integration tests fail with nosnappyjava") start_offset = self.current_offset(self.topic, 0) self.assert_produce_request([ create_snappy_message([("Snappy 1 %d" % i, None) for i in range(100)]), create_snappy_message([("Snappy 2 %d" % i, None) for i in range(100)]), ], start_offset, 200, ) def test_produce_mixed(self): start_offset = self.current_offset(self.topic, 0) msg_count = 1+100 messages = [ create_message(b"Just a plain message"), create_gzip_message([ (("Gzipped %d" % i).encode('utf-8'), None) for i in range(100)]), ] # All snappy integration tests fail with nosnappyjava if False and has_snappy(): msg_count += 100 messages.append(create_snappy_message([("Snappy %d" % i, None) for i in range(100)])) self.assert_produce_request(messages, start_offset, msg_count) def test_produce_100k_gzipped(self): start_offset = self.current_offset(self.topic, 0) self.assert_produce_request([ create_gzip_message([ (("Gzipped batch 1, message %d" % i).encode('utf-8'), None) for i in range(50000)]) ], start_offset, 50000, ) self.assert_produce_request([ create_gzip_message([ (("Gzipped batch 1, message %d" % i).encode('utf-8'), None) for i in range(50000)]) ], start_offset+50000, 50000, ) ############################ # SimpleProducer Tests # ############################ def test_simple_producer_new_topic(self): producer = SimpleProducer(self.client) resp = producer.send_messages('new_topic', self.msg('foobar')) self.assert_produce_response(resp, 0) producer.stop() def test_simple_producer(self): partitions = self.client.get_partition_ids_for_topic(self.topic) start_offsets = [self.current_offset(self.topic, p) for p in partitions] producer = SimpleProducer(self.client, random_start=False) # Goes to first partition, randomly. resp = producer.send_messages(self.topic, self.msg("one"), self.msg("two")) self.assert_produce_response(resp, start_offsets[0]) # Goes to the next partition, randomly. resp = producer.send_messages(self.topic, self.msg("three")) self.assert_produce_response(resp, start_offsets[1]) self.assert_fetch_offset(partitions[0], start_offsets[0], [ self.msg("one"), self.msg("two") ]) self.assert_fetch_offset(partitions[1], start_offsets[1], [ self.msg("three") ]) # Goes back to the first partition because there's only two partitions resp = producer.send_messages(self.topic, self.msg("four"), self.msg("five")) self.assert_produce_response(resp, start_offsets[0]+2) self.assert_fetch_offset(partitions[0], start_offsets[0], [ self.msg("one"), self.msg("two"), self.msg("four"), self.msg("five") ]) producer.stop() def test_producer_random_order(self): producer = SimpleProducer(self.client, random_start=True) resp1 = producer.send_messages(self.topic, self.msg("one"), self.msg("two")) resp2 = producer.send_messages(self.topic, self.msg("three")) resp3 = producer.send_messages(self.topic, self.msg("four"), self.msg("five")) self.assertEqual(resp1[0].partition, resp3[0].partition) self.assertNotEqual(resp1[0].partition, resp2[0].partition) def test_producer_ordered_start(self): producer = SimpleProducer(self.client, random_start=False) resp1 = producer.send_messages(self.topic, self.msg("one"), self.msg("two")) resp2 = producer.send_messages(self.topic, self.msg("three")) resp3 = producer.send_messages(self.topic, self.msg("four"), self.msg("five")) self.assertEqual(resp1[0].partition, 0) self.assertEqual(resp2[0].partition, 1) self.assertEqual(resp3[0].partition, 0) def test_async_simple_producer(self): partition = self.client.get_partition_ids_for_topic(self.topic)[0] start_offset = self.current_offset(self.topic, partition) producer = SimpleProducer(self.client, async=True, random_start=False) resp = producer.send_messages(self.topic, self.msg("one")) self.assertEqual(len(resp), 0) # flush messages producer.stop() self.assert_fetch_offset(partition, start_offset, [ self.msg("one") ]) def test_batched_simple_producer__triggers_by_message(self): partitions = self.client.get_partition_ids_for_topic(self.topic) start_offsets = [self.current_offset(self.topic, p) for p in partitions] # Configure batch producer batch_messages = 5 batch_interval = 5 producer = SimpleProducer( self.client, async=True, batch_send_every_n=batch_messages, batch_send_every_t=batch_interval, random_start=False) # Send 4 messages -- should not trigger a batch resp = producer.send_messages( self.topic, self.msg("one"), self.msg("two"), self.msg("three"), self.msg("four"), ) # Batch mode is async. No ack self.assertEqual(len(resp), 0) # It hasn't sent yet self.assert_fetch_offset(partitions[0], start_offsets[0], []) self.assert_fetch_offset(partitions[1], start_offsets[1], []) # send 3 more messages -- should trigger batch on first 5 resp = producer.send_messages( self.topic, self.msg("five"), self.msg("six"), self.msg("seven"), ) # Batch mode is async. No ack self.assertEqual(len(resp), 0) # Wait until producer has pulled all messages from internal queue # this should signal that the first batch was sent, and the producer # is now waiting for enough messages to batch again (or a timeout) timeout = 5 start = time.time() while not producer.queue.empty(): if time.time() - start > timeout: self.fail('timeout waiting for producer queue to empty') time.sleep(0.1) # send messages groups all *msgs in a single call to the same partition # so we should see all messages from the first call in one partition self.assert_fetch_offset(partitions[0], start_offsets[0], [ self.msg("one"), self.msg("two"), self.msg("three"), self.msg("four"), ]) # Because we are batching every 5 messages, we should only see one self.assert_fetch_offset(partitions[1], start_offsets[1], [ self.msg("five"), ]) producer.stop() def test_batched_simple_producer__triggers_by_time(self): partitions = self.client.get_partition_ids_for_topic(self.topic) start_offsets = [self.current_offset(self.topic, p) for p in partitions] batch_interval = 5 producer = SimpleProducer( self.client, async=True, batch_send_every_n=100, batch_send_every_t=batch_interval, random_start=False) # Send 5 messages and do a fetch resp = producer.send_messages( self.topic, self.msg("one"), self.msg("two"), self.msg("three"), self.msg("four"), ) # Batch mode is async. No ack self.assertEqual(len(resp), 0) # It hasn't sent yet self.assert_fetch_offset(partitions[0], start_offsets[0], []) self.assert_fetch_offset(partitions[1], start_offsets[1], []) resp = producer.send_messages(self.topic, self.msg("five"), self.msg("six"), self.msg("seven"), ) # Batch mode is async. No ack self.assertEqual(len(resp), 0) # Wait the timeout out time.sleep(batch_interval) self.assert_fetch_offset(partitions[0], start_offsets[0], [ self.msg("one"), self.msg("two"), self.msg("three"), self.msg("four"), ]) self.assert_fetch_offset(partitions[1], start_offsets[1], [ self.msg("five"), self.msg("six"), self.msg("seven"), ]) producer.stop() ############################ # KeyedProducer Tests # ############################ @kafka_versions('>=0.8.1') def test_keyedproducer_null_payload(self): partitions = self.client.get_partition_ids_for_topic(self.topic) start_offsets = [self.current_offset(self.topic, p) for p in partitions] producer = KeyedProducer(self.client, partitioner=RoundRobinPartitioner) key = "test" resp = producer.send_messages(self.topic, self.key("key1"), self.msg("one")) self.assert_produce_response(resp, start_offsets[0]) resp = producer.send_messages(self.topic, self.key("key2"), None) self.assert_produce_response(resp, start_offsets[1]) resp = producer.send_messages(self.topic, self.key("key3"), None) self.assert_produce_response(resp, start_offsets[0]+1) resp = producer.send_messages(self.topic, self.key("key4"), self.msg("four")) self.assert_produce_response(resp, start_offsets[1]+1) self.assert_fetch_offset(partitions[0], start_offsets[0], [ self.msg("one"), None ]) self.assert_fetch_offset(partitions[1], start_offsets[1], [ None, self.msg("four") ]) producer.stop() def test_round_robin_partitioner(self): partitions = self.client.get_partition_ids_for_topic(self.topic) start_offsets = [self.current_offset(self.topic, p) for p in partitions] producer = KeyedProducer(self.client, partitioner=RoundRobinPartitioner) resp1 = producer.send_messages(self.topic, self.key("key1"), self.msg("one")) resp2 = producer.send_messages(self.topic, self.key("key2"), self.msg("two")) resp3 = producer.send_messages(self.topic, self.key("key3"), self.msg("three")) resp4 = producer.send_messages(self.topic, self.key("key4"), self.msg("four")) self.assert_produce_response(resp1, start_offsets[0]+0) self.assert_produce_response(resp2, start_offsets[1]+0) self.assert_produce_response(resp3, start_offsets[0]+1) self.assert_produce_response(resp4, start_offsets[1]+1) self.assert_fetch_offset(partitions[0], start_offsets[0], [ self.msg("one"), self.msg("three") ]) self.assert_fetch_offset(partitions[1], start_offsets[1], [ self.msg("two"), self.msg("four") ]) producer.stop() def test_hashed_partitioner(self): partitions = self.client.get_partition_ids_for_topic(self.topic) start_offsets = [self.current_offset(self.topic, p) for p in partitions] producer = KeyedProducer(self.client, partitioner=HashedPartitioner) resp1 = producer.send_messages(self.topic, self.key("1"), self.msg("one")) resp2 = producer.send_messages(self.topic, self.key("2"), self.msg("two")) resp3 = producer.send_messages(self.topic, self.key("3"), self.msg("three")) resp4 = producer.send_messages(self.topic, self.key("3"), self.msg("four")) resp5 = producer.send_messages(self.topic, self.key("4"), self.msg("five")) offsets = {partitions[0]: start_offsets[0], partitions[1]: start_offsets[1]} messages = {partitions[0]: [], partitions[1]: []} keys = [self.key(k) for k in ["1", "2", "3", "3", "4"]] resps = [resp1, resp2, resp3, resp4, resp5] msgs = [self.msg(m) for m in ["one", "two", "three", "four", "five"]] for key, resp, msg in zip(keys, resps, msgs): k = hash(key) % 2 partition = partitions[k] offset = offsets[partition] self.assert_produce_response(resp, offset) offsets[partition] += 1 messages[partition].append(msg) self.assert_fetch_offset(partitions[0], start_offsets[0], messages[partitions[0]]) self.assert_fetch_offset(partitions[1], start_offsets[1], messages[partitions[1]]) producer.stop() def test_async_keyed_producer(self): partition = self.client.get_partition_ids_for_topic(self.topic)[0] start_offset = self.current_offset(self.topic, partition) producer = KeyedProducer(self.client, partitioner=RoundRobinPartitioner, async=True, batch_send_every_t=1) resp = producer.send_messages(self.topic, self.key("key1"), self.msg("one")) self.assertEqual(len(resp), 0) # wait for the server to report a new highwatermark while self.current_offset(self.topic, partition) == start_offset: time.sleep(0.1) self.assert_fetch_offset(partition, start_offset, [ self.msg("one") ]) producer.stop() ############################ # Producer ACK Tests # ############################ def test_acks_none(self): partition = self.client.get_partition_ids_for_topic(self.topic)[0] start_offset = self.current_offset(self.topic, partition) producer = Producer( self.client, req_acks=Producer.ACK_NOT_REQUIRED, ) resp = producer.send_messages(self.topic, partition, self.msg("one")) # No response from produce request with no acks required self.assertEqual(len(resp), 0) # But the message should still have been delivered self.assert_fetch_offset(partition, start_offset, [ self.msg("one") ]) producer.stop() def test_acks_local_write(self): partition = self.client.get_partition_ids_for_topic(self.topic)[0] start_offset = self.current_offset(self.topic, partition) producer = Producer( self.client, req_acks=Producer.ACK_AFTER_LOCAL_WRITE, ) resp = producer.send_messages(self.topic, partition, self.msg("one")) self.assert_produce_response(resp, start_offset) self.assert_fetch_offset(partition, start_offset, [ self.msg("one") ]) producer.stop() def test_acks_cluster_commit(self): partition = self.client.get_partition_ids_for_topic(self.topic)[0] start_offset = self.current_offset(self.topic, partition) producer = Producer( self.client, req_acks=Producer.ACK_AFTER_CLUSTER_COMMIT, ) resp = producer.send_messages(self.topic, partition, self.msg("one")) self.assert_produce_response(resp, start_offset) self.assert_fetch_offset(partition, start_offset, [ self.msg("one") ]) producer.stop() def assert_produce_request(self, messages, initial_offset, message_ct, partition=0): produce = ProduceRequestPayload(self.topic, partition, messages=messages) # There should only be one response message from the server. # This will throw an exception if there's more than one. resp = self.client.send_produce_request([ produce ]) self.assert_produce_response(resp, initial_offset) self.assertEqual(self.current_offset(self.topic, partition), initial_offset + message_ct) def assert_produce_response(self, resp, initial_offset): self.assertEqual(len(resp), 1) self.assertEqual(resp[0].error, 0) self.assertEqual(resp[0].offset, initial_offset) def assert_fetch_offset(self, partition, start_offset, expected_messages): # There should only be one response message from the server. # This will throw an exception if there's more than one. resp, = self.client.send_fetch_request([FetchRequestPayload(self.topic, partition, start_offset, 1024)]) self.assertEqual(resp.error, 0) self.assertEqual(resp.partition, partition) messages = [ x.message.value for x in resp.messages ] self.assertEqual(messages, expected_messages) self.assertEqual(resp.highwaterMark, start_offset+len(expected_messages)) kafka-1.3.2/test/test_producer_legacy.py0000644001271300127130000002344112702214455020125 0ustar dpowers00000000000000# -*- coding: utf-8 -*- import collections import logging import threading import time from mock import MagicMock, patch from . import unittest from kafka import SimpleClient, SimpleProducer, KeyedProducer from kafka.errors import ( AsyncProducerQueueFull, FailedPayloadsError, NotLeaderForPartitionError) from kafka.producer.base import Producer, _send_upstream from kafka.protocol import CODEC_NONE from kafka.structs import ( ProduceResponsePayload, RetryOptions, TopicPartition) from six.moves import queue, xrange class TestKafkaProducer(unittest.TestCase): def test_producer_message_types(self): producer = Producer(MagicMock()) topic = b"test-topic" partition = 0 bad_data_types = (u'你怎么样?', 12, ['a', 'list'], ('a', 'tuple'), {'a': 'dict'}, None,) for m in bad_data_types: with self.assertRaises(TypeError): logging.debug("attempting to send message of type %s", type(m)) producer.send_messages(topic, partition, m) good_data_types = (b'a string!',) for m in good_data_types: # This should not raise an exception producer.send_messages(topic, partition, m) def test_keyedproducer_message_types(self): client = MagicMock() client.get_partition_ids_for_topic.return_value = [0, 1] producer = KeyedProducer(client) topic = b"test-topic" key = b"testkey" bad_data_types = (u'你怎么样?', 12, ['a', 'list'], ('a', 'tuple'), {'a': 'dict'},) for m in bad_data_types: with self.assertRaises(TypeError): logging.debug("attempting to send message of type %s", type(m)) producer.send_messages(topic, key, m) good_data_types = (b'a string!', None,) for m in good_data_types: # This should not raise an exception producer.send_messages(topic, key, m) def test_topic_message_types(self): client = MagicMock() def partitions(topic): return [0, 1] client.get_partition_ids_for_topic = partitions producer = SimpleProducer(client, random_start=False) topic = b"test-topic" producer.send_messages(topic, b'hi') assert client.send_produce_request.called @patch('kafka.producer.base._send_upstream') def test_producer_async_queue_overfilled(self, mock): queue_size = 2 producer = Producer(MagicMock(), async=True, async_queue_maxsize=queue_size) topic = b'test-topic' partition = 0 message = b'test-message' with self.assertRaises(AsyncProducerQueueFull): message_list = [message] * (queue_size + 1) producer.send_messages(topic, partition, *message_list) self.assertEqual(producer.queue.qsize(), queue_size) for _ in xrange(producer.queue.qsize()): producer.queue.get() def test_producer_sync_fail_on_error(self): error = FailedPayloadsError('failure') with patch.object(SimpleClient, 'load_metadata_for_topics'): with patch.object(SimpleClient, 'ensure_topic_exists'): with patch.object(SimpleClient, 'get_partition_ids_for_topic', return_value=[0, 1]): with patch.object(SimpleClient, '_send_broker_aware_request', return_value = [error]): client = SimpleClient(MagicMock()) producer = SimpleProducer(client, async=False, sync_fail_on_error=False) # This should not raise (response,) = producer.send_messages('foobar', b'test message') self.assertEqual(response, error) producer = SimpleProducer(client, async=False, sync_fail_on_error=True) with self.assertRaises(FailedPayloadsError): producer.send_messages('foobar', b'test message') def test_cleanup_is_not_called_on_stopped_producer(self): producer = Producer(MagicMock(), async=True) producer.stopped = True with patch.object(producer, 'stop') as mocked_stop: producer._cleanup_func(producer) self.assertEqual(mocked_stop.call_count, 0) def test_cleanup_is_called_on_running_producer(self): producer = Producer(MagicMock(), async=True) producer.stopped = False with patch.object(producer, 'stop') as mocked_stop: producer._cleanup_func(producer) self.assertEqual(mocked_stop.call_count, 1) class TestKafkaProducerSendUpstream(unittest.TestCase): def setUp(self): self.client = MagicMock() self.queue = queue.Queue() def _run_process(self, retries_limit=3, sleep_timeout=1): # run _send_upstream process with the queue stop_event = threading.Event() retry_options = RetryOptions(limit=retries_limit, backoff_ms=50, retry_on_timeouts=False) self.thread = threading.Thread( target=_send_upstream, args=(self.queue, self.client, CODEC_NONE, 0.3, # batch time (seconds) 3, # batch length Producer.ACK_AFTER_LOCAL_WRITE, Producer.DEFAULT_ACK_TIMEOUT, retry_options, stop_event)) self.thread.daemon = True self.thread.start() time.sleep(sleep_timeout) stop_event.set() def test_wo_retries(self): # lets create a queue and add 10 messages for 1 partition for i in range(10): self.queue.put((TopicPartition("test", 0), "msg %i", "key %i")) self._run_process() # the queue should be void at the end of the test self.assertEqual(self.queue.empty(), True) # there should be 4 non-void cals: # 3 batches of 3 msgs each + 1 batch of 1 message self.assertEqual(self.client.send_produce_request.call_count, 4) def test_first_send_failed(self): # lets create a queue and add 10 messages for 10 different partitions # to show how retries should work ideally for i in range(10): self.queue.put((TopicPartition("test", i), "msg %i", "key %i")) # Mock offsets counter for closure offsets = collections.defaultdict(lambda: collections.defaultdict(lambda: 0)) self.client.is_first_time = True def send_side_effect(reqs, *args, **kwargs): if self.client.is_first_time: self.client.is_first_time = False return [FailedPayloadsError(req) for req in reqs] responses = [] for req in reqs: offset = offsets[req.topic][req.partition] offsets[req.topic][req.partition] += len(req.messages) responses.append( ProduceResponsePayload(req.topic, req.partition, 0, offset) ) return responses self.client.send_produce_request.side_effect = send_side_effect self._run_process(2) # the queue should be void at the end of the test self.assertEqual(self.queue.empty(), True) # there should be 5 non-void calls: 1st failed batch of 3 msgs # plus 3 batches of 3 msgs each + 1 batch of 1 message self.assertEqual(self.client.send_produce_request.call_count, 5) def test_with_limited_retries(self): # lets create a queue and add 10 messages for 10 different partitions # to show how retries should work ideally for i in range(10): self.queue.put((TopicPartition("test", i), "msg %i" % i, "key %i" % i)) def send_side_effect(reqs, *args, **kwargs): return [FailedPayloadsError(req) for req in reqs] self.client.send_produce_request.side_effect = send_side_effect self._run_process(3, 3) # the queue should be void at the end of the test self.assertEqual(self.queue.empty(), True) # there should be 16 non-void calls: # 3 initial batches of 3 msgs each + 1 initial batch of 1 msg + # 3 retries of the batches above = (1 + 3 retries) * 4 batches = 16 self.assertEqual(self.client.send_produce_request.call_count, 16) def test_async_producer_not_leader(self): for i in range(10): self.queue.put((TopicPartition("test", i), "msg %i", "key %i")) # Mock offsets counter for closure offsets = collections.defaultdict(lambda: collections.defaultdict(lambda: 0)) self.client.is_first_time = True def send_side_effect(reqs, *args, **kwargs): if self.client.is_first_time: self.client.is_first_time = False return [ProduceResponsePayload(req.topic, req.partition, NotLeaderForPartitionError.errno, -1) for req in reqs] responses = [] for req in reqs: offset = offsets[req.topic][req.partition] offsets[req.topic][req.partition] += len(req.messages) responses.append( ProduceResponsePayload(req.topic, req.partition, 0, offset) ) return responses self.client.send_produce_request.side_effect = send_side_effect self._run_process(2) # the queue should be void at the end of the test self.assertEqual(self.queue.empty(), True) # there should be 5 non-void calls: 1st failed batch of 3 msgs # + 3 batches of 3 msgs each + 1 batch of 1 msg = 1 + 3 + 1 = 5 self.assertEqual(self.client.send_produce_request.call_count, 5) def tearDown(self): for _ in xrange(self.queue.qsize()): self.queue.get() kafka-1.3.2/test/test_protocol.py0000644001271300127130000002203113025302127016602 0ustar dpowers00000000000000#pylint: skip-file import io import struct import pytest import six from kafka.protocol.api import RequestHeader from kafka.protocol.commit import GroupCoordinatorRequest from kafka.protocol.fetch import FetchResponse from kafka.protocol.message import Message, MessageSet, PartialMessage from kafka.protocol.types import Int16, Int32, Int64, String def test_create_message(): payload = b'test' key = b'key' msg = Message(payload, key=key) assert msg.magic == 0 assert msg.attributes == 0 assert msg.key == key assert msg.value == payload def test_encode_message_v0(): message = Message(b'test', key=b'key') encoded = message.encode() expect = b''.join([ struct.pack('>i', -1427009701), # CRC struct.pack('>bb', 0, 0), # Magic, flags struct.pack('>i', 3), # Length of key b'key', # key struct.pack('>i', 4), # Length of value b'test', # value ]) assert encoded == expect def test_encode_message_v1(): message = Message(b'test', key=b'key', magic=1, timestamp=1234) encoded = message.encode() expect = b''.join([ struct.pack('>i', 1331087195), # CRC struct.pack('>bb', 1, 0), # Magic, flags struct.pack('>q', 1234), # Timestamp struct.pack('>i', 3), # Length of key b'key', # key struct.pack('>i', 4), # Length of value b'test', # value ]) assert encoded == expect def test_decode_message(): encoded = b''.join([ struct.pack('>i', -1427009701), # CRC struct.pack('>bb', 0, 0), # Magic, flags struct.pack('>i', 3), # Length of key b'key', # key struct.pack('>i', 4), # Length of value b'test', # value ]) decoded_message = Message.decode(encoded) msg = Message(b'test', key=b'key') msg.encode() # crc is recalculated during encoding assert decoded_message == msg def test_encode_message_set(): messages = [ Message(b'v1', key=b'k1'), Message(b'v2', key=b'k2') ] encoded = MessageSet.encode([(0, msg.encode()) for msg in messages]) expect = b''.join([ struct.pack('>q', 0), # MsgSet Offset struct.pack('>i', 18), # Msg Size struct.pack('>i', 1474775406), # CRC struct.pack('>bb', 0, 0), # Magic, flags struct.pack('>i', 2), # Length of key b'k1', # Key struct.pack('>i', 2), # Length of value b'v1', # Value struct.pack('>q', 0), # MsgSet Offset struct.pack('>i', 18), # Msg Size struct.pack('>i', -16383415), # CRC struct.pack('>bb', 0, 0), # Magic, flags struct.pack('>i', 2), # Length of key b'k2', # Key struct.pack('>i', 2), # Length of value b'v2', # Value ]) expect = struct.pack('>i', len(expect)) + expect assert encoded == expect def test_decode_message_set(): encoded = b''.join([ struct.pack('>q', 0), # MsgSet Offset struct.pack('>i', 18), # Msg Size struct.pack('>i', 1474775406), # CRC struct.pack('>bb', 0, 0), # Magic, flags struct.pack('>i', 2), # Length of key b'k1', # Key struct.pack('>i', 2), # Length of value b'v1', # Value struct.pack('>q', 1), # MsgSet Offset struct.pack('>i', 18), # Msg Size struct.pack('>i', -16383415), # CRC struct.pack('>bb', 0, 0), # Magic, flags struct.pack('>i', 2), # Length of key b'k2', # Key struct.pack('>i', 2), # Length of value b'v2', # Value ]) msgs = MessageSet.decode(encoded, bytes_to_read=len(encoded)) assert len(msgs) == 2 msg1, msg2 = msgs returned_offset1, message1_size, decoded_message1 = msg1 returned_offset2, message2_size, decoded_message2 = msg2 assert returned_offset1 == 0 message1 = Message(b'v1', key=b'k1') message1.encode() assert decoded_message1 == message1 assert returned_offset2 == 1 message2 = Message(b'v2', key=b'k2') message2.encode() assert decoded_message2 == message2 def test_encode_message_header(): expect = b''.join([ struct.pack('>h', 10), # API Key struct.pack('>h', 0), # API Version struct.pack('>i', 4), # Correlation Id struct.pack('>h', len('client3')), # Length of clientId b'client3', # ClientId ]) req = GroupCoordinatorRequest[0]('foo') header = RequestHeader(req, correlation_id=4, client_id='client3') assert header.encode() == expect def test_decode_message_set_partial(): encoded = b''.join([ struct.pack('>q', 0), # Msg Offset struct.pack('>i', 18), # Msg Size struct.pack('>i', 1474775406), # CRC struct.pack('>bb', 0, 0), # Magic, flags struct.pack('>i', 2), # Length of key b'k1', # Key struct.pack('>i', 2), # Length of value b'v1', # Value struct.pack('>q', 1), # Msg Offset struct.pack('>i', 24), # Msg Size (larger than remaining MsgSet size) struct.pack('>i', -16383415), # CRC struct.pack('>bb', 0, 0), # Magic, flags struct.pack('>i', 2), # Length of key b'k2', # Key struct.pack('>i', 8), # Length of value b'ar', # Value (truncated) ]) msgs = MessageSet.decode(encoded, bytes_to_read=len(encoded)) assert len(msgs) == 2 msg1, msg2 = msgs returned_offset1, message1_size, decoded_message1 = msg1 returned_offset2, message2_size, decoded_message2 = msg2 assert returned_offset1 == 0 message1 = Message(b'v1', key=b'k1') message1.encode() assert decoded_message1 == message1 assert returned_offset2 is None assert message2_size is None assert decoded_message2 == PartialMessage() def test_decode_fetch_response_partial(): encoded = b''.join([ Int32.encode(1), # Num Topics (Array) String('utf-8').encode('foobar'), Int32.encode(2), # Num Partitions (Array) Int32.encode(0), # Partition id Int16.encode(0), # Error Code Int64.encode(1234), # Highwater offset Int32.encode(52), # MessageSet size Int64.encode(0), # Msg Offset Int32.encode(18), # Msg Size struct.pack('>i', 1474775406), # CRC struct.pack('>bb', 0, 0), # Magic, flags struct.pack('>i', 2), # Length of key b'k1', # Key struct.pack('>i', 2), # Length of value b'v1', # Value Int64.encode(1), # Msg Offset struct.pack('>i', 24), # Msg Size (larger than remaining MsgSet size) struct.pack('>i', -16383415), # CRC struct.pack('>bb', 0, 0), # Magic, flags struct.pack('>i', 2), # Length of key b'k2', # Key struct.pack('>i', 8), # Length of value b'ar', # Value (truncated) Int32.encode(1), Int16.encode(0), Int64.encode(2345), Int32.encode(52), # MessageSet size Int64.encode(0), # Msg Offset Int32.encode(18), # Msg Size struct.pack('>i', 1474775406), # CRC struct.pack('>bb', 0, 0), # Magic, flags struct.pack('>i', 2), # Length of key b'k1', # Key struct.pack('>i', 2), # Length of value b'v1', # Value Int64.encode(1), # Msg Offset struct.pack('>i', 24), # Msg Size (larger than remaining MsgSet size) struct.pack('>i', -16383415), # CRC struct.pack('>bb', 0, 0), # Magic, flags struct.pack('>i', 2), # Length of key b'k2', # Key struct.pack('>i', 8), # Length of value b'ar', # Value (truncated) ]) resp = FetchResponse[0].decode(io.BytesIO(encoded)) assert len(resp.topics) == 1 topic, partitions = resp.topics[0] assert topic == 'foobar' assert len(partitions) == 2 m1 = partitions[0][3] assert len(m1) == 2 assert m1[1] == (None, None, PartialMessage()) kafka-1.3.2/test/test_protocol_legacy.py0000644001271300127130000011400112730712510020130 0ustar dpowers00000000000000#pylint: skip-file from contextlib import contextmanager import struct import six from mock import patch, sentinel from . import unittest from kafka.codec import has_snappy, gzip_decode, snappy_decode from kafka.errors import ( ChecksumError, KafkaUnavailableError, UnsupportedCodecError, ConsumerFetchSizeTooSmall, ProtocolError) from kafka.protocol import ( ATTRIBUTE_CODEC_MASK, CODEC_NONE, CODEC_GZIP, CODEC_SNAPPY, KafkaProtocol, create_message, create_gzip_message, create_snappy_message, create_message_set) from kafka.structs import ( OffsetRequestPayload, OffsetResponsePayload, OffsetCommitRequestPayload, OffsetCommitResponsePayload, OffsetFetchRequestPayload, OffsetFetchResponsePayload, ProduceRequestPayload, ProduceResponsePayload, FetchRequestPayload, FetchResponsePayload, Message, OffsetAndMessage, BrokerMetadata, ConsumerMetadataResponse) class TestProtocol(unittest.TestCase): def test_create_message(self): payload = "test" key = "key" msg = create_message(payload, key) self.assertEqual(msg.magic, 0) self.assertEqual(msg.attributes, 0) self.assertEqual(msg.key, key) self.assertEqual(msg.value, payload) def test_create_gzip(self): payloads = [(b"v1", None), (b"v2", None)] msg = create_gzip_message(payloads) self.assertEqual(msg.magic, 0) self.assertEqual(msg.attributes, ATTRIBUTE_CODEC_MASK & CODEC_GZIP) self.assertEqual(msg.key, None) # Need to decode to check since gzipped payload is non-deterministic decoded = gzip_decode(msg.value) expect = b"".join([ struct.pack(">q", 0), # MsgSet offset struct.pack(">i", 16), # MsgSet size struct.pack(">i", 1285512130), # CRC struct.pack(">bb", 0, 0), # Magic, flags struct.pack(">i", -1), # -1 indicates a null key struct.pack(">i", 2), # Msg length (bytes) b"v1", # Message contents struct.pack(">q", 0), # MsgSet offset struct.pack(">i", 16), # MsgSet size struct.pack(">i", -711587208), # CRC struct.pack(">bb", 0, 0), # Magic, flags struct.pack(">i", -1), # -1 indicates a null key struct.pack(">i", 2), # Msg length (bytes) b"v2", # Message contents ]) self.assertEqual(decoded, expect) def test_create_gzip_keyed(self): payloads = [(b"v1", b"k1"), (b"v2", b"k2")] msg = create_gzip_message(payloads) self.assertEqual(msg.magic, 0) self.assertEqual(msg.attributes, ATTRIBUTE_CODEC_MASK & CODEC_GZIP) self.assertEqual(msg.key, None) # Need to decode to check since gzipped payload is non-deterministic decoded = gzip_decode(msg.value) expect = b"".join([ struct.pack(">q", 0), # MsgSet Offset struct.pack(">i", 18), # Msg Size struct.pack(">i", 1474775406), # CRC struct.pack(">bb", 0, 0), # Magic, flags struct.pack(">i", 2), # Length of key b"k1", # Key struct.pack(">i", 2), # Length of value b"v1", # Value struct.pack(">q", 0), # MsgSet Offset struct.pack(">i", 18), # Msg Size struct.pack(">i", -16383415), # CRC struct.pack(">bb", 0, 0), # Magic, flags struct.pack(">i", 2), # Length of key b"k2", # Key struct.pack(">i", 2), # Length of value b"v2", # Value ]) self.assertEqual(decoded, expect) @unittest.skipUnless(has_snappy(), "Snappy not available") def test_create_snappy(self): payloads = [(b"v1", None), (b"v2", None)] msg = create_snappy_message(payloads) self.assertEqual(msg.magic, 0) self.assertEqual(msg.attributes, ATTRIBUTE_CODEC_MASK & CODEC_SNAPPY) self.assertEqual(msg.key, None) decoded = snappy_decode(msg.value) expect = b"".join([ struct.pack(">q", 0), # MsgSet offset struct.pack(">i", 16), # MsgSet size struct.pack(">i", 1285512130), # CRC struct.pack(">bb", 0, 0), # Magic, flags struct.pack(">i", -1), # -1 indicates a null key struct.pack(">i", 2), # Msg length (bytes) b"v1", # Message contents struct.pack(">q", 0), # MsgSet offset struct.pack(">i", 16), # MsgSet size struct.pack(">i", -711587208), # CRC struct.pack(">bb", 0, 0), # Magic, flags struct.pack(">i", -1), # -1 indicates a null key struct.pack(">i", 2), # Msg length (bytes) b"v2", # Message contents ]) self.assertEqual(decoded, expect) @unittest.skipUnless(has_snappy(), "Snappy not available") def test_create_snappy_keyed(self): payloads = [(b"v1", b"k1"), (b"v2", b"k2")] msg = create_snappy_message(payloads) self.assertEqual(msg.magic, 0) self.assertEqual(msg.attributes, ATTRIBUTE_CODEC_MASK & CODEC_SNAPPY) self.assertEqual(msg.key, None) decoded = snappy_decode(msg.value) expect = b"".join([ struct.pack(">q", 0), # MsgSet Offset struct.pack(">i", 18), # Msg Size struct.pack(">i", 1474775406), # CRC struct.pack(">bb", 0, 0), # Magic, flags struct.pack(">i", 2), # Length of key b"k1", # Key struct.pack(">i", 2), # Length of value b"v1", # Value struct.pack(">q", 0), # MsgSet Offset struct.pack(">i", 18), # Msg Size struct.pack(">i", -16383415), # CRC struct.pack(">bb", 0, 0), # Magic, flags struct.pack(">i", 2), # Length of key b"k2", # Key struct.pack(">i", 2), # Length of value b"v2", # Value ]) self.assertEqual(decoded, expect) def test_encode_message_header(self): expect = b"".join([ struct.pack(">h", 10), # API Key struct.pack(">h", 0), # API Version struct.pack(">i", 4), # Correlation Id struct.pack(">h", len("client3")), # Length of clientId b"client3", # ClientId ]) encoded = KafkaProtocol._encode_message_header(b"client3", 4, 10) self.assertEqual(encoded, expect) def test_encode_message(self): message = create_message(b"test", b"key") encoded = KafkaProtocol._encode_message(message) expect = b"".join([ struct.pack(">i", -1427009701), # CRC struct.pack(">bb", 0, 0), # Magic, flags struct.pack(">i", 3), # Length of key b"key", # key struct.pack(">i", 4), # Length of value b"test", # value ]) self.assertEqual(encoded, expect) @unittest.skip('needs updating for new protocol classes') def test_decode_message(self): encoded = b"".join([ struct.pack(">i", -1427009701), # CRC struct.pack(">bb", 0, 0), # Magic, flags struct.pack(">i", 3), # Length of key b"key", # key struct.pack(">i", 4), # Length of value b"test", # value ]) offset = 10 (returned_offset, decoded_message) = list(KafkaProtocol._decode_message(encoded, offset))[0] self.assertEqual(returned_offset, offset) self.assertEqual(decoded_message, create_message(b"test", b"key")) def test_encode_message_failure(self): with self.assertRaises(ProtocolError): KafkaProtocol._encode_message(Message(1, 0, "key", "test")) @unittest.skip('needs updating for new protocol classes') def test_encode_message_set(self): message_set = [ create_message(b"v1", b"k1"), create_message(b"v2", b"k2") ] encoded = KafkaProtocol._encode_message_set(message_set) expect = b"".join([ struct.pack(">q", 0), # MsgSet Offset struct.pack(">i", 18), # Msg Size struct.pack(">i", 1474775406), # CRC struct.pack(">bb", 0, 0), # Magic, flags struct.pack(">i", 2), # Length of key b"k1", # Key struct.pack(">i", 2), # Length of value b"v1", # Value struct.pack(">q", 0), # MsgSet Offset struct.pack(">i", 18), # Msg Size struct.pack(">i", -16383415), # CRC struct.pack(">bb", 0, 0), # Magic, flags struct.pack(">i", 2), # Length of key b"k2", # Key struct.pack(">i", 2), # Length of value b"v2", # Value ]) self.assertEqual(encoded, expect) @unittest.skip('needs updating for new protocol classes') def test_decode_message_set(self): encoded = b"".join([ struct.pack(">q", 0), # MsgSet Offset struct.pack(">i", 18), # Msg Size struct.pack(">i", 1474775406), # CRC struct.pack(">bb", 0, 0), # Magic, flags struct.pack(">i", 2), # Length of key b"k1", # Key struct.pack(">i", 2), # Length of value b"v1", # Value struct.pack(">q", 1), # MsgSet Offset struct.pack(">i", 18), # Msg Size struct.pack(">i", -16383415), # CRC struct.pack(">bb", 0, 0), # Magic, flags struct.pack(">i", 2), # Length of key b"k2", # Key struct.pack(">i", 2), # Length of value b"v2", # Value ]) msgs = list(KafkaProtocol._decode_message_set_iter(encoded)) self.assertEqual(len(msgs), 2) msg1, msg2 = msgs returned_offset1, decoded_message1 = msg1 returned_offset2, decoded_message2 = msg2 self.assertEqual(returned_offset1, 0) self.assertEqual(decoded_message1, create_message(b"v1", b"k1")) self.assertEqual(returned_offset2, 1) self.assertEqual(decoded_message2, create_message(b"v2", b"k2")) @unittest.skip('needs updating for new protocol classes') def test_decode_message_gzip(self): gzip_encoded = (b'\xc0\x11\xb2\xf0\x00\x01\xff\xff\xff\xff\x00\x00\x000' b'\x1f\x8b\x08\x00\xa1\xc1\xc5R\x02\xffc`\x80\x03\x01' b'\x9f\xf9\xd1\x87\x18\x18\xfe\x03\x01\x90\xc7Tf\xc8' b'\x80$wu\x1aW\x05\x92\x9c\x11\x00z\xc0h\x888\x00\x00' b'\x00') offset = 11 messages = list(KafkaProtocol._decode_message(gzip_encoded, offset)) self.assertEqual(len(messages), 2) msg1, msg2 = messages returned_offset1, decoded_message1 = msg1 self.assertEqual(returned_offset1, 0) self.assertEqual(decoded_message1, create_message(b"v1")) returned_offset2, decoded_message2 = msg2 self.assertEqual(returned_offset2, 0) self.assertEqual(decoded_message2, create_message(b"v2")) @unittest.skip('needs updating for new protocol classes') @unittest.skipUnless(has_snappy(), "Snappy not available") def test_decode_message_snappy(self): snappy_encoded = (b'\xec\x80\xa1\x95\x00\x02\xff\xff\xff\xff\x00\x00' b'\x00,8\x00\x00\x19\x01@\x10L\x9f[\xc2\x00\x00\xff' b'\xff\xff\xff\x00\x00\x00\x02v1\x19\x1bD\x00\x10\xd5' b'\x96\nx\x00\x00\xff\xff\xff\xff\x00\x00\x00\x02v2') offset = 11 messages = list(KafkaProtocol._decode_message(snappy_encoded, offset)) self.assertEqual(len(messages), 2) msg1, msg2 = messages returned_offset1, decoded_message1 = msg1 self.assertEqual(returned_offset1, 0) self.assertEqual(decoded_message1, create_message(b"v1")) returned_offset2, decoded_message2 = msg2 self.assertEqual(returned_offset2, 0) self.assertEqual(decoded_message2, create_message(b"v2")) @unittest.skip('needs updating for new protocol classes') def test_decode_message_checksum_error(self): invalid_encoded_message = b"This is not a valid encoded message" iter = KafkaProtocol._decode_message(invalid_encoded_message, 0) self.assertRaises(ChecksumError, list, iter) # NOTE: The error handling in _decode_message_set_iter() is questionable. # If it's modified, the next two tests might need to be fixed. @unittest.skip('needs updating for new protocol classes') def test_decode_message_set_fetch_size_too_small(self): with self.assertRaises(ConsumerFetchSizeTooSmall): list(KafkaProtocol._decode_message_set_iter('a')) @unittest.skip('needs updating for new protocol classes') def test_decode_message_set_stop_iteration(self): encoded = b"".join([ struct.pack(">q", 0), # MsgSet Offset struct.pack(">i", 18), # Msg Size struct.pack(">i", 1474775406), # CRC struct.pack(">bb", 0, 0), # Magic, flags struct.pack(">i", 2), # Length of key b"k1", # Key struct.pack(">i", 2), # Length of value b"v1", # Value struct.pack(">q", 1), # MsgSet Offset struct.pack(">i", 18), # Msg Size struct.pack(">i", -16383415), # CRC struct.pack(">bb", 0, 0), # Magic, flags struct.pack(">i", 2), # Length of key b"k2", # Key struct.pack(">i", 2), # Length of value b"v2", # Value b"@1$%(Y!", # Random padding ]) msgs = MessageSet.decode(io.BytesIO(encoded)) self.assertEqual(len(msgs), 2) msg1, msg2 = msgs returned_offset1, msg_size1, decoded_message1 = msg1 returned_offset2, msg_size2, decoded_message2 = msg2 self.assertEqual(returned_offset1, 0) self.assertEqual(decoded_message1.value, b"v1") self.assertEqual(decoded_message1.key, b"k1") self.assertEqual(returned_offset2, 1) self.assertEqual(decoded_message2.value, b"v2") self.assertEqual(decoded_message2.key, b"k2") @unittest.skip('needs updating for new protocol classes') def test_encode_produce_request(self): requests = [ ProduceRequestPayload("topic1", 0, [ kafka.protocol.message.Message(b"a"), kafka.protocol.message.Message(b"b") ]), ProduceRequestPayload("topic2", 1, [ kafka.protocol.message.Message(b"c") ]) ] msg_a_binary = KafkaProtocol._encode_message(create_message(b"a")) msg_b_binary = KafkaProtocol._encode_message(create_message(b"b")) msg_c_binary = KafkaProtocol._encode_message(create_message(b"c")) header = b"".join([ struct.pack('>i', 0x94), # The length of the message overall struct.pack('>h', 0), # Msg Header, Message type = Produce struct.pack('>h', 0), # Msg Header, API version struct.pack('>i', 2), # Msg Header, Correlation ID struct.pack('>h7s', 7, b"client1"), # Msg Header, The client ID struct.pack('>h', 2), # Num acks required struct.pack('>i', 100), # Request Timeout struct.pack('>i', 2), # The number of requests ]) total_len = len(msg_a_binary) + len(msg_b_binary) topic1 = b"".join([ struct.pack('>h6s', 6, b'topic1'), # The topic1 struct.pack('>i', 1), # One message set struct.pack('>i', 0), # Partition 0 struct.pack('>i', total_len + 24), # Size of the incoming message set struct.pack('>q', 0), # No offset specified struct.pack('>i', len(msg_a_binary)), # Length of message msg_a_binary, # Actual message struct.pack('>q', 0), # No offset specified struct.pack('>i', len(msg_b_binary)), # Length of message msg_b_binary, # Actual message ]) topic2 = b"".join([ struct.pack('>h6s', 6, b'topic2'), # The topic1 struct.pack('>i', 1), # One message set struct.pack('>i', 1), # Partition 1 struct.pack('>i', len(msg_c_binary) + 12), # Size of the incoming message set struct.pack('>q', 0), # No offset specified struct.pack('>i', len(msg_c_binary)), # Length of message msg_c_binary, # Actual message ]) expected1 = b"".join([ header, topic1, topic2 ]) expected2 = b"".join([ header, topic2, topic1 ]) encoded = KafkaProtocol.encode_produce_request(b"client1", 2, requests, 2, 100) self.assertIn(encoded, [ expected1, expected2 ]) @unittest.skip('needs updating for new protocol classes') def test_decode_produce_response(self): t1 = b"topic1" t2 = b"topic2" _long = int if six.PY2: _long = long encoded = struct.pack('>iih%dsiihqihqh%dsiihq' % (len(t1), len(t2)), 2, 2, len(t1), t1, 2, 0, 0, _long(10), 1, 1, _long(20), len(t2), t2, 1, 0, 0, _long(30)) responses = list(KafkaProtocol.decode_produce_response(encoded)) self.assertEqual(responses, [ProduceResponse(t1, 0, 0, _long(10)), ProduceResponse(t1, 1, 1, _long(20)), ProduceResponse(t2, 0, 0, _long(30))]) @unittest.skip('needs updating for new protocol classes') def test_encode_fetch_request(self): requests = [ FetchRequest(b"topic1", 0, 10, 1024), FetchRequest(b"topic2", 1, 20, 100), ] header = b"".join([ struct.pack('>i', 89), # The length of the message overall struct.pack('>h', 1), # Msg Header, Message type = Fetch struct.pack('>h', 0), # Msg Header, API version struct.pack('>i', 3), # Msg Header, Correlation ID struct.pack('>h7s', 7, b"client1"),# Msg Header, The client ID struct.pack('>i', -1), # Replica Id struct.pack('>i', 2), # Max wait time struct.pack('>i', 100), # Min bytes struct.pack('>i', 2), # Num requests ]) topic1 = b"".join([ struct.pack('>h6s', 6, b'topic1'),# Topic struct.pack('>i', 1), # Num Payloads struct.pack('>i', 0), # Partition 0 struct.pack('>q', 10), # Offset struct.pack('>i', 1024), # Max Bytes ]) topic2 = b"".join([ struct.pack('>h6s', 6, b'topic2'),# Topic struct.pack('>i', 1), # Num Payloads struct.pack('>i', 1), # Partition 0 struct.pack('>q', 20), # Offset struct.pack('>i', 100), # Max Bytes ]) expected1 = b"".join([ header, topic1, topic2 ]) expected2 = b"".join([ header, topic2, topic1 ]) encoded = KafkaProtocol.encode_fetch_request(b"client1", 3, requests, 2, 100) self.assertIn(encoded, [ expected1, expected2 ]) @unittest.skip('needs updating for new protocol classes') def test_decode_fetch_response(self): t1 = b"topic1" t2 = b"topic2" msgs = [create_message(msg) for msg in [b"message1", b"hi", b"boo", b"foo", b"so fun!"]] ms1 = KafkaProtocol._encode_message_set([msgs[0], msgs[1]]) ms2 = KafkaProtocol._encode_message_set([msgs[2]]) ms3 = KafkaProtocol._encode_message_set([msgs[3], msgs[4]]) encoded = struct.pack('>iih%dsiihqi%dsihqi%dsh%dsiihqi%ds' % (len(t1), len(ms1), len(ms2), len(t2), len(ms3)), 4, 2, len(t1), t1, 2, 0, 0, 10, len(ms1), ms1, 1, 1, 20, len(ms2), ms2, len(t2), t2, 1, 0, 0, 30, len(ms3), ms3) responses = list(KafkaProtocol.decode_fetch_response(encoded)) def expand_messages(response): return FetchResponsePayload(response.topic, response.partition, response.error, response.highwaterMark, list(response.messages)) expanded_responses = list(map(expand_messages, responses)) expect = [FetchResponsePayload(t1, 0, 0, 10, [OffsetAndMessage(0, msgs[0]), OffsetAndMessage(0, msgs[1])]), FetchResponsePayload(t1, 1, 1, 20, [OffsetAndMessage(0, msgs[2])]), FetchResponsePayload(t2, 0, 0, 30, [OffsetAndMessage(0, msgs[3]), OffsetAndMessage(0, msgs[4])])] self.assertEqual(expanded_responses, expect) @unittest.skip('needs updating for new protocol classes') def test_encode_metadata_request_no_topics(self): expected = b"".join([ struct.pack(">i", 17), # Total length of the request struct.pack('>h', 3), # API key metadata fetch struct.pack('>h', 0), # API version struct.pack('>i', 4), # Correlation ID struct.pack('>h3s', 3, b"cid"),# The client ID struct.pack('>i', 0), # No topics, give all the data! ]) encoded = KafkaProtocol.encode_metadata_request(b"cid", 4) self.assertEqual(encoded, expected) @unittest.skip('needs updating for new protocol classes') def test_encode_metadata_request_with_topics(self): expected = b"".join([ struct.pack(">i", 25), # Total length of the request struct.pack('>h', 3), # API key metadata fetch struct.pack('>h', 0), # API version struct.pack('>i', 4), # Correlation ID struct.pack('>h3s', 3, b"cid"),# The client ID struct.pack('>i', 2), # Number of topics in the request struct.pack('>h2s', 2, b"t1"), # Topic "t1" struct.pack('>h2s', 2, b"t2"), # Topic "t2" ]) encoded = KafkaProtocol.encode_metadata_request(b"cid", 4, [b"t1", b"t2"]) self.assertEqual(encoded, expected) def _create_encoded_metadata_response(self, brokers, topics): encoded = [] encoded.append(struct.pack('>ii', 3, len(brokers))) for broker in brokers: encoded.append(struct.pack('>ih%dsi' % len(broker.host), broker.nodeId, len(broker.host), broker.host, broker.port)) encoded.append(struct.pack('>i', len(topics))) for topic in topics: encoded.append(struct.pack('>hh%dsi' % len(topic.topic), topic.error, len(topic.topic), topic.topic, len(topic.partitions))) for metadata in topic.partitions: encoded.append(struct.pack('>hiii', metadata.error, metadata.partition, metadata.leader, len(metadata.replicas))) if len(metadata.replicas) > 0: encoded.append(struct.pack('>%di' % len(metadata.replicas), *metadata.replicas)) encoded.append(struct.pack('>i', len(metadata.isr))) if len(metadata.isr) > 0: encoded.append(struct.pack('>%di' % len(metadata.isr), *metadata.isr)) return b''.join(encoded) @unittest.skip('needs updating for new protocol classes') def test_decode_metadata_response(self): node_brokers = [ BrokerMetadata(0, b"brokers1.kafka.rdio.com", 1000), BrokerMetadata(1, b"brokers1.kafka.rdio.com", 1001), BrokerMetadata(3, b"brokers2.kafka.rdio.com", 1000) ] ''' topic_partitions = [ TopicMetadata(b"topic1", 0, [ PartitionMetadata(b"topic1", 0, 1, (0, 2), (2,), 0), PartitionMetadata(b"topic1", 1, 3, (0, 1), (0, 1), 1) ]), TopicMetadata(b"topic2", 1, [ PartitionMetadata(b"topic2", 0, 0, (), (), 0), ]), ] encoded = self._create_encoded_metadata_response(node_brokers, topic_partitions) decoded = KafkaProtocol.decode_metadata_response(encoded) self.assertEqual(decoded, (node_brokers, topic_partitions)) ''' def test_encode_consumer_metadata_request(self): expected = b"".join([ struct.pack(">i", 17), # Total length of the request struct.pack('>h', 10), # API key consumer metadata struct.pack('>h', 0), # API version struct.pack('>i', 4), # Correlation ID struct.pack('>h3s', 3, b"cid"),# The client ID struct.pack('>h2s', 2, b"g1"), # Group "g1" ]) encoded = KafkaProtocol.encode_consumer_metadata_request(b"cid", 4, b"g1") self.assertEqual(encoded, expected) def test_decode_consumer_metadata_response(self): encoded = b"".join([ struct.pack(">i", 42), # Correlation ID struct.pack(">h", 0), # No Error struct.pack(">i", 1), # Broker ID struct.pack(">h23s", 23, b"brokers1.kafka.rdio.com"), # Broker Host struct.pack(">i", 1000), # Broker Port ]) results = KafkaProtocol.decode_consumer_metadata_response(encoded) self.assertEqual(results, ConsumerMetadataResponse(error = 0, nodeId = 1, host = b'brokers1.kafka.rdio.com', port = 1000) ) @unittest.skip('needs updating for new protocol classes') def test_encode_offset_request(self): expected = b"".join([ struct.pack(">i", 21), # Total length of the request struct.pack('>h', 2), # Message type = offset fetch struct.pack('>h', 0), # API version struct.pack('>i', 4), # Correlation ID struct.pack('>h3s', 3, b"cid"), # The client ID struct.pack('>i', -1), # Replica Id struct.pack('>i', 0), # No topic/partitions ]) encoded = KafkaProtocol.encode_offset_request(b"cid", 4) self.assertEqual(encoded, expected) @unittest.skip('needs updating for new protocol classes') def test_encode_offset_request__no_payload(self): expected = b"".join([ struct.pack(">i", 65), # Total length of the request struct.pack('>h', 2), # Message type = offset fetch struct.pack('>h', 0), # API version struct.pack('>i', 4), # Correlation ID struct.pack('>h3s', 3, b"cid"), # The client ID struct.pack('>i', -1), # Replica Id struct.pack('>i', 1), # Num topics struct.pack(">h6s", 6, b"topic1"),# Topic for the request struct.pack(">i", 2), # Two partitions struct.pack(">i", 3), # Partition 3 struct.pack(">q", -1), # No time offset struct.pack(">i", 1), # One offset requested struct.pack(">i", 4), # Partition 3 struct.pack(">q", -1), # No time offset struct.pack(">i", 1), # One offset requested ]) encoded = KafkaProtocol.encode_offset_request(b"cid", 4, [ OffsetRequest(b'topic1', 3, -1, 1), OffsetRequest(b'topic1', 4, -1, 1), ]) self.assertEqual(encoded, expected) @unittest.skip('needs updating for new protocol classes') def test_decode_offset_response(self): encoded = b"".join([ struct.pack(">i", 42), # Correlation ID struct.pack(">i", 1), # One topics struct.pack(">h6s", 6, b"topic1"),# First topic struct.pack(">i", 2), # Two partitions struct.pack(">i", 2), # Partition 2 struct.pack(">h", 0), # No error struct.pack(">i", 1), # One offset struct.pack(">q", 4), # Offset 4 struct.pack(">i", 4), # Partition 4 struct.pack(">h", 0), # No error struct.pack(">i", 1), # One offset struct.pack(">q", 8), # Offset 8 ]) results = KafkaProtocol.decode_offset_response(encoded) self.assertEqual(set(results), set([ OffsetResponse(topic = b'topic1', partition = 2, error = 0, offsets=(4,)), OffsetResponse(topic = b'topic1', partition = 4, error = 0, offsets=(8,)), ])) @unittest.skip('needs updating for new protocol classes') def test_encode_offset_commit_request(self): header = b"".join([ struct.pack('>i', 99), # Total message length struct.pack('>h', 8), # Message type = offset commit struct.pack('>h', 0), # API version struct.pack('>i', 42), # Correlation ID struct.pack('>h9s', 9, b"client_id"),# The client ID struct.pack('>h8s', 8, b"group_id"), # The group to commit for struct.pack('>i', 2), # Num topics ]) topic1 = b"".join([ struct.pack(">h6s", 6, b"topic1"), # Topic for the request struct.pack(">i", 2), # Two partitions struct.pack(">i", 0), # Partition 0 struct.pack(">q", 123), # Offset 123 struct.pack(">h", -1), # Null metadata struct.pack(">i", 1), # Partition 1 struct.pack(">q", 234), # Offset 234 struct.pack(">h", -1), # Null metadata ]) topic2 = b"".join([ struct.pack(">h6s", 6, b"topic2"), # Topic for the request struct.pack(">i", 1), # One partition struct.pack(">i", 2), # Partition 2 struct.pack(">q", 345), # Offset 345 struct.pack(">h", -1), # Null metadata ]) expected1 = b"".join([ header, topic1, topic2 ]) expected2 = b"".join([ header, topic2, topic1 ]) encoded = KafkaProtocol.encode_offset_commit_request(b"client_id", 42, b"group_id", [ OffsetCommitRequest(b"topic1", 0, 123, None), OffsetCommitRequest(b"topic1", 1, 234, None), OffsetCommitRequest(b"topic2", 2, 345, None), ]) self.assertIn(encoded, [ expected1, expected2 ]) @unittest.skip('needs updating for new protocol classes') def test_decode_offset_commit_response(self): encoded = b"".join([ struct.pack(">i", 42), # Correlation ID struct.pack(">i", 1), # One topic struct.pack(">h6s", 6, b"topic1"),# First topic struct.pack(">i", 2), # Two partitions struct.pack(">i", 2), # Partition 2 struct.pack(">h", 0), # No error struct.pack(">i", 4), # Partition 4 struct.pack(">h", 0), # No error ]) results = KafkaProtocol.decode_offset_commit_response(encoded) self.assertEqual(set(results), set([ OffsetCommitResponse(topic = b'topic1', partition = 2, error = 0), OffsetCommitResponse(topic = b'topic1', partition = 4, error = 0), ])) @unittest.skip('needs updating for new protocol classes') def test_encode_offset_fetch_request(self): header = b"".join([ struct.pack('>i', 69), # Total message length struct.pack('>h', 9), # Message type = offset fetch struct.pack('>h', 0), # API version struct.pack('>i', 42), # Correlation ID struct.pack('>h9s', 9, b"client_id"),# The client ID struct.pack('>h8s', 8, b"group_id"), # The group to commit for struct.pack('>i', 2), # Num topics ]) topic1 = b"".join([ struct.pack(">h6s", 6, b"topic1"), # Topic for the request struct.pack(">i", 2), # Two partitions struct.pack(">i", 0), # Partition 0 struct.pack(">i", 1), # Partition 1 ]) topic2 = b"".join([ struct.pack(">h6s", 6, b"topic2"), # Topic for the request struct.pack(">i", 1), # One partitions struct.pack(">i", 2), # Partition 2 ]) expected1 = b"".join([ header, topic1, topic2 ]) expected2 = b"".join([ header, topic2, topic1 ]) encoded = KafkaProtocol.encode_offset_fetch_request(b"client_id", 42, b"group_id", [ OffsetFetchRequest(b"topic1", 0), OffsetFetchRequest(b"topic1", 1), OffsetFetchRequest(b"topic2", 2), ]) self.assertIn(encoded, [ expected1, expected2 ]) @unittest.skip('needs updating for new protocol classes') def test_decode_offset_fetch_response(self): encoded = b"".join([ struct.pack(">i", 42), # Correlation ID struct.pack(">i", 1), # One topics struct.pack(">h6s", 6, b"topic1"),# First topic struct.pack(">i", 2), # Two partitions struct.pack(">i", 2), # Partition 2 struct.pack(">q", 4), # Offset 4 struct.pack(">h4s", 4, b"meta"), # Metadata struct.pack(">h", 0), # No error struct.pack(">i", 4), # Partition 4 struct.pack(">q", 8), # Offset 8 struct.pack(">h4s", 4, b"meta"), # Metadata struct.pack(">h", 0), # No error ]) results = KafkaProtocol.decode_offset_fetch_response(encoded) self.assertEqual(set(results), set([ OffsetFetchResponse(topic = b'topic1', partition = 2, offset = 4, error = 0, metadata = b"meta"), OffsetFetchResponse(topic = b'topic1', partition = 4, offset = 8, error = 0, metadata = b"meta"), ])) @contextmanager def mock_create_message_fns(self): import kafka.protocol with patch.object(kafka.protocol.legacy, "create_message", return_value=sentinel.message): with patch.object(kafka.protocol.legacy, "create_gzip_message", return_value=sentinel.gzip_message): with patch.object(kafka.protocol.legacy, "create_snappy_message", return_value=sentinel.snappy_message): yield def test_create_message_set(self): messages = [(1, "k1"), (2, "k2"), (3, "k3")] # Default codec is CODEC_NONE. Expect list of regular messages. expect = [sentinel.message] * len(messages) with self.mock_create_message_fns(): message_set = create_message_set(messages) self.assertEqual(message_set, expect) # CODEC_NONE: Expect list of regular messages. expect = [sentinel.message] * len(messages) with self.mock_create_message_fns(): message_set = create_message_set(messages, CODEC_NONE) self.assertEqual(message_set, expect) # CODEC_GZIP: Expect list of one gzip-encoded message. expect = [sentinel.gzip_message] with self.mock_create_message_fns(): message_set = create_message_set(messages, CODEC_GZIP) self.assertEqual(message_set, expect) # CODEC_SNAPPY: Expect list of one snappy-encoded message. expect = [sentinel.snappy_message] with self.mock_create_message_fns(): message_set = create_message_set(messages, CODEC_SNAPPY) self.assertEqual(message_set, expect) # Unknown codec should raise UnsupportedCodecError. with self.assertRaises(UnsupportedCodecError): create_message_set(messages, -1) kafka-1.3.2/test/test_sender.py0000644001271300127130000000271213025302127016225 0ustar dpowers00000000000000# pylint: skip-file from __future__ import absolute_import import io import pytest from kafka.client_async import KafkaClient from kafka.cluster import ClusterMetadata import kafka.errors as Errors from kafka.future import Future from kafka.metrics import Metrics from kafka.producer.buffer import MessageSetBuffer from kafka.protocol.produce import ProduceRequest from kafka.producer.record_accumulator import RecordAccumulator, RecordBatch from kafka.producer.sender import Sender from kafka.structs import TopicPartition, OffsetAndMetadata @pytest.fixture def client(mocker): _cli = mocker.Mock(spec=KafkaClient(bootstrap_servers=[], api_version=(0, 9))) _cli.cluster = mocker.Mock(spec=ClusterMetadata()) return _cli @pytest.fixture def accumulator(): return RecordAccumulator() @pytest.fixture def metrics(): return Metrics() @pytest.fixture def sender(client, accumulator, metrics): return Sender(client, client.cluster, accumulator, metrics) @pytest.mark.parametrize(("api_version", "produce_version"), [ ((0, 10), 2), ((0, 9), 1), ((0, 8), 0) ]) def test_produce_request(sender, mocker, api_version, produce_version): sender.config['api_version'] = api_version tp = TopicPartition('foo', 0) records = MessageSetBuffer(io.BytesIO(), 100000) batch = RecordBatch(tp, records) produce_request = sender._produce_request(0, 0, 0, [batch]) assert isinstance(produce_request, ProduceRequest[produce_version]) kafka-1.3.2/test/test_util.py0000644001271300127130000001003312702214455015724 0ustar dpowers00000000000000# -*- coding: utf-8 -*- import struct import six from . import unittest import kafka.errors import kafka.util import kafka.structs class UtilTest(unittest.TestCase): @unittest.skip("Unwritten") def test_relative_unpack(self): pass def test_write_int_string(self): self.assertEqual( kafka.util.write_int_string(b'some string'), b'\x00\x00\x00\x0bsome string' ) def test_write_int_string__unicode(self): with self.assertRaises(TypeError) as cm: kafka.util.write_int_string(u'unicode') #: :type: TypeError te = cm.exception if six.PY2: self.assertIn('unicode', str(te)) else: self.assertIn('str', str(te)) self.assertIn('to be bytes', str(te)) def test_write_int_string__empty(self): self.assertEqual( kafka.util.write_int_string(b''), b'\x00\x00\x00\x00' ) def test_write_int_string__null(self): self.assertEqual( kafka.util.write_int_string(None), b'\xff\xff\xff\xff' ) def test_read_int_string(self): self.assertEqual(kafka.util.read_int_string(b'\xff\xff\xff\xff', 0), (None, 4)) self.assertEqual(kafka.util.read_int_string(b'\x00\x00\x00\x00', 0), (b'', 4)) self.assertEqual(kafka.util.read_int_string(b'\x00\x00\x00\x0bsome string', 0), (b'some string', 15)) def test_read_int_string__insufficient_data(self): with self.assertRaises(kafka.errors.BufferUnderflowError): kafka.util.read_int_string(b'\x00\x00\x00\x021', 0) def test_write_short_string(self): self.assertEqual( kafka.util.write_short_string(b'some string'), b'\x00\x0bsome string' ) def test_write_short_string__unicode(self): with self.assertRaises(TypeError) as cm: kafka.util.write_short_string(u'hello') #: :type: TypeError te = cm.exception if six.PY2: self.assertIn('unicode', str(te)) else: self.assertIn('str', str(te)) self.assertIn('to be bytes', str(te)) def test_write_short_string__empty(self): self.assertEqual( kafka.util.write_short_string(b''), b'\x00\x00' ) def test_write_short_string__null(self): self.assertEqual( kafka.util.write_short_string(None), b'\xff\xff' ) def test_write_short_string__too_long(self): with self.assertRaises(struct.error): kafka.util.write_short_string(b' ' * 33000) def test_read_short_string(self): self.assertEqual(kafka.util.read_short_string(b'\xff\xff', 0), (None, 2)) self.assertEqual(kafka.util.read_short_string(b'\x00\x00', 0), (b'', 2)) self.assertEqual(kafka.util.read_short_string(b'\x00\x0bsome string', 0), (b'some string', 13)) def test_read_int_string__insufficient_data2(self): with self.assertRaises(kafka.errors.BufferUnderflowError): kafka.util.read_int_string('\x00\x021', 0) def test_relative_unpack2(self): self.assertEqual( kafka.util.relative_unpack('>hh', b'\x00\x01\x00\x00\x02', 0), ((1, 0), 4) ) def test_relative_unpack3(self): with self.assertRaises(kafka.errors.BufferUnderflowError): kafka.util.relative_unpack('>hh', '\x00', 0) def test_group_by_topic_and_partition(self): t = kafka.structs.TopicPartition l = [ t("a", 1), t("a", 2), t("a", 3), t("b", 3), ] self.assertEqual(kafka.util.group_by_topic_and_partition(l), { "a": { 1: t("a", 1), 2: t("a", 2), 3: t("a", 3), }, "b": { 3: t("b", 3), } }) # should not be able to group duplicate topic-partitions t1 = t("a", 1) with self.assertRaises(AssertionError): kafka.util.group_by_topic_and_partition([t1, t1]) kafka-1.3.2/test/testutil.py0000644001271300127130000000737012702214455015577 0ustar dpowers00000000000000import functools import logging import operator import os import random import socket import string import time import uuid from six.moves import xrange from . import unittest from kafka import SimpleClient from kafka.structs import OffsetRequestPayload __all__ = [ 'random_string', 'get_open_port', 'kafka_versions', 'KafkaIntegrationTestCase', 'Timer', ] def random_string(l): return "".join(random.choice(string.ascii_letters) for i in xrange(l)) def kafka_versions(*versions): def version_str_to_list(s): return list(map(int, s.split('.'))) # e.g., [0, 8, 1, 1] def construct_lambda(s): if s[0].isdigit(): op_str = '=' v_str = s elif s[1].isdigit(): op_str = s[0] # ! < > = v_str = s[1:] elif s[2].isdigit(): op_str = s[0:2] # >= <= v_str = s[2:] else: raise ValueError('Unrecognized kafka version / operator: %s' % s) op_map = { '=': operator.eq, '!': operator.ne, '>': operator.gt, '<': operator.lt, '>=': operator.ge, '<=': operator.le } op = op_map[op_str] version = version_str_to_list(v_str) return lambda a: op(version_str_to_list(a), version) validators = map(construct_lambda, versions) def kafka_versions(func): @functools.wraps(func) def wrapper(self): kafka_version = os.environ.get('KAFKA_VERSION') if not kafka_version: self.skipTest("no kafka version set in KAFKA_VERSION env var") for f in validators: if not f(kafka_version): self.skipTest("unsupported kafka version") return func(self) return wrapper return kafka_versions def get_open_port(): sock = socket.socket() sock.bind(("", 0)) port = sock.getsockname()[1] sock.close() return port class KafkaIntegrationTestCase(unittest.TestCase): create_client = True topic = None zk = None server = None def setUp(self): super(KafkaIntegrationTestCase, self).setUp() if not os.environ.get('KAFKA_VERSION'): self.skipTest('Integration test requires KAFKA_VERSION') if not self.topic: topic = "%s-%s" % (self.id()[self.id().rindex(".") + 1:], random_string(10)) self.topic = topic if self.create_client: self.client = SimpleClient('%s:%d' % (self.server.host, self.server.port)) self.client.ensure_topic_exists(self.topic) self._messages = {} def tearDown(self): super(KafkaIntegrationTestCase, self).tearDown() if not os.environ.get('KAFKA_VERSION'): return if self.create_client: self.client.close() def current_offset(self, topic, partition): try: offsets, = self.client.send_offset_request([OffsetRequestPayload(topic, partition, -1, 1)]) except: # XXX: We've seen some UnknownErrors here and cant debug w/o server logs self.zk.child.dump_logs() self.server.child.dump_logs() raise else: return offsets.offsets[0] def msgs(self, iterable): return [ self.msg(x) for x in iterable ] def msg(self, s): if s not in self._messages: self._messages[s] = '%s-%s-%s' % (s, self.id(), str(uuid.uuid4())) return self._messages[s].encode('utf-8') def key(self, k): return k.encode('utf-8') class Timer(object): def __enter__(self): self.start = time.time() return self def __exit__(self, *args): self.end = time.time() self.interval = self.end - self.start