eredis-1.1.0/0000755000232200023220000000000013145103330013323 5ustar debalancedebalanceeredis-1.1.0/README.md0000644000232200023220000001706213145103330014610 0ustar debalancedebalance# eredis Non-blocking Redis client with focus on performance and robustness. Supported Redis features: * Any command, through eredis:q/2 * Transactions * Pipelining * Authentication & multiple dbs * Pubsub ## Example If you have Redis running on localhost, with default settings, you may copy and paste the following into a shell to try out Eredis: git clone git://github.com/wooga/eredis.git cd eredis ./rebar compile erl -pa ebin/ {ok, C} = eredis:start_link(). {ok, <<"OK">>} = eredis:q(C, ["SET", "foo", "bar"]). {ok, <<"bar">>} = eredis:q(C, ["GET", "foo"]). MSET and MGET: ```erlang KeyValuePairs = ["key1", "value1", "key2", "value2", "key3", "value3"]. {ok, <<"OK">>} = eredis:q(C, ["MSET" | KeyValuePairs]). {ok, Values} = eredis:q(C, ["MGET" | ["key1", "key2", "key3"]]). ``` HASH ```erlang HashObj = ["id", "objectId", "message", "message", "receiver", "receiver", "status", "read"]. eredis:q(C, ["HMSET", "key" | HashObj]). {ok, Values} = eredis:q(C, ["HGETALL", "key"]). ``` LIST ```erlang eredis:q(C, ["LPUSH", "keylist", "value"]). eredis:q(C, ["RPUSH", "keylist", "value"]). eredis:q(C, ["LRANGE", "keylist",0,-1]). ``` Transactions: ```erlang {ok, <<"OK">>} = eredis:q(C, ["MULTI"]). {ok, <<"QUEUED">>} = eredis:q(C, ["SET", "foo", "bar"]). {ok, <<"QUEUED">>} = eredis:q(C, ["SET", "bar", "baz"]). {ok, [<<"OK">>, <<"OK">>]} = eredis:q(C, ["EXEC"]). ``` Pipelining: ```erlang P1 = [["SET", a, "1"], ["LPUSH", b, "3"], ["LPUSH", b, "2"]]. [{ok, <<"OK">>}, {ok, <<"1">>}, {ok, <<"2">>}] = eredis:qp(C, P1). ``` Pubsub: ```erl 1> eredis_sub:sub_example(). received {subscribed,<<"foo">>,<0.34.0>} {<0.34.0>,<0.37.0>} 2> eredis_sub:pub_example(). received {message,<<"foo">>,<<"bar">>,<0.34.0>} ``` Pattern Subscribe: ```erl 1> eredis_sub:psub_example(). received {subscribed,<<"foo*">>,<0.33.0>} {<0.33.0>,<0.36.0>} 2> eredis_sub:ppub_example(). received {pmessage,<<"foo*">>,<<"foo123">>,<<"bar">>,<0.33.0>} ok 3> ``` EUnit tests: ```console ./rebar eunit ``` ## Commands Eredis has one main function to interact with redis, which is `eredis:q(Client::pid(), Command::iolist())`. The response will either be `{ok, Value::binary() | [binary()]}` or `{error, Message::binary()}`. The value is always the exact value returned by Redis, without any type conversion. If Redis returns a list of values, this list is returned in the exact same order without any type conversion. To send multiple requests to redis in a batch, aka. pipelining requests, you may use `eredis:qp(Client::pid(), [Command::iolist()])`. This function returns `{ok, [Value::binary()]}` where the values are the redis responses in the same order as the commands you provided. To start the client, use any of the `eredis:start_link/0,1,2,3,4,5` functions. They all include sensible defaults. `start_link/5` takes the following arguments: * Host, dns name or ip adress as string * Port, integer, default is 6379 * Database, integer or 0 for default database * Password, string or empty string([]) for no password * Reconnect sleep, integer of milliseconds to sleep between reconnect attempts ## Reconnecting on Redis down / network failure / timeout / etc When Eredis for some reason looses the connection to Redis, Eredis will keep trying to reconnect until a connection is successfully established, which includes the `AUTH` and `SELECT` calls. The sleep time between attempts to reconnect can be set in the `eredis:start_link/5` call. As long as the connection is down, Eredis will respond to any request immediately with `{error, no_connection}` without actually trying to connect. This serves as a kind of circuit breaker and prevents a stampede of clients just waiting for a failed connection attempt or `gen_server:call` timeout. Note: If Eredis is starting up and cannot connect, it will fail immediately with `{connection_error, Reason}`. ## Pubsub Thanks to Dave Peticolas (jdavisp3), eredis supports pubsub. `eredis_sub` offers a separate client that will forward channel messages from Redis to an Erlang process in a "active-once" pattern similar to gen_tcp sockets. After every message sent, the controlling process must acknowledge receipt using `eredis_sub:ack_message/1`. If the controlling process does not process messages fast enough, eredis will queue the messages up to a certain queue size controlled by configuration. When the max size is reached, eredis will either drop messages or crash, also based on configuration. Subscriptions are managed using `eredis_sub:subscribe/2` and `eredis_sub:unsubscribe/2`. When Redis acknowledges the change in subscription, a message is sent to the controlling process for each channel. eredis also supports Pattern Subscribe using `eredis_sub:psubscribe/2` and `eredis_sub:unsubscribe/2`. As with normal subscriptions, a message is sent to the controlling process for each channel. As of v1.0.7 the controlling process will be notified in case of reconnection attempts or failures. See `test/eredis_sub_tests` for details. ## AUTH and SELECT Eredis also implements the AUTH and SELECT calls for you. When the client is started with something else than default values for password and database, it will issue the `AUTH` and `SELECT` commands appropriately, even when reconnecting after a timeout. ## Benchmarking Using basho_bench(https://github.com/basho/basho_bench/) you may benchmark Eredis on your own hardware using the provided config and driver. See `priv/basho_bench_driver_eredis.config` and `src/basho_bench_driver_eredis.erl`. ## Queueing Eredis uses the same queueing mechanism as Erldis. `eredis:q/2` uses `gen_server:call/2` to do a blocking call to the client gen_server. The client will immediately send the request to Redis, add the caller to the queue and reply with `noreply`. This frees the gen_server up to accept new requests and parse responses as they come on the socket. When data is received on the socket, we call `eredis_parser:parse/2` until it returns a value, we then use `gen_server:reply/2` to reply to the first process waiting in the queue. This queueing mechanism works because Redis guarantees that the response will be in the same order as the requests. ## Response parsing The response parser is the biggest difference between Eredis and other libraries like Erldis, redis-erl and redis_pool. The common approach is to either directly block or use active once to get the first part of the response, then repeatedly use `gen_tcp:recv/2` to get more data when needed. Profiling identified this as a bottleneck, in particular for `MGET` and `HMGET`. To be as fast as possible, Eredis takes a different approach. The socket is always set to active once, which will let us receive data fast without blocking the gen_server. The tradeoff is that we must parse partial responses, which makes the parser more complex. In order to make multibulk responses more efficient, the parser will parse all data available and continue where it left off when more data is available. ## Future improvements When the parser is accumulating data, a new binary is generated for every call to `parse/2`. This might create binaries that will be reference counted. This could be improved by replacing it with an iolist. When parsing bulk replies, the parser knows the size of the bulk. If the bulk is big and would come in many chunks, this could improved by having the client explicitly use `gen_tcp:recv/2` to fetch the entire bulk at once. ## Credits Although this project is almost a complete rewrite, many patterns are the same as you find in Erldis, most notably the queueing of requests. `create_multibulk/1` and `to_binary/1` were taken verbatim from Erldis. eredis-1.1.0/CHANGELOG.md0000644000232200023220000000546513145103330015146 0ustar debalancedebalance# CHANGELOG ## v1.1.0 * Merged a ton of of old and neglected pull requests. Thanks to patient contributors: * Emil Falk * Evgeny Khramtsov * Kevin Wilson * Luis Rascão * Аверьянов Илья (savonarola on github) * ololoru * Giacomo Olgeni * Removed rebar binary, made everything a bit more rebar3 & mix friendly. ## v1.0.8 * Fixed include directive to work with rebar 2.5.1. Thanks to Feng Hao for the patch. ## v1.0.7 * If an eredis_sub_client needs to reconnect to Redis, the controlling process is now notified with the message `{eredis_reconnect_attempt, Pid}`. If the reconnection attempt fails, the message is `{eredis_reconnect_failed, Pid, Reason}`. Thanks to Piotr Nosek for the patch. * No more deprecation warnings of the `queue` type on OTP 17. Thanks to Daniel Kempkens for the patch. * Various spec fixes. Thanks to Hernan Rivas Acosta and Anton Kalyaev. ## v1.0.6 * If the connection to Redis is lost, requests in progress will receive `{error, tcp_closed}` instead of the `gen_server:call` timing out. Thanks to Seth Falcon for the patch. ## v1.0.5 * Added support for not selecting any specific database. Thanks to Mikl Kurkov for the patch. ## v1.0.4 * Added `eredis:q_noreply/2` which sends a fire-and-forget request to Redis. Thanks to Ransom Richardson for the patch. * Various type annotation improvements, typo fixes and robustness improvements. Thanks to Michael Gregson, Matthew Conway and Ransom Richardson. ## v1.0.3 * Fixed bug in eredis_sub where when the connection to Redis was lost, the socket would not be set into {active, once} on reconnect. Thanks to georgeye for the patch. ## v1.0.2 * Fixed bug in eredis_sub where the socket was incorrectly set to `{active, once}` twice. At large volumes of messages, this resulted in too many messages from the socket and we would be unable to keep up. Thanks to pmembrey for reporting. ## v1.0 * Support added for pubsub thanks to Dave Peticolas (jdavisp3). Implemented in `eredis_sub` and `eredis_sub_client` is a subscriber that will forward messages from Redis to an Erlang process with flow control. The user can configure to either drop messages or crash the driver if a certain queue size inside the driver is reached. * Fixed error handling when eredis starts up and Redis is still loading the dataset into memory. ## v0.7.0 * Support added for pipelining requests, which allows batching multiple requests in a single call to eredis. Thanks to Dave Peticolas (jdavisp3) for the implementation. ## v0.6.0 * Support added for transactions, by Dave Peticolas (jdavisp3) who implemented parsing of nested multibulks. ## v0.5.0 * Configurable reconnect sleep time, by Valentino Volonghi (dialtone) * Support for using eredis as a poolboy worker, by Valentino Volonghi (dialtone) eredis-1.1.0/Makefile0000644000232200023220000000127113145103330014764 0ustar debalancedebalanceAPP=eredis PRE17 := $(shell ERL_FLAGS="" erl -eval 'io:format("~s~n", [case re:run(erlang:system_info(otp_release), "^R") of nomatch -> ""; _ -> pre17 end]), halt().' -noshell) .PHONY: all compile clean Emakefile all: compile compile: ebin/$(APP).app Emakefile erl -noinput -eval 'up_to_date = make:all()' -s erlang halt clean: rm -f -- ebin/*.beam Emakefile ebin/$(APP).app ebin/$(APP).app: src/$(APP).app.src mkdir -p ebin cp -f -- $< $@ ifdef DEBUG EXTRA_OPTS:=debug_info, endif ifdef TEST EXTRA_OPTS:=$(EXTRA_OPTS) {d,'TEST', true}, endif ifndef PRE17 EXTRA_OPTS:=$(EXTRA_OPTS) {d,namespaced_types}, endif Emakefile: Emakefile.src sed "s/{{EXTRA_OPTS}}/$(EXTRA_OPTS)/" $< > $@ eredis-1.1.0/priv/0000755000232200023220000000000013145103330014303 5ustar debalancedebalanceeredis-1.1.0/priv/basho_bench_eredis_pipeline.config0000644000232200023220000000053313145103330023146 0ustar debalancedebalance{mode, max}. %{mode, {rate, 5}}. {duration, 15}. {concurrent, 30}. {driver, basho_bench_driver_eredis}. {code_paths, ["../eredis/ebin"]}. {operations, [{pipeline_get,100}, {pipeline_put,1}]}. {key_generator, {uniform_int, 10000}}. {value_generator, {function, basho_bench_driver_eredis, value_gen, []}}. %{value_generator, {fixed_bin, 1}}. eredis-1.1.0/priv/basho_bench_eredis.config0000644000232200023220000000050713145103330021262 0ustar debalancedebalance{mode, max}. %{mode, {rate, 5}}. {duration, 15}. {concurrent, 30}. {driver, basho_bench_driver_eredis}. {code_paths, ["../eredis/ebin"]}. {operations, [{get,1}, {put,4}]}. {key_generator, {uniform_int, 10000}}. {value_generator, {function, basho_bench_driver_eredis, value_gen, []}}. %{value_generator, {fixed_bin, 1}}. eredis-1.1.0/priv/basho_bench_erldis.config0000644000232200023220000000055013145103330021267 0ustar debalancedebalance{mode, max}. %{mode, {rate, 5}}. {duration, 15}. {concurrent, 10}. {driver, basho_bench_driver_erldis}. {code_paths, ["../eredis/ebin", "../erldis/ebin/"]}. {operations, [{get,1}, {put,4}]}. {key_generator, {uniform_int, 10000}}. {value_generator, {function, basho_bench_driver_erldis, value_gen, []}}. {value_generator, {fixed_bin, 64}}. eredis-1.1.0/src/0000755000232200023220000000000013145103330014112 5ustar debalancedebalanceeredis-1.1.0/src/eredis_sub_client.erl0000644000232200023220000003165313145103330020310 0ustar debalancedebalance%% %% eredis_pubsub_client %% %% This client implements a subscriber to a Redis pubsub channel. It %% is implemented in the same way as eredis_client, except channel %% messages are streamed to the controlling process. Messages are %% queued and delivered when the client acknowledges receipt. %% %% There is one consuming process per eredis_sub_client. -module(eredis_sub_client). -behaviour(gen_server). -include("eredis.hrl"). -include("eredis_sub.hrl"). %% API -export([start_link/6, stop/1]). %% gen_server callbacks -export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2, code_change/3]). %% %% API %% -spec start_link(Host::list(), Port::integer(), Password::string(), ReconnectSleep::reconnect_sleep(), MaxQueueSize::integer() | infinity, QueueBehaviour::drop | exit) -> {ok, Pid::pid()} | {error, Reason::term()}. start_link(Host, Port, Password, ReconnectSleep, MaxQueueSize, QueueBehaviour) -> Args = [Host, Port, Password, ReconnectSleep, MaxQueueSize, QueueBehaviour], gen_server:start_link(?MODULE, Args, []). stop(Pid) -> gen_server:call(Pid, stop). %%==================================================================== %% gen_server callbacks %%==================================================================== init([Host, Port, Password, ReconnectSleep, MaxQueueSize, QueueBehaviour]) -> State = #state{host = Host, port = Port, password = list_to_binary(Password), reconnect_sleep = ReconnectSleep, channels = [], parser_state = eredis_parser:init(), msg_queue = queue:new(), max_queue_size = MaxQueueSize, queue_behaviour = QueueBehaviour}, case connect(State) of {ok, NewState} -> ok = inet:setopts(NewState#state.socket, [{active, once}]), {ok, NewState}; {error, Reason} -> {stop, Reason} end. %% Set the controlling process. All messages on all channels are directed here. handle_call({controlling_process, Pid}, _From, State) -> case State#state.controlling_process of undefined -> ok; {OldRef, _OldPid} -> erlang:demonitor(OldRef) end, Ref = erlang:monitor(process, Pid), {reply, ok, State#state{controlling_process={Ref, Pid}, msg_state = ready}}; handle_call(get_channels, _From, State) -> {reply, {ok, State#state.channels}, State}; handle_call(stop, _From, State) -> {stop, normal, ok, State}; handle_call(_Request, _From, State) -> {reply, unknown_request, State}. %% Controlling process acks, but we have no connection. When the %% connection comes back up, we should be ready to forward a message %% again. handle_cast({ack_message, Pid}, #state{controlling_process={_, Pid}, socket = undefined} = State) -> {noreply, State#state{msg_state = ready}}; %% Controlling process acknowledges receipt of previous message. Send %% the next if there is any messages queued or ask for more on the %% socket. handle_cast({ack_message, Pid}, #state{controlling_process={_, Pid}} = State) -> NewState = case queue:out(State#state.msg_queue) of {empty, _Queue} -> State#state{msg_state = ready}; {{value, Msg}, Queue} -> send_to_controller(Msg, State), State#state{msg_queue = Queue, msg_state = need_ack} end, {noreply, NewState}; handle_cast({subscribe, Pid, Channels}, #state{controlling_process = {_, Pid}} = State) -> Command = eredis:create_multibulk(["SUBSCRIBE" | Channels]), ok = gen_tcp:send(State#state.socket, Command), NewChannels = add_channels(Channels, State#state.channels), {noreply, State#state{channels = NewChannels}}; handle_cast({psubscribe, Pid, Channels}, #state{controlling_process = {_, Pid}} = State) -> Command = eredis:create_multibulk(["PSUBSCRIBE" | Channels]), ok = gen_tcp:send(State#state.socket, Command), NewChannels = add_channels(Channels, State#state.channels), {noreply, State#state{channels = NewChannels}}; handle_cast({unsubscribe, Pid, Channels}, #state{controlling_process = {_, Pid}} = State) -> Command = eredis:create_multibulk(["UNSUBSCRIBE" | Channels]), ok = gen_tcp:send(State#state.socket, Command), NewChannels = remove_channels(Channels, State#state.channels), {noreply, State#state{channels = NewChannels}}; handle_cast({punsubscribe, Pid, Channels}, #state{controlling_process = {_, Pid}} = State) -> Command = eredis:create_multibulk(["PUNSUBSCRIBE" | Channels]), ok = gen_tcp:send(State#state.socket, Command), NewChannels = remove_channels(Channels, State#state.channels), {noreply, State#state{channels = NewChannels}}; handle_cast({ack_message, _}, State) -> {noreply, State}; handle_cast(_Msg, State) -> {noreply, State}. %% Receive data from socket, see handle_response/2 handle_info({tcp, _Socket, Bs}, State) -> ok = inet:setopts(State#state.socket, [{active, once}]), NewState = handle_response(Bs, State), case queue:len(NewState#state.msg_queue) > NewState#state.max_queue_size of true -> case State#state.queue_behaviour of drop -> Msg = {dropped, queue:len(NewState#state.msg_queue)}, send_to_controller(Msg, NewState), {noreply, NewState#state{msg_queue = queue:new()}}; exit -> {stop, max_queue_size, State} end; false -> {noreply, NewState} end; handle_info({tcp_error, _Socket, _Reason}, State) -> %% This will be followed by a close {noreply, State}; %% Socket got closed, for example by Redis terminating idle %% clients. If desired, spawn of a new process which will try to reconnect and %% notify us when Redis is ready. In the meantime, we can respond with %% an error message to all our clients. handle_info({tcp_closed, _Socket}, #state{reconnect_sleep = no_reconnect} = State) -> %% If we aren't going to reconnect, then there is nothing else for this process to do. {stop, normal, State#state{socket = undefined}}; handle_info({tcp_closed, _Socket}, State) -> Self = self(), send_to_controller({eredis_disconnected, Self}, State), spawn(fun() -> reconnect_loop(Self, State) end), %% Throw away the socket. The absence of a socket is used to %% signal we are "down" {noreply, State#state{socket = undefined}}; %% Controller might want to be notified about every reconnect attempt handle_info(reconnect_attempt, State) -> send_to_controller({eredis_reconnect_attempt, self()}, State), {noreply, State}; %% Controller might want to be notified about every reconnect failure and reason handle_info({reconnect_failed, Reason}, State) -> send_to_controller({eredis_reconnect_failed, self(), {error, {connection_error, Reason}}}, State), {noreply, State}; %% Redis is ready to accept requests, the given Socket is a socket %% already connected and authenticated. handle_info({connection_ready, Socket}, #state{socket = undefined} = State) -> send_to_controller({eredis_connected, self()}, State), ok = inet:setopts(Socket, [{active, once}]), {noreply, State#state{socket = Socket}}; %% Our controlling process is down. handle_info({'DOWN', Ref, process, Pid, _Reason}, #state{controlling_process={Ref, Pid}} = State) -> {stop, shutdown, State#state{controlling_process=undefined, msg_state=ready, msg_queue=queue:new()}}; %% eredis can be used in Poolboy, but it requires to support a simple API %% that Poolboy uses to manage the connections. handle_info(stop, State) -> {stop, shutdown, State}; handle_info(_Info, State) -> {stop, {unhandled_message, _Info}, State}. terminate(_Reason, State) -> case State#state.socket of undefined -> ok; Socket -> gen_tcp:close(Socket) end, ok. code_change(_OldVsn, State, _Extra) -> {ok, State}. %%-------------------------------------------------------------------- %%% Internal functions %%-------------------------------------------------------------------- -spec remove_channels([binary()], [binary()]) -> [binary()]. remove_channels(Channels, OldChannels) -> lists:foldl(fun lists:delete/2, OldChannels, Channels). -spec add_channels([binary()], [binary()]) -> [binary()]. add_channels(Channels, OldChannels) -> lists:foldl(fun(C, Cs) -> case lists:member(C, Cs) of true -> Cs; false -> [C|Cs] end end, OldChannels, Channels). -spec handle_response(Data::binary(), State::#state{}) -> NewState::#state{}. %% @doc: Handle the response coming from Redis. This should only be %% channel messages that we should forward to the controlling process %% or queue if the previous message has not been acked. If there are %% more than a single response in the data we got, queue the responses %% and serve them up when the controlling process is ready handle_response(Data, #state{parser_state = ParserState} = State) -> case eredis_parser:parse(ParserState, Data) of {ReturnCode, Value, NewParserState} -> reply({ReturnCode, Value}, State#state{parser_state=NewParserState}); {ReturnCode, Value, Rest, NewParserState} -> NewState = reply({ReturnCode, Value}, State#state{parser_state=NewParserState}), handle_response(Rest, NewState); {continue, NewParserState} -> State#state{parser_state = NewParserState} end. %% @doc: Sends a reply to the controlling process if the process has %% acknowledged the previous process, otherwise the message is queued %% for later delivery. reply({ok, [<<"message">>, Channel, Message]}, State) -> queue_or_send({message, Channel, Message, self()}, State); reply({ok, [<<"pmessage">>, Pattern, Channel, Message]}, State) -> queue_or_send({pmessage, Pattern, Channel, Message, self()}, State); reply({ok, [<<"subscribe">>, Channel, _]}, State) -> queue_or_send({subscribed, Channel, self()}, State); reply({ok, [<<"psubscribe">>, Channel, _]}, State) -> queue_or_send({subscribed, Channel, self()}, State); reply({ok, [<<"unsubscribe">>, Channel, _]}, State) -> queue_or_send({unsubscribed, Channel, self()}, State); reply({ok, [<<"punsubscribe">>, Channel, _]}, State) -> queue_or_send({unsubscribed, Channel, self()}, State); reply({ReturnCode, Value}, State) -> throw({unexpected_response_from_redis, ReturnCode, Value, State}). queue_or_send(Msg, State) -> case State#state.msg_state of need_ack -> MsgQueue = queue:in(Msg, State#state.msg_queue), State#state{msg_queue = MsgQueue}; ready -> send_to_controller(Msg, State), State#state{msg_state = need_ack} end. %% @doc: Helper for connecting to Redis. These commands are %% synchronous and if Redis returns something we don't expect, we %% crash. Returns {ok, State} or {error, Reason}. connect(State) -> case gen_tcp:connect(State#state.host, State#state.port, ?SOCKET_OPTS) of {ok, Socket} -> case authenticate(Socket, State#state.password) of ok -> {ok, State#state{socket = Socket}}; {error, Reason} -> {error, {authentication_error, Reason}} end; {error, Reason} -> {error, {connection_error, Reason}} end. authenticate(_Socket, <<>>) -> ok; authenticate(Socket, Password) -> eredis_client:do_sync_command(Socket, ["AUTH", " \"", Password, "\"\r\n"]). %% @doc: Loop until a connection can be established, this includes %% successfully issuing the auth and select calls. When we have a %% connection, give the socket to the redis client. reconnect_loop(Client, #state{reconnect_sleep=ReconnectSleep}=State) -> Client ! reconnect_attempt, case catch(connect(State)) of {ok, #state{socket = Socket}} -> gen_tcp:controlling_process(Socket, Client), Client ! {connection_ready, Socket}; {error, Reason} -> Client ! {reconnect_failed, Reason}, timer:sleep(ReconnectSleep), reconnect_loop(Client, State); %% Something bad happened when connecting, like Redis might be %% loading the dataset and we got something other than 'OK' in %% auth or select _ -> timer:sleep(ReconnectSleep), reconnect_loop(Client, State) end. send_to_controller(_Msg, #state{controlling_process=undefined}) -> ok; send_to_controller(Msg, #state{controlling_process={_Ref, Pid}}) -> %%error_logger:info_msg("~p ! ~p~n", [Pid, Msg]), Pid ! Msg. eredis-1.1.0/src/eredis_client.erl0000644000232200023220000003475613145103330017446 0ustar debalancedebalance%% %% eredis_client %% %% The client is implemented as a gen_server which keeps one socket %% open to a single Redis instance. Users call us using the API in %% eredis.erl. %% %% The client works like this: %% * When starting up, we connect to Redis with the given connection %% information, or fail. %% * Users calls us using gen_server:call, we send the request to Redis, %% add the calling process at the end of the queue and reply with %% noreply. We are then free to handle new requests and may reply to %% the user later. %% * We receive data on the socket, we parse the response and reply to %% the client at the front of the queue. If the parser does not have %% enough data to parse the complete response, we will wait for more %% data to arrive. %% * For pipeline commands, we include the number of responses we are %% waiting for in each element of the queue. Responses are queued until %% we have all the responses we need and then reply with all of them. %% -module(eredis_client). -behaviour(gen_server). -include("eredis.hrl"). %% API -export([start_link/6, stop/1, select_database/2]). -export([do_sync_command/2]). %% gen_server callbacks -export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2, code_change/3]). -record(state, { host :: string() | undefined, port :: integer() | undefined, password :: binary() | undefined, database :: binary() | undefined, reconnect_sleep :: reconnect_sleep() | undefined, connect_timeout :: integer() | undefined, socket :: port() | undefined, parser_state :: #pstate{} | undefined, queue :: eredis_queue() | undefined }). %% %% API %% -spec start_link(Host::list(), Port::integer(), Database::integer() | undefined, Password::string(), ReconnectSleep::reconnect_sleep(), ConnectTimeout::integer() | undefined) -> {ok, Pid::pid()} | {error, Reason::term()}. start_link(Host, Port, Database, Password, ReconnectSleep, ConnectTimeout) -> gen_server:start_link(?MODULE, [Host, Port, Database, Password, ReconnectSleep, ConnectTimeout], []). stop(Pid) -> gen_server:call(Pid, stop). %%==================================================================== %% gen_server callbacks %%==================================================================== init([Host, Port, Database, Password, ReconnectSleep, ConnectTimeout]) -> State = #state{host = Host, port = Port, database = read_database(Database), password = list_to_binary(Password), reconnect_sleep = ReconnectSleep, connect_timeout = ConnectTimeout, parser_state = eredis_parser:init(), queue = queue:new()}, case connect(State) of {ok, NewState} -> {ok, NewState}; {error, Reason} -> {stop, Reason} end. handle_call({request, Req}, From, State) -> do_request(Req, From, State); handle_call({pipeline, Pipeline}, From, State) -> do_pipeline(Pipeline, From, State); handle_call(stop, _From, State) -> {stop, normal, ok, State}; handle_call(_Request, _From, State) -> {reply, unknown_request, State}. handle_cast({request, Req}, State) -> case do_request(Req, undefined, State) of {reply, _Reply, State1} -> {noreply, State1}; {noreply, State1} -> {noreply, State1} end; handle_cast({request, Req, Pid}, State) -> case do_request(Req, Pid, State) of {reply, Reply, State1} -> safe_send(Pid, {response, Reply}), {noreply, State1}; {noreply, State1} -> {noreply, State1} end; handle_cast(_Msg, State) -> {noreply, State}. %% Receive data from socket, see handle_response/2. Match `Socket' to %% enforce sanity. handle_info({tcp, Socket, Bs}, #state{socket = Socket} = State) -> ok = inet:setopts(Socket, [{active, once}]), {noreply, handle_response(Bs, State)}; handle_info({tcp, Socket, _}, #state{socket = OurSocket} = State) when OurSocket =/= Socket -> %% Ignore tcp messages when the socket in message doesn't match %% our state. In order to test behavior around receiving %% tcp_closed message with clients waiting in queue, we send a %% fake tcp_close message. This allows us to ignore messages that %% arrive after that while we are reconnecting. {noreply, State}; handle_info({tcp_error, _Socket, _Reason}, State) -> %% This will be followed by a close {noreply, State}; %% Socket got closed, for example by Redis terminating idle %% clients. If desired, spawn of a new process which will try to reconnect and %% notify us when Redis is ready. In the meantime, we can respond with %% an error message to all our clients. handle_info({tcp_closed, _Socket}, #state{reconnect_sleep = no_reconnect, queue = Queue} = State) -> reply_all({error, tcp_closed}, Queue), %% If we aren't going to reconnect, then there is nothing else for %% this process to do. {stop, normal, State#state{socket = undefined}}; handle_info({tcp_closed, _Socket}, #state{queue = Queue} = State) -> Self = self(), spawn(fun() -> reconnect_loop(Self, State) end), %% tell all of our clients what has happened. reply_all({error, tcp_closed}, Queue), %% Throw away the socket and the queue, as we will never get a %% response to the requests sent on the old socket. The absence of %% a socket is used to signal we are "down" {noreply, State#state{socket = undefined, queue = queue:new()}}; %% Redis is ready to accept requests, the given Socket is a socket %% already connected and authenticated. handle_info({connection_ready, Socket}, #state{socket = undefined} = State) -> {noreply, State#state{socket = Socket}}; %% eredis can be used in Poolboy, but it requires to support a simple API %% that Poolboy uses to manage the connections. handle_info(stop, State) -> {stop, shutdown, State}; handle_info(_Info, State) -> {stop, {unhandled_message, _Info}, State}. terminate(_Reason, State) -> case State#state.socket of undefined -> ok; Socket -> gen_tcp:close(Socket) end, ok. code_change(_OldVsn, State, _Extra) -> {ok, State}. %%-------------------------------------------------------------------- %%% Internal functions %%-------------------------------------------------------------------- -spec do_request(Req::iolist(), From::pid(), #state{}) -> {noreply, #state{}} | {reply, Reply::any(), #state{}}. %% @doc: Sends the given request to redis. If we do not have a %% connection, returns error. do_request(_Req, _From, #state{socket = undefined} = State) -> {reply, {error, no_connection}, State}; do_request(Req, From, State) -> case gen_tcp:send(State#state.socket, Req) of ok -> NewQueue = queue:in({1, From}, State#state.queue), {noreply, State#state{queue = NewQueue}}; {error, Reason} -> {reply, {error, Reason}, State} end. -spec do_pipeline(Pipeline::pipeline(), From::pid(), #state{}) -> {noreply, #state{}} | {reply, Reply::any(), #state{}}. %% @doc: Sends the entire pipeline to redis. If we do not have a %% connection, returns error. do_pipeline(_Pipeline, _From, #state{socket = undefined} = State) -> {reply, {error, no_connection}, State}; do_pipeline(Pipeline, From, State) -> case gen_tcp:send(State#state.socket, Pipeline) of ok -> NewQueue = queue:in({length(Pipeline), From, []}, State#state.queue), {noreply, State#state{queue = NewQueue}}; {error, Reason} -> {reply, {error, Reason}, State} end. -spec handle_response(Data::binary(), State::#state{}) -> NewState::#state{}. %% @doc: Handle the response coming from Redis. This includes parsing %% and replying to the correct client, handling partial responses, %% handling too much data and handling continuations. handle_response(Data, #state{parser_state = ParserState, queue = Queue} = State) -> case eredis_parser:parse(ParserState, Data) of %% Got complete response, return value to client {ReturnCode, Value, NewParserState} -> NewQueue = reply({ReturnCode, Value}, Queue), State#state{parser_state = NewParserState, queue = NewQueue}; %% Got complete response, with extra data, reply to client and %% recurse over the extra data {ReturnCode, Value, Rest, NewParserState} -> NewQueue = reply({ReturnCode, Value}, Queue), handle_response(Rest, State#state{parser_state = NewParserState, queue = NewQueue}); %% Parser needs more data, the parser state now contains the %% continuation data and we will try calling parse again when %% we have more data {continue, NewParserState} -> State#state{parser_state = NewParserState} end. %% @doc: Sends a value to the first client in queue. Returns the new %% queue without this client. If we are still waiting for parts of a %% pipelined request, push the reply to the the head of the queue and %% wait for another reply from redis. reply(Value, Queue) -> case queue:out(Queue) of {{value, {1, From}}, NewQueue} -> safe_reply(From, Value), NewQueue; {{value, {1, From, Replies}}, NewQueue} -> safe_reply(From, lists:reverse([Value | Replies])), NewQueue; {{value, {N, From, Replies}}, NewQueue} when N > 1 -> queue:in_r({N - 1, From, [Value | Replies]}, NewQueue); {empty, Queue} -> %% Oops error_logger:info_msg("Nothing in queue, but got value from parser~n"), throw(empty_queue) end. %% @doc Send `Value' to each client in queue. Only useful for sending %% an error message. Any in-progress reply data is ignored. -spec reply_all(any(), eredis_queue()) -> ok. reply_all(Value, Queue) -> case queue:peek(Queue) of empty -> ok; {value, Item} -> safe_reply(receipient(Item), Value), reply_all(Value, queue:drop(Queue)) end. receipient({_, From}) -> From; receipient({_, From, _}) -> From. safe_reply(undefined, _Value) -> ok; safe_reply(Pid, Value) when is_pid(Pid) -> safe_send(Pid, {response, Value}); safe_reply(From, Value) -> gen_server:reply(From, Value). safe_send(Pid, Value) -> try erlang:send(Pid, Value) catch Err:Reason -> error_logger:info_msg("Failed to send message to ~p with reason ~p~n", [Pid, {Err, Reason}]) end. %% @doc: Helper for connecting to Redis, authenticating and selecting %% the correct database. These commands are synchronous and if Redis %% returns something we don't expect, we crash. Returns {ok, State} or %% {SomeError, Reason}. connect(State) -> {ok, {AFamily, Addr}} = get_addr(State#state.host), case gen_tcp:connect(Addr, State#state.port, [AFamily | ?SOCKET_OPTS], State#state.connect_timeout) of {ok, Socket} -> case authenticate(Socket, State#state.password) of ok -> case select_database(Socket, State#state.database) of ok -> {ok, State#state{socket = Socket}}; {error, Reason} -> {error, {select_error, Reason}} end; {error, Reason} -> {error, {authentication_error, Reason}} end; {error, Reason} -> {error, {connection_error, Reason}} end. get_addr(Hostname) -> case inet:parse_address(Hostname) of {ok, {_,_,_,_} = Addr} -> {ok, {inet, Addr}}; {ok, {_,_,_,_,_,_,_,_} = Addr} -> {ok, {inet6, Addr}}; {error, einval} -> case inet:getaddr(Hostname, inet6) of {error, _} -> case inet:getaddr(Hostname, inet) of {ok, Addr}-> {ok, {inet, Addr}}; {error, _} = Res -> Res end; {ok, Addr} -> {ok, {inet6, Addr}} end end. select_database(_Socket, undefined) -> ok; select_database(_Socket, <<"0">>) -> ok; select_database(Socket, Database) -> do_sync_command(Socket, ["SELECT", " ", Database, "\r\n"]). authenticate(_Socket, <<>>) -> ok; authenticate(Socket, Password) -> do_sync_command(Socket, ["AUTH", " \"", Password, "\"\r\n"]). %% @doc: Executes the given command synchronously, expects Redis to %% return "+OK\r\n", otherwise it will fail. do_sync_command(Socket, Command) -> ok = inet:setopts(Socket, [{active, false}]), case gen_tcp:send(Socket, Command) of ok -> %% Hope there's nothing else coming down on the socket.. case gen_tcp:recv(Socket, 0, ?RECV_TIMEOUT) of {ok, <<"+OK\r\n">>} -> ok = inet:setopts(Socket, [{active, once}]), ok; Other -> {error, {unexpected_data, Other}} end; {error, Reason} -> {error, Reason} end. %% @doc: Loop until a connection can be established, this includes %% successfully issuing the auth and select calls. When we have a %% connection, give the socket to the redis client. reconnect_loop(Client, #state{reconnect_sleep = ReconnectSleep} = State) -> case catch(connect(State)) of {ok, #state{socket = Socket}} -> Client ! {connection_ready, Socket}, gen_tcp:controlling_process(Socket, Client), Msgs = get_all_messages([]), [Client ! M || M <- Msgs]; {error, _Reason} -> timer:sleep(ReconnectSleep), reconnect_loop(Client, State); %% Something bad happened when connecting, like Redis might be %% loading the dataset and we got something other than 'OK' in %% auth or select _ -> timer:sleep(ReconnectSleep), reconnect_loop(Client, State) end. read_database(undefined) -> undefined; read_database(Database) when is_integer(Database) -> list_to_binary(integer_to_list(Database)). get_all_messages(Acc) -> receive M -> [M | Acc] after 0 -> lists:reverse(Acc) end. eredis-1.1.0/src/eredis.app.src0000644000232200023220000000035313145103330016656 0ustar debalancedebalance{application, eredis, [ {description, "Erlang Redis Client"}, {vsn, "1.1.0"}, {modules, [eredis, eredis_client, eredis_parser, eredis_sub, eredis_sub_client]}, {registered, []}, {applications, [kernel, stdlib]} ]}. eredis-1.1.0/src/eredis_parser.erl0000644000232200023220000002566313145103330017461 0ustar debalancedebalance%% %% Parser of the Redis protocol, see http://redis.io/topics/protocol %% %% The idea behind this parser is that we accept any binary data %% available on the socket. If there is not enough data to parse a %% complete response, we ask the caller to call us later when there is %% more data. If there is too much data, we only parse the first %% response and let the caller call us again with the rest. %% %% This approach lets us write a "pure" parser that does not depend on %% manipulating the socket, which erldis and redis-erl is %% doing. Instead, we may ask the socket to send us data as fast as %% possible and parse it continously. The overhead of manipulating the %% socket when parsing multibulk responses is killing the performance %% of erldis. %% %% Future improvements: %% * When we return a bulk continuation, we also include the size of %% the bulk. The caller may use this to explicitly call %% gen_tcp:recv/2 with the desired size. -module(eredis_parser). -include("eredis.hrl"). -include_lib("eunit/include/eunit.hrl"). -export([init/0, parse/2]). %% Exported for testing -export([parse_bulk/1, parse_bulk/2, parse_multibulk/1, parse_multibulk/2, buffer_create/0, buffer_create/1]). %% %% API %% %% @doc: Initialize the parser init() -> #pstate{}. -spec parse(State::#pstate{}, Data::binary()) -> {ok, return_value(), NewState::#pstate{}} | {ok, return_value(), Rest::binary(), NewState::#pstate{}} | {error, ErrString::binary(), NewState::#pstate{}} | {error, ErrString::binary(), Rest::binary(), NewState::#pstate{}} | {continue, NewState::#pstate{}}. %% @doc: Parses the (possibly partial) response from Redis. Returns %% either {ok, Value, NewState}, {ok, Value, Rest, NewState} or %% {continue, NewState}. External entry point for parsing. %% %% In case {ok, Value, NewState} is returned, Value contains the value %% returned by Redis. NewState will be an empty parser state. %% %% In case {ok, Value, Rest, NewState} is returned, Value contains the %% most recent value returned by Redis, while Rest contains any extra %% data that was given, but was not part of the same response. In this %% case you should immeditely call parse again with Rest as the Data %% argument and NewState as the State argument. %% %% In case {continue, NewState} is returned, more data is needed %% before a complete value can be returned. As soon as you have more %% data, call parse again with NewState as the State argument and any %% new binary data as the Data argument. %% Parser in initial state, the data we receive will be the beginning %% of a response parse(#pstate{state = undefined} = State, NewData) -> %% Look at the first byte to get the type of reply case NewData of %% Status <<$+, Data/binary>> -> return_result(parse_simple(Data), State, status_continue); %% Error <<$-, Data/binary>> -> return_error(parse_simple(Data), State, status_continue); %% Integer reply <<$:, Data/binary>> -> return_result(parse_simple(Data), State, status_continue); %% Multibulk <<$*, _Rest/binary>> -> return_result(parse_multibulk(NewData), State, multibulk_continue); %% Bulk <<$$, _Rest/binary>> -> return_result(parse_bulk(NewData), State, bulk_continue); _ -> %% TODO: Handle the case where we start parsing a new %% response, but cannot make any sense of it {error, unknown_response} end; %% The following clauses all match on different continuation states parse(#pstate{state = bulk_continue, continuation_data = ContinuationData} = State, NewData) -> return_result(parse_bulk(ContinuationData, NewData), State, bulk_continue); parse(#pstate{state = multibulk_continue, continuation_data = ContinuationData} = State, NewData) -> return_result(parse_multibulk(ContinuationData, NewData), State, multibulk_continue); parse(#pstate{state = status_continue, continuation_data = ContinuationData} = State, NewData) -> return_result(parse_simple(ContinuationData, NewData), State, status_continue). %% %% MULTIBULK %% parse_multibulk(Data) when is_binary(Data) -> parse_multibulk(buffer_create(Data)); parse_multibulk(Buffer) -> case get_newline_pos(Buffer) of undefined -> {continue, {incomplete_size, Buffer}}; NewlinePos -> OffsetNewlinePos = NewlinePos - 1, <<$*, Size:OffsetNewlinePos/binary, ?NL, Bulk/binary>> = buffer_to_binary(Buffer), IntSize = list_to_integer(binary_to_list(Size)), do_parse_multibulk(IntSize, buffer_create(Bulk)) end. %% Size of multibulk was incomplete, try again parse_multibulk({incomplete_size, Buffer}, NewData0) -> NewBuffer = buffer_append(Buffer, NewData0), parse_multibulk(NewBuffer); %% Ran out of data inside do_parse_multibulk in parse_bulk, must %% continue traversing the bulks parse_multibulk({in_parsing_bulks, Count, Buffer, Acc}, NewData0) -> NewBuffer = buffer_append(Buffer, NewData0), %% Continue where we left off do_parse_multibulk(Count, NewBuffer, Acc). %% @doc: Parses the given number of bulks from Data. If Data does not %% contain enough bulks, {continue, ContinuationData} is returned with %% enough information to start parsing with the correct count and %% accumulated data. do_parse_multibulk(Count, Buffer) -> do_parse_multibulk(Count, Buffer, []). do_parse_multibulk(-1, Buffer, []) -> {ok, undefined, buffer_to_binary(Buffer)}; do_parse_multibulk(0, Buffer, Acc) -> {ok, lists:reverse(Acc), buffer_to_binary(Buffer)}; do_parse_multibulk(Count, Buffer, Acc) -> case buffer_size(Buffer) == 0 of true -> {continue, {in_parsing_bulks, Count, buffer_create(), Acc}}; false -> %% Try parsing the first bulk in Data, if it works, we get the %% extra data back that was not part of the bulk which we can %% recurse on. If the bulk does not contain enough data, we %% return with a continuation and enough data to pick up where we %% left off. In the continuation we will get more data %% automagically in Data, so parsing the bulk might work. case parse_bulk(Buffer) of {ok, Value, Rest} -> do_parse_multibulk(Count - 1, buffer_create(Rest), [Value | Acc]); {continue, _} -> {continue, {in_parsing_bulks, Count, Buffer, Acc}} end end. %% %% BULK %% parse_bulk(Data) when is_binary(Data) -> parse_bulk(buffer_create(Data)); parse_bulk(Buffer) -> case buffer_hd(Buffer) of [$*] -> parse_multibulk(Buffer); [$+] -> parse_simple(buffer_tl(Buffer)); [$-] -> parse_simple(buffer_tl(Buffer)); [$:] -> parse_simple(buffer_tl(Buffer)); [$$] -> do_parse_bulk(Buffer) end. %% Bulk, at beginning of response do_parse_bulk(Buffer) -> %% Find the position of the first terminator, everything up until %% this point contains the size specifier. If we cannot find it, %% we received a partial response and need more data case get_newline_pos(Buffer) of undefined -> {continue, {incomplete_size, Buffer}}; NewlinePos -> OffsetNewlinePos = NewlinePos - 1, % Take into account the first $ <<$$, Size:OffsetNewlinePos/binary, Bulk/binary>> = buffer_to_binary(Buffer), IntSize = list_to_integer(binary_to_list(Size)), if %% Nil response from redis IntSize =:= -1 -> <> = Bulk, {ok, undefined, Rest}; %% We have enough data for the entire bulk size(Bulk) - (size(<>) * 2) >= IntSize -> <> = Bulk, {ok, Value, Rest}; true -> %% Need more data, so we send the bulk without the %% size specifier to our future self {continue, {IntSize, buffer_create(Bulk)}} end end. %% Bulk, continuation from partial bulk size parse_bulk({incomplete_size, Buffer}, NewData0) -> NewBuffer = buffer_append(Buffer, NewData0), parse_bulk(NewBuffer); %% Bulk, continuation from partial bulk value parse_bulk({IntSize, Buffer0}, Data) -> Buffer = buffer_append(Buffer0, Data), case buffer_size(Buffer) - (size(<>) * 2) >= IntSize of true -> <> = buffer_to_binary(Buffer), {ok, Value, Rest}; false -> {continue, {IntSize, Buffer}} end. %% %% SIMPLE REPLIES %% %% Handles replies on the following format: %% TData\r\n %% Where T is a type byte, like '+', '-', ':'. Data is terminated by \r\n %% @doc: Parse simple replies. Data must not contain type %% identifier. Type must be handled by the caller. parse_simple(Data) when is_binary(Data) -> parse_simple(buffer_create(Data)); parse_simple(Buffer) -> case get_newline_pos(Buffer) of undefined -> {continue, {incomplete_simple, Buffer}}; NewlinePos -> <> = buffer_to_binary(Buffer), {ok, Value, Rest} end. parse_simple({incomplete_simple, Buffer}, NewData0) -> NewBuffer = buffer_append(Buffer, NewData0), parse_simple(NewBuffer). %% %% INTERNAL HELPERS %% get_newline_pos({B, _}) -> case re:run(B, ?NL) of {match, [{Pos, _}]} -> Pos; nomatch -> undefined end. buffer_create() -> {[], 0}. buffer_create(Data) -> {[Data], byte_size(Data)}. buffer_append({List, Size}, Binary) -> NewList = case List of [] -> [Binary]; [Head | Tail] -> [Head, Tail, Binary] end, {NewList, Size + byte_size(Binary)}. buffer_hd({[<> | _], _}) -> [Char]; buffer_hd({[], _}) -> []. buffer_tl({[<<_, RestBin/binary>> | Rest], Size}) -> {[RestBin | Rest], Size - 1}. buffer_to_binary({List, _}) -> iolist_to_binary(List). buffer_size({_, Size}) -> Size. %% @doc: Helper for handling the result of parsing. Will update the %% parser state with the continuation of given name if necessary. return_result({ok, Value, <<>>}, _State, _StateName) -> {ok, Value, init()}; return_result({ok, Value, Rest}, _State, _StateName) -> {ok, Value, Rest, init()}; return_result({continue, ContinuationData}, State, StateName) -> {continue, State#pstate{state = StateName, continuation_data = ContinuationData}}. %% @doc: Helper for returning an error. Uses return_result/3 and just transforms the {ok, ...} tuple into an error tuple return_error(Result, State, StateName) -> case return_result(Result, State, StateName) of {ok, Value, ParserState} -> {error, Value, ParserState}; {ok, Value, Rest, ParserState} -> {error, Value, Rest, ParserState}; Res -> Res end. eredis-1.1.0/src/basho_bench_driver_erldis.erl0000644000232200023220000000130013145103330021760 0ustar debalancedebalance-module(basho_bench_driver_erldis). -export([new/1, run/4]). new(_Id) -> case erldis_client:connect() of {ok, Pid} -> {ok, Pid}; {error, {already_started, Pid}} -> {ok, Pid} end. run(get, KeyGen, _ValueGen, Client) -> Start = KeyGen(), case erldis:mget(Client, lists:seq(Start, Start + 500)) of {error, Reason} -> {error, Reason, Client}; _Value -> {ok, Client} end; run(put, KeyGen, ValueGen, Client) -> case erldis:set(Client, integer_to_list(KeyGen()), ValueGen()) of {error, Reason} -> {error, Reason, Client}; _Value -> {ok, Client} end. eredis-1.1.0/src/eredis.erl0000644000232200023220000001327613145103330016102 0ustar debalancedebalance%% %% Erlang Redis client %% %% Usage: %% {ok, Client} = eredis:start_link(). %% {ok, <<"OK">>} = eredis:q(Client, ["SET", "foo", "bar"]). %% {ok, <<"bar">>} = eredis:q(Client, ["GET", "foo"]). -module(eredis). -include("eredis.hrl"). %% Default timeout for calls to the client gen_server %% Specified in http://www.erlang.org/doc/man/gen_server.html#call-3 -define(TIMEOUT, 5000). -export([start_link/0, start_link/1, start_link/2, start_link/3, start_link/4, start_link/5, start_link/6, stop/1, q/2, q/3, qp/2, qp/3, q_noreply/2, q_async/2, q_async/3]). %% Exported for testing -export([create_multibulk/1]). %% Type of gen_server process id -type client() :: pid() | atom() | {atom(),atom()} | {global,term()} | {via,atom(),term()}. %% %% PUBLIC API %% start_link() -> start_link("127.0.0.1", 6379, 0, ""). start_link(Host, Port) -> start_link(Host, Port, 0, ""). start_link(Host, Port, Database) -> start_link(Host, Port, Database, ""). start_link(Host, Port, Database, Password) -> start_link(Host, Port, Database, Password, 100). start_link(Host, Port, Database, Password, ReconnectSleep) -> start_link(Host, Port, Database, Password, ReconnectSleep, ?TIMEOUT). start_link(Host, Port, Database, Password, ReconnectSleep, ConnectTimeout) when is_list(Host), is_integer(Port), is_integer(Database) orelse Database == undefined, is_list(Password), is_integer(ReconnectSleep) orelse ReconnectSleep =:= no_reconnect, is_integer(ConnectTimeout) -> eredis_client:start_link(Host, Port, Database, Password, ReconnectSleep, ConnectTimeout). %% @doc: Callback for starting from poolboy -spec start_link(server_args()) -> {ok, Pid::pid()} | {error, Reason::term()}. start_link(Args) -> Host = proplists:get_value(host, Args, "127.0.0.1"), Port = proplists:get_value(port, Args, 6379), Database = proplists:get_value(database, Args, 0), Password = proplists:get_value(password, Args, ""), ReconnectSleep = proplists:get_value(reconnect_sleep, Args, 100), ConnectTimeout = proplists:get_value(connect_timeout, Args, ?TIMEOUT), start_link(Host, Port, Database, Password, ReconnectSleep, ConnectTimeout). stop(Client) -> eredis_client:stop(Client). -spec q(Client::client(), Command::[any()]) -> {ok, return_value()} | {error, Reason::binary() | no_connection}. %% @doc: Executes the given command in the specified connection. The %% command must be a valid Redis command and may contain arbitrary %% data which will be converted to binaries. The returned values will %% always be binaries. q(Client, Command) -> call(Client, Command, ?TIMEOUT). q(Client, Command, Timeout) -> call(Client, Command, Timeout). -spec qp(Client::client(), Pipeline::pipeline()) -> [{ok, return_value()} | {error, Reason::binary()}] | {error, no_connection}. %% @doc: Executes the given pipeline (list of commands) in the %% specified connection. The commands must be valid Redis commands and %% may contain arbitrary data which will be converted to binaries. The %% values returned by each command in the pipeline are returned in a list. qp(Client, Pipeline) -> pipeline(Client, Pipeline, ?TIMEOUT). qp(Client, Pipeline, Timeout) -> pipeline(Client, Pipeline, Timeout). -spec q_noreply(Client::client(), Command::[any()]) -> ok. %% @doc Executes the command but does not wait for a response and ignores any errors. %% @see q/2 q_noreply(Client, Command) -> cast(Client, Command). -spec q_async(Client::client(), Command::[any()]) -> ok. % @doc Executes the command, and sends a message to this process with the response (with either error or success). Message is of the form `{response, Reply}', where `Reply' is the reply expected from `q/2'. q_async(Client, Command) -> q_async(Client, Command, self()). -spec q_async(Client::client(), Command::[any()], Pid::pid()|atom()) -> ok. %% @doc Executes the command, and sends a message to `Pid' with the response (with either or success). %% @see 1_async/2 q_async(Client, Command, Pid) when is_pid(Pid) -> Request = {request, create_multibulk(Command), Pid}, gen_server:cast(Client, Request). %% %% INTERNAL HELPERS %% call(Client, Command, Timeout) -> Request = {request, create_multibulk(Command)}, gen_server:call(Client, Request, Timeout). pipeline(_Client, [], _Timeout) -> []; pipeline(Client, Pipeline, Timeout) -> Request = {pipeline, [create_multibulk(Command) || Command <- Pipeline]}, gen_server:call(Client, Request, Timeout). cast(Client, Command) -> Request = {request, create_multibulk(Command)}, gen_server:cast(Client, Request). -spec create_multibulk(Args::[any()]) -> Command::iolist(). %% @doc: Creates a multibulk command with all the correct size headers create_multibulk(Args) -> ArgCount = [<<$*>>, integer_to_list(length(Args)), <>], ArgsBin = lists:map(fun to_bulk/1, lists:map(fun to_binary/1, Args)), [ArgCount, ArgsBin]. to_bulk(B) when is_binary(B) -> [<<$$>>, integer_to_list(iolist_size(B)), <>, B, <>]. %% @doc: Convert given value to binary. Fallbacks to %% term_to_binary/1. For floats, throws {cannot_store_floats, Float} %% as we do not want floats to be stored in Redis. Your future self %% will thank you for this. to_binary(X) when is_list(X) -> list_to_binary(X); to_binary(X) when is_atom(X) -> atom_to_binary(X, utf8); to_binary(X) when is_binary(X) -> X; to_binary(X) when is_integer(X) -> integer_to_binary(X); to_binary(X) when is_float(X) -> throw({cannot_store_floats, X}); to_binary(X) -> term_to_binary(X). eredis-1.1.0/src/basho_bench_driver_eredis.erl0000644000232200023220000001234113145103330021760 0ustar debalancedebalance-module(basho_bench_driver_eredis). -export([new/1, run/4]). -export([value_gen/1]). new(_Id) -> case whereis(eredis_driver) of undefined -> case eredis:start_link() of {ok, Client} -> register(eredis_driver, Client), {ok, Client}; {error, Reason} -> {error, Reason} end; Pid -> {ok, Pid} end. run(get, KeyGen, _ValueGen, Client) -> Start = KeyGen(), %%case eredis:q(["MGET" | lists:seq(Start, Start + 500)]) of case catch(eredis:q(Client, ["GET", Start], 100)) of {ok, _Value} -> {ok, Client}; {error, Reason} -> {error, Reason, Client}; {'EXIT', {timeout, _}} -> {error, timeout, Client} end; run(pipeline_get, KeyGen, _ValueGen, Client) -> Seq = lists:seq(1, 5), P = [["GET", KeyGen()] || _ <- Seq], case catch(eredis:qp(Client, P, 500)) of {error, Reason} -> {error, Reason, Client}; {'EXIT', {timeout, _}} -> {error, timeout, Client}; Res -> case check_pipeline_get(Res, Seq) of ok -> {ok, Client}; {error, Reason} -> {error, Reason, Client} end end; run(put, KeyGen, ValueGen, Client) -> case catch(eredis:q(Client, ["SET", KeyGen(), ValueGen()], 100)) of {ok, <<"OK">>} -> {ok, Client}; {error, Reason} -> {error, Reason, Client}; {'EXIT', {timeout, _}} -> {error, timeout, Client} end; run(pipeline_put, KeyGen, ValueGen, Client) -> Seq = lists:seq(1, 5), P = [["SET", KeyGen(), ValueGen()] || _ <- Seq], R = [{ok, <<"OK">>} || _ <- Seq], case catch(eredis:qp(Client, P, 500)) of R -> {ok, Client}; {error, Reason} -> {error, Reason, Client}; {'EXIT', {timeout, _}} -> {error, timeout, Client} end. check_pipeline_get([], []) -> ok; check_pipeline_get([{ok, _}|Res], [_|Seq]) -> check_pipeline_get(Res, Seq); check_pipeline_get([{error, Reason}], _) -> {error, Reason}. value_gen(_Id) -> fun() -> %% %% Example data from http://json.org/example.html <<"{\"web-app\":{\"servlet\":[{\"servlet-name\":\"cofaxCDS\",\"servlet-class\":\"org.cofax.cds.CDSServlet\",\"init-param\":{\"configGlossary:installationAt\":\"Philadelphia,PA\",\"configGlossary:adminEmail\":\"ksm@pobox.com\",\"configGlossary:poweredBy\":\"Cofax\",\"configGlossary:poweredByIcon\":\"/images/cofax.gif\",\"configGlossary:staticPath\":\"/content/static\",\"templateProcessorClass\":\"org.cofax.WysiwygTemplate\",\"templateLoaderClass\":\"org.cofax.FilesTemplateLoader\",\"templatePath\":\"templates\",\"templateOverridePath\":\"\",\"defaultListTemplate\":\"listTemplate.htm\",\"defaultFileTemplate\":\"articleTemplate.htm\",\"useJSP\":false,\"jspListTemplate\":\"listTemplate.jsp\",\"jspFileTemplate\":\"articleTemplate.jsp\",\"cachePackageTagsTrack\":200,\"cachePackageTagsStore\":200,\"cachePackageTagsRefresh\":60,\"cacheTemplatesTrack\":100,\"cacheTemplatesStore\":50,\"cacheTemplatesRefresh\":15,\"cachePagesTrack\":200,\"cachePagesStore\":100,\"cachePagesRefresh\":10,\"cachePagesDirtyRead\":10,\"searchEngineListTemplate\":\"forSearchEnginesList.htm\",\"searchEngineFileTemplate\":\"forSearchEngines.htm\",\"searchEngineRobotsDb\":\"WEB-INF/robots.db\",\"useDataStore\":true,\"dataStoreClass\":\"org.cofax.SqlDataStore\",\"redirectionClass\":\"org.cofax.SqlRedirection\",\"dataStoreName\":\"cofax\",\"dataStoreDriver\":\"com.microsoft.jdbc.sqlserver.SQLServerDriver\",\"dataStoreUrl\":\"jdbc:microsoft:sqlserver://LOCALHOST:1433;DatabaseName=goon\",\"dataStoreUser\":\"sa\",\"dataStorePassword\":\"dataStoreTestQuery\",\"dataStoreTestQuery\":\"SETNOCOUNTON;selecttest='test';\",\"dataStoreLogFile\":\"/usr/local/tomcat/logs/datastore.log\",\"dataStoreInitConns\":10,\"dataStoreMaxConns\":100,\"dataStoreConnUsageLimit\":100,\"dataStoreLogLevel\":\"debug\",\"maxUrlLength\":500}},{\"servlet-name\":\"cofaxEmail\",\"servlet-class\":\"org.cofax.cds.EmailServlet\",\"init-param\":{\"mailHost\":\"mail1\",\"mailHostOverride\":\"mail2\"}},{\"servlet-name\":\"cofaxAdmin\",\"servlet-class\":\"org.cofax.cds.AdminServlet\"},{\"servlet-name\":\"fileServlet\",\"servlet-class\":\"org.cofax.cds.FileServlet\"},{\"servlet-name\":\"cofaxTools\",\"servlet-class\":\"org.cofax.cms.CofaxToolsServlet\",\"init-param\":{\"templatePath\":\"toolstemplates/\",\"log\":1,\"logLocation\":\"/usr/local/tomcat/logs/CofaxTools.log\",\"logMaxSize\":\"\",\"dataLog\":1,\"dataLogLocation\":\"/usr/local/tomcat/logs/dataLog.log\",\"dataLogMaxSize\":\"\",\"removePageCache\":\"/content/admin/remove?cache=pages&id=\",\"removeTemplateCache\":\"/content/admin/remove?cache=templates&id=\",\"fileTransferFolder\":\"/usr/local/tomcat/webapps/content/fileTransferFolder\",\"lookInContext\":1,\"adminGroupID\":4,\"betaServer\":true}}],\"servlet-mapping\":{\"cofaxCDS\":\"/\",\"cofaxEmail\":\"/cofaxutil/aemail/*\",\"cofaxAdmin\":\"/admin/*\",\"fileServlet\":\"/static/*\",\"cofaxTools\":\"/tools/*\"},\"taglib\":{\"taglib-uri\":\"cofax.tld\",\"taglib-location\":\"/WEB-INF/tlds/cofax.tld\"}}">> end. eredis-1.1.0/src/eredis_sub.erl0000644000232200023220000001462513145103330016752 0ustar debalancedebalance%% %% Erlang PubSub Redis client %% -module(eredis_sub). -include("eredis.hrl"). %% Default timeout for calls to the client gen_server %% Specified in http://www.erlang.org/doc/man/gen_server.html#call-3 -define(TIMEOUT, 5000). -export([start_link/0, start_link/1, start_link/3, start_link/6, stop/1, controlling_process/1, controlling_process/2, controlling_process/3, ack_message/1, subscribe/2, unsubscribe/2, channels/1]). -export([psubscribe/2,punsubscribe/2]). -export([receiver/1, sub_example/0, pub_example/0]). -export([psub_example/0,ppub_example/0]). %% %% PUBLIC API %% start_link() -> start_link([]). start_link(Host, Port, Password) -> start_link(Host, Port, Password, 100, infinity, drop). start_link(Host, Port, Password, ReconnectSleep, MaxQueueSize, QueueBehaviour) when is_list(Host) andalso is_integer(Port) andalso is_list(Password) andalso (is_integer(ReconnectSleep) orelse ReconnectSleep =:= no_reconnect) andalso (is_integer(MaxQueueSize) orelse MaxQueueSize =:= infinity) andalso (QueueBehaviour =:= drop orelse QueueBehaviour =:= exit) -> eredis_sub_client:start_link(Host, Port, Password, ReconnectSleep, MaxQueueSize, QueueBehaviour). %% @doc: Callback for starting from poolboy -spec start_link(server_args()) -> {ok, Pid::pid()} | {error, Reason::term()}. start_link(Args) -> Host = proplists:get_value(host, Args, "127.0.0.1"), Port = proplists:get_value(port, Args, 6379), Password = proplists:get_value(password, Args, ""), ReconnectSleep = proplists:get_value(reconnect_sleep, Args, 100), MaxQueueSize = proplists:get_value(max_queue_size, Args, infinity), QueueBehaviour = proplists:get_value(queue_behaviour, Args, drop), start_link(Host, Port, Password, ReconnectSleep, MaxQueueSize, QueueBehaviour). stop(Pid) -> eredis_sub_client:stop(Pid). -spec controlling_process(Client::pid()) -> ok. %% @doc: Make the calling process the controlling process. The %% controlling process received pubsub-related messages, of which %% there are three kinds. In each message, the pid refers to the %% eredis client process. %% %% {message, Channel::binary(), Message::binary(), pid()} %% This is sent for each pubsub message received by the client. %% %% {pmessage, Pattern::binary(), Channel::binary(), Message::binary(), pid()} %% This is sent for each pattern pubsub message received by the client. %% %% {dropped, NumMessages::integer(), pid()} %% If the queue reaches the max size as specified in start_link %% and the behaviour is to drop messages, this message is sent when %% the queue is flushed. %% %% {subscribed, Channel::binary(), pid()} %% When using eredis_sub:subscribe(pid()), this message will be %% sent for each channel Redis aknowledges the subscription. The %% opposite, 'unsubscribed' is sent when Redis aknowledges removal %% of a subscription. %% %% {eredis_disconnected, pid()} %% This is sent when the eredis client is disconnected from redis. %% %% {eredis_connected, pid()} %% This is sent when the eredis client reconnects to redis after %% an existing connection was disconnected. %% %% Any message of the form {message, _, _, _} must be acknowledged %% before any subsequent message of the same form is sent. This %% prevents the controlling process from being overrun with redis %% pubsub messages. See ack_message/1. controlling_process(Client) -> controlling_process(Client, self()). -spec controlling_process(Client::pid(), Pid::pid()) -> ok. %% @doc: Make the given process (pid) the controlling process. controlling_process(Client, Pid) -> controlling_process(Client, Pid, ?TIMEOUT). %% @doc: Make the given process (pid) the controlling process subscriber %% with the given Timeout. controlling_process(Client, Pid, Timeout) -> gen_server:call(Client, {controlling_process, Pid}, Timeout). -spec ack_message(Client::pid()) -> ok. %% @doc: acknowledge the receipt of a pubsub message. each pubsub %% message must be acknowledged before the next one is received ack_message(Client) -> gen_server:cast(Client, {ack_message, self()}). %% @doc: Subscribe to the given channels. Returns immediately. The %% result will be delivered to the controlling process as any other %% message. Delivers {subscribed, Channel::binary(), pid()} -spec subscribe(pid(), [channel()]) -> ok. subscribe(Client, Channels) -> gen_server:cast(Client, {subscribe, self(), Channels}). %% @doc: Pattern subscribe to the given channels. Returns immediately. The %% result will be delivered to the controlling process as any other %% message. Delivers {subscribed, Channel::binary(), pid()} -spec psubscribe(pid(), [channel()]) -> ok. psubscribe(Client, Channels) -> gen_server:cast(Client, {psubscribe, self(), Channels}). unsubscribe(Client, Channels) -> gen_server:cast(Client, {unsubscribe, self(), Channels}). punsubscribe(Client, Channels) -> gen_server:cast(Client, {punsubscribe, self(), Channels}). %% @doc: Returns the channels the given client is currently %% subscribing to. Note: this list is based on the channels at startup %% and any channel added during runtime. It might not immediately %% reflect the channels Redis thinks the client is subscribed to. channels(Client) -> gen_server:call(Client, get_channels). %% %% STUFF FOR TRYING OUT PUBSUB %% receiver(Sub) -> receive Msg -> io:format("received ~p~n", [Msg]), ack_message(Sub), ?MODULE:receiver(Sub) end. sub_example() -> {ok, Sub} = start_link(), Receiver = spawn_link(fun () -> controlling_process(Sub), subscribe(Sub, [<<"foo">>]), receiver(Sub) end), {Sub, Receiver}. psub_example() -> {ok, Sub} = start_link(), Receiver = spawn_link(fun () -> controlling_process(Sub), psubscribe(Sub, [<<"foo*">>]), receiver(Sub) end), {Sub, Receiver}. pub_example() -> {ok, P} = eredis:start_link(), eredis:q(P, ["PUBLISH", "foo", "bar"]), eredis_client:stop(P). ppub_example() -> {ok, P} = eredis:start_link(), eredis:q(P, ["PUBLISH", "foo123", "bar"]), eredis_client:stop(P). eredis-1.1.0/.travis.yml0000644000232200023220000000037413145103330015440 0ustar debalancedebalancelanguage: erlang notifications: email: false otp_release: - 20.0 - 19.3 - 19.2 - 19.1 - 19.0 - 18.3 - 18.2.1 - 18.1 - 17.5 - 17.4 - 17.3 - 17.1 - 17.0 - R16B03-1 - R16B03 - R16B02 - R16B01 services: - redis-server eredis-1.1.0/rebar.config0000644000232200023220000000043513145103330015607 0ustar debalancedebalance{erl_opts, [ {platform_define, "^[0-9]+", namespaced_types} ]}. {cover_enabled, true}. %% basho_bench_driver_erldis calls undefined functions, so disable xref_checks. %% This allows this project to be used as a dependency by other rebar projects %% that use xref. {xref_checks, []}. eredis-1.1.0/AUTHORS0000644000232200023220000000026113145103330014372 0ustar debalancedebalanceKnut Nesheim tomlion Metin Akat Ville Tuulos adzeitor Valentino Volonghi Dave Peticolas Ransom Richardson Michael Gregson Matthew Conway Aleksey Morarash Mikl Kurkov Seth Falconeredis-1.1.0/Emakefile.src0000644000232200023220000000041513145103330015716 0ustar debalancedebalance{['src/eredis*'], [{outdir, "ebin"}, {i, "include"}, {{EXTRA_OPTS}} debug_info, warn_unused_function, warn_bif_clash, warn_deprecated_function, warn_obsolete_guard, warn_shadow_vars, warn_export_vars, warn_unused_records, warn_unused_import ]}. eredis-1.1.0/include/0000755000232200023220000000000013145103330014746 5ustar debalancedebalanceeredis-1.1.0/include/eredis_sub.hrl0000644000232200023220000000206713145103330017606 0ustar debalancedebalance%% State in eredis_sub_client -record(state, { host :: string() | undefined, port :: integer() | undefined, password :: binary() | undefined, reconnect_sleep :: integer() | undefined | no_reconnect, socket :: port() | undefined, parser_state :: #pstate{} | undefined, %% Channels we should subscribe to channels = [] :: [channel()], % The process we send pubsub and connection state messages to. controlling_process :: undefined | {reference(), pid()}, % This is the queue of messages to send to the controlling % process. msg_queue :: eredis_queue(), %% When the queue reaches this size, either drop all %% messages or exit. max_queue_size :: integer() | inifinity, queue_behaviour :: drop | exit, % The msg_state keeps track of whether we are waiting % for the controlling process to acknowledge the last % message. msg_state = need_ack :: ready | need_ack }). eredis-1.1.0/include/eredis.hrl0000644000232200023220000000236413145103330016735 0ustar debalancedebalance%% Public types -type reconnect_sleep() :: no_reconnect | integer(). -type option() :: {host, string()} | {port, integer()} | {database, string()} | {password, string()} | {reconnect_sleep, reconnect_sleep()}. -type server_args() :: [option()]. -type return_value() :: undefined | binary() | [binary() | nonempty_list()]. -type pipeline() :: [iolist()]. -type channel() :: binary(). %% Continuation data is whatever data returned by any of the parse %% functions. This is used to continue where we left off the next time %% the user calls parse/2. -type continuation_data() :: any(). -type parser_state() :: status_continue | bulk_continue | multibulk_continue. %% Internal types -ifdef(namespaced_types). -type eredis_queue() :: queue:queue(). -else. -type eredis_queue() :: queue(). -endif. %% Internal parser state. Is returned from parse/2 and must be %% included on the next calls to parse/2. -record(pstate, { state = undefined :: parser_state() | undefined, continuation_data :: continuation_data() | undefined }). -define(NL, "\r\n"). -define(SOCKET_OPTS, [binary, {active, once}, {packet, raw}, {reuseaddr, false}, {send_timeout, ?SEND_TIMEOUT}]). -define(RECV_TIMEOUT, 5000). -define(SEND_TIMEOUT, 5000). eredis-1.1.0/mix.exs0000644000232200023220000000061113145103330014637 0ustar debalancedebalancedefmodule Eredis.Mixfile do use Mix.Project def project do [ app: :eredis, version: "1.1.0", elixir: "~> 1.5.1", start_permanent: Mix.env == :prod, deps: deps() ] end # Run "mix help compile.app" to learn about applications. def application do [] end # Run "mix help deps" to learn about dependencies. defp deps do [] end end eredis-1.1.0/test/0000755000232200023220000000000013145103330014302 5ustar debalancedebalanceeredis-1.1.0/test/eredis_parser_tests.erl0000644000232200023220000004644013145103330021067 0ustar debalancedebalance%% %% Parser tests. In particular tests for partial responses. This would %% probably be a very good candidate for testing with quickcheck or %% properl. %% -module(eredis_parser_tests). -include("eredis.hrl"). -include_lib("eunit/include/eunit.hrl"). -import(eredis_parser, [parse/2, init/0, parse_bulk/1, parse_bulk/2, parse_multibulk/1, parse_multibulk/2, buffer_create/0, buffer_create/1]). % parse a binary one byte at a time one_byte_parse(B) -> one_byte_parse(init(), B). one_byte_parse(S, <<>>) -> parse(S, <<>>); one_byte_parse(S, <>) -> parse(S, <>); one_byte_parse(S, <>) -> case parse(S, <>) of {continue, NewState} -> one_byte_parse(NewState, B); {ok, Value, Rest, NewState} -> {ok, Value, <>, NewState}; {error, Err, Rest, NewState} -> {error, Err, <>, NewState}; Other -> Other end. parse_bulk_test() -> B = <<"$3\r\nbar\r\n">>, ?assertEqual({ok, <<"bar">>, #pstate{}}, parse(#pstate{}, B)). parse_split_bulk_test() -> State1 = init(), B1 = <<"$3\r\n">>, B2 = <<"bar\r\n">>, {continue, State2} = parse(State1, B1), Buffer = buffer_create(<<"\r\n">>), ?assertEqual(#pstate{state = bulk_continue, continuation_data = {3, Buffer}}, State2), ?assertMatch({ok, <<"bar">>, _}, parse(State2, B2)). parse_very_split_bulk_test() -> State1 = init(), B1 = <<"$1">>, B2 = <<"3\r\n">>, B3 = <<"foobarbazquux\r\n">>, %% 13 bytes Buffer1 = buffer_create(<<"$1">>), ?assertEqual({continue, #pstate{state = bulk_continue, continuation_data = {incomplete_size, Buffer1}}}, parse(State1, B1)), {continue, State2} = parse(State1, B1), Buffer2 = buffer_create(<<"\r\n">>), ?assertEqual({continue, #pstate{state = bulk_continue, continuation_data = {13, Buffer2}}}, parse(State2, B2)), {continue, State3} = parse(State2, B2), ?assertMatch({ok, <<"foobarbazquux">>, _}, parse(State3, B3)). too_much_data_test() -> B = <<"$1\r\n1\r\n$1\r\n2\r\n$1\r\n3\r\n">>, ?assertEqual({ok, <<"1">>, <<"$1\r\n2\r\n$1\r\n3\r\n">>}, parse_bulk(B)). too_much_data_in_continuation_test() -> B1 = <<"$1\r\n">>, B2 = <<"1\r\n$1\r\n2\r\n$1\r\n3\r\n">>, Buffer = buffer_create(<<"\r\n">>), ?assertEqual({continue, {1, Buffer}}, parse_bulk(B1)), {continue, ContinuationData1} = parse_bulk(B1), ?assertEqual({ok, <<"1">>, <<"$1\r\n2\r\n$1\r\n3\r\n">>}, parse_bulk(ContinuationData1, B2)). bulk_test_() -> B = <<"$3\r\nbar\r\n">>, ?_assertEqual({ok, <<"bar">>, <<>>}, parse_bulk(B)). bulk_split_test() -> B1 = <<"$3\r\n">>, B2 = <<"bar\r\n">>, Buffer = buffer_create(<<"\r\n">>), ?assertEqual({continue, {3, Buffer}}, parse_bulk(B1)), {continue, Res} = parse_bulk(B1), ?assertEqual({ok, <<"bar">>, <<>>}, parse_bulk(Res, B2)). bulk_very_split_test() -> B1 = <<"$1">>, B2 = <<"3\r\n">>, B3 = <<"foobarbazquux\r\n">>, %% 13 bytes Buffer1 = buffer_create(<<"$1">>), ?assertEqual({continue, {incomplete_size, Buffer1}}, parse_bulk(B1)), {continue, ContinuationData1} = parse_bulk(B1), Buffer2 = buffer_create(<<"\r\n">>), ?assertEqual({continue, {13, Buffer2}}, parse_bulk(ContinuationData1, B2)), {continue, ContinuationData2} = parse_bulk(ContinuationData1, B2), ?assertEqual({ok, <<"foobarbazquux">>, <<>>}, parse_bulk(ContinuationData2, B3)). bulk_split_on_newline_test() -> B1 = <<"$13\r\nfoobarbazquux">>, B2 = <<"\r\n">>, %% 13 bytes Buffer = buffer_create(<<"\r\nfoobarbazquux">>), ?assertEqual({continue, {13, Buffer}}, parse_bulk(B1)), {continue, ContinuationData1} = parse_bulk(B1), ?assertEqual({ok, <<"foobarbazquux">>, <<>>}, parse_bulk(ContinuationData1, B2)). bulk_nil_test() -> B = <<"$-1\r\n">>, ?assertEqual({ok, undefined, init()}, parse(init(), B)). bulk_nil_chunked_test() -> State1 = init(), B1 = <<"$-1">>, B2 = <<"\r\n">>, Buffer = buffer_create(<<"$-1">>), ?assertEqual({continue, #pstate{state = bulk_continue, continuation_data = {incomplete_size,Buffer}}}, parse(State1, B1)), {continue, State2} = parse(State1, B1), ?assertEqual({ok, undefined, init()}, parse(State2, B2)). bulk_nil_with_extra_test() -> B = <<"$-1\r\n$3\r\nfoo\r\n">>, ?assertEqual({ok, undefined, <<"$3\r\nfoo\r\n">>, init()}, parse(init(), B)). bulk_crap_test() -> B = <<"\r\n">>, ?assertEqual({error, unknown_response}, parse(init(), B)). multibulk_test() -> %% [{1, 1}, {2, 2}, {3, 3}] B = <<"*3\r\n$1\r\n1\r\n$1\r\n2\r\n$1\r\n3\r\n">>, ?assertEqual({ok, [<<"1">>, <<"2">>, <<"3">>], <<>>}, parse_multibulk(B)). multibulk_parse_test() -> %% [{1, 1}, {2, 2}, {3, 3}] B = <<"*3\r\n$1\r\n1\r\n$1\r\n2\r\n$1\r\n3\r\n">>, ?assertEqual({ok, [<<"1">>, <<"2">>, <<"3">>], #pstate{}}, parse(init(), B)). multibulk_one_byte_parse_test() -> %% [{1, 1}, {2, 2}, {3, 3}] B = <<"*3\r\n$1\r\n1\r\n$1\r\n2\r\n$1\r\n3\r\n">>, ?assertEqual({ok, [<<"1">>, <<"2">>, <<"3">>], #pstate{}}, one_byte_parse(B)). nested_multibulk_test() -> %% [[1, 2], [3, 4]] B = <<"*2\r\n*2\r\n$1\r\n1\r\n$1\r\n2\r\n*2\r\n$1\r\n3\r\n$1\r\n4\r\n">>, ?assertEqual({ok, [[<<"1">>, <<"2">>], [<<"3">>, <<"4">>]], <<>>}, parse_multibulk(B)). nested_multibulk_parse_test() -> %% [[1, 2], [3, 4]] B = <<"*2\r\n*2\r\n$1\r\n1\r\n$1\r\n2\r\n*2\r\n$1\r\n3\r\n$1\r\n4\r\n">>, ?assertEqual({ok, [[<<"1">>, <<"2">>], [<<"3">>, <<"4">>]], #pstate{}}, parse(init(), B)). nested_multibulk_one_byte_parse_test() -> %% [[1, 2], [3, 4]] B = <<"*2\r\n*2\r\n$1\r\n1\r\n$1\r\n2\r\n*2\r\n$1\r\n3\r\n$1\r\n4\r\n">>, ?assertEqual({ok, [[<<"1">>, <<"2">>], [<<"3">>, <<"4">>]], #pstate{}}, one_byte_parse(B)). multibulk_split_parse_test() -> %% [{1, 1}, {2, 2}, {3, 3}] B1 = <<"*3\r\n$1\r\n1\r\n$1">>, B2 = <<"\r\n2\r\n$1\r\n3\r\n">>, State1 = init(), Buffer = buffer_create(<<"$1">>), ?assertEqual({continue, #pstate{state = multibulk_continue, continuation_data = {in_parsing_bulks,2,Buffer,[<<"1">>]}}}, parse(State1, B1)), {continue, State2} = parse(State1, B1), ?assertMatch({ok, [<<"1">>, <<"2">>, <<"3">>], _}, parse(State2, B2)). multibulk_split_test() -> %% Split into 2 parts: <<"*3\r\n$1\r\n1\r\n$1\r\n2\r\n$1\r\n3\r\n">> B1 = <<"*3\r\n$1\r\n1\r\n$1">>, B2 = <<"\r\n2\r\n$1\r\n3\r\n">>, {continue, ContinuationData1} = parse_multibulk(B1), Result = parse_multibulk(ContinuationData1, B2), ?assertEqual({ok, [<<"1">>, <<"2">>, <<"3">>], <<>>}, Result). multibulk_very_split_test() -> %% Split into 4 parts: <<"*3\r\n$1\r\n1\r\n$1\r\n2\r\n$1\r\n3\r\n">> B1 = <<"*">>, B2 = <<"3\r\n$1\r">>, B3 = <<"\n1\r\n$1\r\n2\r\n$1">>, B4 = <<"\r\n3\r\n">>, Buffer = buffer_create(<<"*">>), ?assertEqual({continue, {incomplete_size, Buffer}}, parse_multibulk(B1)), {continue, ContinuationData1} = parse_multibulk(B1), {continue, ContinuationData2} = parse_multibulk(ContinuationData1, B2), {continue, ContinuationData3} = parse_multibulk(ContinuationData2, B3), Result = parse_multibulk(ContinuationData3, B4), ?assertEqual({ok, [<<"1">>, <<"2">>, <<"3">>], <<>>}, Result). multibulk_newline_split_test() -> %% Split into 4 parts: <<"*3\r\n$1\r\n1\r\n$1\r\n2\r\n$1\r\n3\r\n">> B1 = <<"*2\r\n$1\r\n1">>, B2 = <<"\r\n$1\r\n2\r\n">>, Buffer = buffer_create(<<"$1\r\n1">>), ?assertEqual({continue, {in_parsing_bulks, 2, Buffer, []}}, parse_multibulk(B1)), {continue, ContinuationData1} = parse_multibulk(B1), ?assertEqual({ok, [<<"1">>, <<"2">>], <<>>}, parse_multibulk(ContinuationData1, B2)). multibulk_nil_test() -> B = <<"*-1\r\n">>, ?assertEqual({ok, undefined, <<>>}, parse_multibulk(B)). multibulk_nil_parse_test() -> B = <<"*-1\r\n">>, ?assertEqual({ok, undefined, #pstate{}}, parse(init(), B)). big_chunks_test() -> %% Real-world example, MGET 1..200 B1 = <<"*200\r\n$1\r\n1\r\n$1\r\n2\r\n$1\r\n3\r\n$1\r\n4\r\n$1\r\n5\r\n$1\r\n6\r\n$1\r\n7\r\n$1\r\n8\r\n$1\r\n9\r\n$2\r\n10\r\n$2\r\n11\r\n$2\r\n12\r\n$2\r\n13\r\n$2\r\n14\r\n$2\r\n15\r\n$2\r\n16\r\n$2\r\n17\r\n$2\r\n18\r\n$2\r\n19\r\n$2\r\n20\r\n$2\r\n21\r\n$2\r\n22\r\n$2\r\n23\r\n$2\r\n24\r\n$2\r\n25\r\n$2\r\n26\r\n$2\r\n27\r\n$2\r\n28\r\n$2\r\n29\r\n$2\r\n30\r\n$2\r\n31\r\n$2\r\n32\r\n$2\r\n33\r\n$2\r\n34\r\n$2\r\n35\r\n$2\r\n36\r\n$2\r\n37\r\n$2\r\n38\r\n$2\r\n39\r\n$2\r\n40\r\n$2\r\n41\r\n$2\r\n42\r\n$2\r\n43\r\n$2\r\n44\r\n$2\r\n45\r\n$2\r\n46\r\n$2\r\n47\r\n$2\r\n48\r\n$2\r\n49\r\n$2\r\n50\r\n$2\r\n51\r\n$2\r\n52\r\n$2\r\n53\r\n$2\r\n54\r\n$2\r\n55\r\n$2\r\n56\r\n$2\r\n57\r\n$2\r\n58\r\n$2\r\n59\r\n$2\r\n60\r\n$2\r\n61\r\n$2\r\n62\r\n$2\r\n63\r\n$2\r\n64\r\n$2\r\n65\r\n$2\r\n66\r\n$2\r\n67\r\n$2\r\n68\r\n$2\r\n69\r\n$2\r\n70\r\n$2\r\n71\r\n$2\r\n72\r\n$2\r\n73\r\n$2\r\n74\r\n$2\r\n75\r\n$2\r\n76\r\n$2\r\n77\r\n$2\r\n78\r\n$2\r\n79\r\n$2\r\n80\r\n$2\r\n81\r\n$2\r\n82\r\n$2\r\n83\r\n$2\r\n84\r\n$2\r\n85\r\n$2\r\n86\r\n$2\r\n87\r\n$2\r\n88\r\n$2\r\n89\r\n$2\r\n90\r\n$2\r\n91\r\n$2\r\n92\r\n$2\r\n93\r\n$2\r\n94\r\n$2\r\n95\r\n$2\r\n96\r\n$2\r\n97\r\n$2\r\n98\r\n$2\r\n99\r\n$3\r\n100\r\n$3\r\n101\r\n$3\r\n102\r\n$3\r\n103\r\n$3\r\n104\r\n$3\r\n105\r\n$3\r\n106\r\n$3\r\n107\r\n$3\r\n108\r\n$3\r\n109\r\n$3\r\n110\r\n$3\r\n111\r\n$3\r\n112\r\n$3\r\n113\r\n$3\r\n114\r\n$3\r\n115\r\n$3\r\n116\r\n$3\r\n117\r\n$3\r\n118\r\n$3\r\n119\r\n$3\r\n120\r\n$3\r\n121\r\n$3\r\n122\r\n$3\r\n123\r\n$3\r\n124\r\n$3\r\n125\r\n$3\r\n126\r\n$3\r\n127\r\n$3\r\n128\r\n$3\r\n129\r\n$3\r\n130\r\n$3\r\n131\r\n$3\r\n132\r\n$3\r\n133\r\n$3\r\n134\r\n$3\r\n135\r\n$3\r\n136\r\n$3\r\n137\r\n$3\r\n138\r\n$3\r\n139\r\n$3\r\n140\r\n$3\r\n141\r\n$3\r\n142\r\n$3\r\n143\r\n$3\r\n144\r\n$3\r\n145\r\n$3\r\n146\r\n$3\r\n147\r\n$3\r\n148\r\n$3\r\n149\r\n$3\r\n150\r\n$3\r\n151\r\n$3\r\n152\r\n$3\r\n153\r\n$3\r\n154\r\n$3\r\n155\r\n$3\r\n156\r\n$3\r\n157\r\n$3\r\n158\r\n$3\r\n159\r\n$3\r\n160\r\n$3\r\n161\r\n$3\r\n162\r\n$3\r\n163\r\n$3\r\n164\r\n$3\r\n165\r\n$3\r\n166\r\n$3\r\n167\r\n$3\r\n168\r\n$3\r\n169\r\n$3\r\n170\r\n$3\r\n171\r\n$3\r\n172\r\n$3\r\n173\r\n$3\r\n1">>, B2 = <<"74\r\n$3\r\n175\r\n$3\r\n176\r\n$3\r\n177\r\n$3\r\n178\r\n$3\r\n179\r\n$3\r\n180\r\n$3\r\n181\r\n$3\r\n182\r\n$3\r\n183\r\n$3\r\n184\r\n$3\r\n185\r\n$3\r\n186\r\n$3\r\n187\r\n$3\r\n188\r\n$3\r\n189\r\n$3\r\n190\r\n$3\r\n191\r\n$3\r\n192\r\n$3\r\n193\r\n$3\r\n194\r\n$3\r\n195\r\n$3\r\n196\r\n$3\r\n197\r\n$3\r\n198\r\n$3\r\n199\r\n$3\r\n200\r\n">>, ExpectedValues = [list_to_binary(integer_to_list(N)) || N <- lists:seq(1, 200)], State1 = init(), ?assertMatch({continue, #pstate{state = multibulk_continue, continuation_data = {in_parsing_bulks, 27, _, _}}}, parse(State1, B1)), {continue, State2} = parse(State1, B1), ?assertMatch({ok, ExpectedValues, #pstate{state = undefined, continuation_data = undefined}}, parse(State2, B2)). chunk_test() -> B1 = <<"*500\r\n$1\r\n1\r\n$1\r\n2\r\n$1\r\n3\r\n$1\r\n4\r\n$1\r\n5\r\n$1\r\n6\r\n$1\r\n7\r\n$1\r\n8\r\n$1\r\n9\r\n$2\r\n10\r\n$2\r\n11\r\n$2\r\n12\r\n$2\r\n13\r\n$2\r\n14\r\n$2\r\n15\r\n$2\r\n16\r\n$2\r\n17\r\n$2\r\n18\r\n$2\r\n19\r\n$2\r\n20\r\n$2\r\n21\r\n$2\r\n22\r\n$2\r\n23\r\n$2\r\n24\r\n$2\r\n25\r\n$2\r\n26\r\n$2\r\n27\r\n$2\r\n28\r\n$2\r\n29\r\n$2\r\n30\r\n$2\r\n31\r\n$2\r\n32\r\n$2\r\n33\r\n$2\r\n34\r\n$2\r\n35\r\n$2\r\n36\r\n$2\r\n37\r\n$2\r\n38\r\n$2\r\n39\r\n$2\r\n40\r\n$2\r\n41\r\n$2\r\n42\r\n$2\r\n43\r\n$2\r\n44\r\n$2\r\n45\r\n$2\r\n46\r\n$2\r\n47\r\n$2\r\n48\r\n$2\r\n49\r\n$2\r\n50\r\n$2\r\n51\r\n$2\r\n52\r\n$2\r\n53\r\n$2\r\n54\r\n$2\r\n55\r\n$2\r\n56\r\n$2\r\n57\r\n$2\r\n58\r\n$2\r\n59\r\n$2\r\n60\r\n$2\r\n61\r\n$2\r\n62\r\n$2\r\n63\r\n$2\r\n64\r\n$2\r\n65\r\n$2\r\n66\r\n$2\r\n67\r\n$2\r\n68\r\n$2\r\n69\r\n$2\r\n70\r\n$2\r\n71\r\n$2\r\n72\r\n$2\r\n73\r\n$2\r\n74\r\n$2\r\n75\r\n$2\r\n76\r\n$2\r\n77\r\n$2\r\n78\r\n$2\r\n79\r\n$2\r\n80\r\n$2\r\n81\r\n$2\r\n82\r\n$2\r\n83\r\n$2\r\n84\r\n$2\r\n85\r\n$2\r\n86\r\n$2\r\n87\r\n$2\r\n88\r\n$2\r\n89\r\n$2\r\n90\r\n$2\r\n91\r\n$2\r\n92\r\n$2\r\n93\r\n$2\r\n94\r\n$2\r\n95\r\n$2\r\n96\r\n$2\r\n97\r\n$2\r\n98\r\n$2\r\n99\r\n$3\r\n100\r\n$3\r\n101\r\n$3\r\n102\r\n$3\r\n103\r\n$3\r\n104\r\n$3\r\n105\r\n$3\r\n106\r\n$3\r\n107\r\n$3\r\n108\r\n$3\r\n109\r\n$3\r\n110\r\n$3\r\n111\r\n$3\r\n112\r\n$3\r\n113\r\n$3\r\n114\r\n$3\r\n115\r\n$3\r\n116\r\n$3\r\n117\r\n$3\r\n118\r\n$3\r\n119\r\n$3\r\n120\r\n$3\r\n121\r\n$3\r\n122\r\n$3\r\n123\r\n$3\r\n124\r\n$3\r\n125\r\n$3\r\n126\r\n$3\r\n127\r\n$3\r\n128\r\n$3\r\n129\r\n$3\r\n130\r\n$3\r\n131\r\n$3\r\n132\r\n$3\r\n133\r\n$3\r\n134\r\n$3\r\n135\r\n$3\r\n136\r\n$3\r\n137\r\n$3\r\n138\r\n$3\r\n139\r\n$3\r\n140\r\n$3\r\n141\r\n$3\r\n142\r\n$3\r\n143\r\n$3\r\n144\r\n$3\r\n145\r\n$3\r\n146\r\n$3\r\n147\r\n$3\r\n148\r\n$3\r\n149\r\n$3\r\n150\r\n$3\r\n151\r\n$3\r\n152\r\n$3\r\n153\r\n$3\r\n154\r\n$3\r\n155\r\n$3\r\n156\r\n$3\r\n157\r\n$3\r\n158\r\n$3\r\n159\r\n$3\r\n160\r\n$3\r\n161\r\n$3\r\n162\r\n$3\r\n163\r\n$3\r\n164\r\n$3\r\n165\r\n$3\r\n166\r\n$3\r\n167\r\n$3\r\n168\r\n$3\r\n169\r\n$3\r\n170\r\n$3\r\n171\r\n$3\r\n172\r\n$3\r\n173\r\n$3\r\n1">>, B2 = <<"74\r\n$3\r\n175\r\n$3\r\n176\r\n$3\r\n177\r\n$3\r\n178\r\n$3\r\n179\r\n$3\r\n180\r\n$3\r\n181\r\n$3\r\n182\r\n$3\r\n183\r\n$3\r\n184\r\n$3\r\n185\r\n$3\r\n186\r\n$3\r\n187\r\n$3\r\n188\r\n$3\r\n189\r\n$3\r\n190\r\n$3\r\n191\r\n$3\r\n192\r\n$3\r\n193\r\n$3\r\n194\r\n$3\r\n195\r\n$3\r\n196\r\n$3\r\n197\r\n$3\r\n198\r\n$3\r\n199\r\n$3\r\n200\r\n$3\r\n201\r\n$3\r\n202\r\n$3\r\n203\r\n$3\r\n204\r\n$3\r\n205\r\n$3\r\n206\r\n$3\r\n207\r\n$3\r\n208\r\n$3\r\n209\r\n$3\r\n210\r\n$3\r\n211\r\n$3\r\n212\r\n$3\r\n213\r\n$3\r\n214\r\n$3\r\n215\r\n$3\r\n216\r\n$3\r\n217\r\n$3\r\n218\r\n$3\r\n219\r\n$3\r\n220\r\n$3\r\n221\r\n$3\r\n222\r\n$3\r\n223\r\n$3\r\n224\r\n$3\r\n225\r\n$3\r\n226\r\n$3\r\n227\r\n$3\r\n228\r\n$3\r\n229\r\n$3\r\n230\r\n$3\r\n231\r\n$3\r\n232\r\n$3\r\n233\r\n$3\r\n234\r\n$3\r\n235\r\n$3\r\n236\r\n$3\r\n237\r\n$3\r\n238\r\n$3\r\n239\r\n$3\r\n240\r\n$3\r\n241\r\n$3\r\n242\r\n$3\r\n243\r\n$3\r\n244\r\n$3\r\n245\r\n$3\r\n246\r\n$3\r\n247\r\n$3\r\n248\r\n$3\r\n249\r\n$3\r\n250\r\n$3\r\n251\r\n$3\r\n252\r\n$3\r\n253\r\n$3\r\n254\r\n$3\r\n255\r\n$3\r\n256\r\n$3\r\n257\r\n$3\r\n258\r\n$3\r\n259\r\n$3\r\n260\r\n$3\r\n261\r\n$3\r\n262\r\n$3\r\n263\r\n$3\r\n264\r\n$3\r\n265\r\n$3\r\n266\r\n$3\r\n267\r\n$3\r\n268\r\n$3\r\n269\r\n$3\r\n270\r\n$3\r\n271\r\n$3\r\n272\r\n$3\r\n273\r\n$3\r\n274\r\n$3\r\n275\r\n$3\r\n276\r\n$3\r\n277\r\n$3\r\n278\r\n$3\r\n279\r\n$3\r\n280\r\n$3\r\n281\r\n$3\r\n282\r\n$3\r\n283\r\n$3\r\n284\r\n$3\r\n285\r\n$3\r\n286\r\n$3\r\n287\r\n$3\r\n288\r\n$3\r\n289\r\n$3\r\n290\r\n$3\r\n291\r\n$3\r\n292\r\n$3\r\n293\r\n$3\r\n294\r\n$3\r\n295\r\n$3\r\n296\r\n$3\r\n297\r\n$3\r\n298\r\n$3\r\n299\r\n$3\r\n300\r\n$3\r\n301\r\n$3\r\n302\r\n$3\r\n303\r\n$3\r\n304\r\n$3\r\n305\r\n$3\r\n306\r\n$3\r\n307\r\n$3\r\n308\r\n$3\r\n309\r\n$3\r\n310\r\n$3\r\n311\r\n$3\r\n312\r\n$3\r\n313\r\n$3\r\n314\r\n$3\r\n315\r\n$3\r\n316\r\n$3\r\n317\r\n$3\r\n318\r\n$3\r\n319\r\n$3\r\n320\r\n$3\r\n321\r\n$3\r\n322\r\n$3\r\n323\r\n$3\r\n324\r\n$3\r\n325\r\n$3\r\n326\r\n$3\r\n327\r\n$3\r\n328\r\n$3\r\n329\r\n$3\r\n330\r\n$3\r\n331\r\n$3\r\n332\r\n$3\r\n333\r\n$3\r\n334\r\n$3\r\n335\r\n$3\r\n336">>, B3 = <<"\r\n$3\r\n337\r\n$3\r\n338\r\n$3\r\n339\r\n$3\r\n340\r\n$3\r\n341\r\n$3\r\n342\r\n$3\r\n343\r\n$3\r\n344\r\n$3\r\n345\r\n$3\r\n346\r\n$3\r\n347\r\n$3\r\n348\r\n$3\r\n349\r\n$3\r\n350\r\n$3\r\n351\r\n$3\r\n352\r\n$3\r\n353\r\n$3\r\n354\r\n$3\r\n355\r\n$3\r\n356\r\n$3\r\n357\r\n$3\r\n358\r\n$3\r\n359\r\n$3\r\n360\r\n$3\r\n361\r\n$3\r\n362\r\n$3\r\n363\r\n$3\r\n364\r\n$3\r\n365\r\n$3\r\n366\r\n$3\r\n367\r\n$3\r\n368\r\n$3\r\n369\r\n$3\r\n370\r\n$3\r\n371\r\n$3\r\n372\r\n$3\r\n373\r\n$3\r\n374\r\n$3\r\n375\r\n$3\r\n376\r\n$3\r\n377\r\n$3\r\n378\r\n$3\r\n379\r\n$3\r\n380\r\n$3\r\n381\r\n$3\r\n382\r\n$3\r\n383\r\n$3\r\n384\r\n$3\r\n385\r\n$3\r\n386\r\n$3\r\n387\r\n$3\r\n388\r\n$3\r\n389\r\n$3\r\n390\r\n$3\r\n391\r\n$3\r\n392\r\n$3\r\n393\r\n$3\r\n394\r\n$3\r\n395\r\n$3\r\n396\r\n$3\r\n397\r\n$3\r\n398\r\n$3\r\n399\r\n$3\r\n400\r\n$3\r\n401\r\n$3\r\n402\r\n$3\r\n403\r\n$3\r\n404\r\n$3\r\n405\r\n$3\r\n406\r\n$3\r\n407\r\n$3\r\n408\r\n$3\r\n409\r\n$3\r\n410\r\n$3\r\n411\r\n$3\r\n412\r\n$3\r\n413\r\n$3\r\n414\r\n$3\r\n415\r\n$3\r\n416\r\n$3\r\n417\r\n$3\r\n418\r\n$3\r\n419\r\n$3\r\n420\r\n$3\r\n421\r\n$3\r\n422\r\n$3\r\n423\r\n$3\r\n424\r\n$3\r\n425\r\n$3\r\n426\r\n$3\r\n427\r\n$3\r\n428\r\n$3\r\n429\r\n$3\r\n430\r\n$3\r\n431\r\n$3\r\n432\r\n$3\r\n433\r\n$3\r\n434\r\n$3\r\n435\r\n$3\r\n436\r\n$3\r\n437\r\n$3\r\n438\r\n$3\r\n439\r\n$3\r\n440\r\n$3\r\n441\r\n$3\r\n442\r\n$3\r\n443\r\n$3\r\n444\r\n$3\r\n445\r\n$3\r\n446\r\n$3\r\n447\r\n$3\r\n448\r\n$3\r\n449\r\n$3\r\n450\r\n$3\r\n451\r\n$3\r\n452\r\n$3\r\n453\r\n$3\r\n454\r\n$3\r\n455\r\n$3\r\n456\r\n$3\r\n457\r\n$3\r\n458\r\n$3\r\n459\r\n$3\r\n460\r\n$3\r\n461\r\n$3\r\n462\r\n$3\r\n463\r\n$3\r\n464\r\n$3\r\n465\r\n$3\r\n466\r\n$3\r\n467\r\n$3\r\n468\r\n$3\r\n469\r\n$3\r\n470\r\n$3\r\n471\r\n$3\r\n472\r\n$3\r\n473\r\n$3\r\n474\r\n$3\r\n475\r\n$3\r\n476\r\n$3\r\n477\r\n$3\r\n478\r\n$3\r\n479\r\n$3\r\n480\r\n$3\r\n481\r\n$3\r\n482\r\n$3\r\n483\r\n$3\r\n484\r\n$3\r\n485\r\n$3\r\n486\r\n$3\r\n487\r\n$3\r\n488\r\n$3\r\n489\r\n$3\r\n490\r\n$3\r\n491\r\n$3\r\n492\r\n$3\r\n493\r\n$3\r\n494\r\n$3\r\n495\r\n$3\r\n496\r\n$3\r\n497\r\n$3\r\n498\r\n">>, {continue, ContinuationData1} = parse_multibulk(B1), {continue, ContinuationData2} = parse_multibulk(ContinuationData1, B2), EmptyBuffer = buffer_create(), ?assertMatch({continue, {in_parsing_bulks, 2, EmptyBuffer, _}}, parse_multibulk(ContinuationData2, B3)). %% @doc: Test a binary string which contains \r\n inside it's data binary_safe_test() -> B = <<"$14\r\nfoobar\r\nbarbaz\r\n">>, ?assertEqual({ok, <<"foobar\r\nbarbaz">>, init()}, parse(init(), B)). status_test() -> B = <<"+OK\r\n">>, ?assertEqual({ok, <<"OK">>, init()}, parse(init(), B)). status_chunked_test() -> B1 = <<"+O">>, B2 = <<"K\r\n">>, State1 = init(), ?assertEqual({continue, #pstate{state = status_continue, continuation_data = {incomplete_simple, buffer_create(<<"O">>)}}}, parse(State1, B1)), {continue, State2} = parse(State1, B1), ?assertEqual({ok, <<"OK">>, init()}, parse(State2, B2)). error_test() -> B = <<"-ERR wrong number of arguments for 'get' command\r\n">>, ?assertEqual({error, <<"ERR wrong number of arguments for 'get' command">>, init()}, parse(init(), B)). integer_test() -> B = <<":2\r\n">>, ?assertEqual({ok, <<"2">>, init()}, parse(init(), B)). integer_reply_inside_multibulk_test() -> B = <<"*2\r\n:1\r\n:1\r\n">>, ?assertEqual({ok, [<<"1">>, <<"1">>], init()}, parse(init(), B)). status_inside_multibulk_test() -> B = <<"*2\r\n+OK\r\n:1\r\n">>, ?assertEqual({ok, [<<"OK">>, <<"1">>], init()}, parse(init(), B)). error_inside_multibulk_test() -> B = <<"*2\r\n-ERR foobar\r\n:1\r\n">>, ?assertEqual({ok, [<<"ERR foobar">>, <<"1">>], init()}, parse(init(), B)). eredis-1.1.0/test/eredis_sub_tests.erl0000644000232200023220000002002513145103330020353 0ustar debalancedebalance-module(eredis_sub_tests). -include_lib("eunit/include/eunit.hrl"). -include("eredis.hrl"). -include("eredis_sub.hrl"). -import(eredis, [create_multibulk/1]). c() -> Res = eredis:start_link(), ?assertMatch({ok, _}, Res), {ok, C} = Res, C. s() -> Res = eredis_sub:start_link("127.0.0.1", 6379, ""), ?assertMatch({ok, _}, Res), {ok, C} = Res, C. add_channels(Sub, Channels) -> ok = eredis_sub:controlling_process(Sub), ok = eredis_sub:subscribe(Sub, Channels), lists:foreach( fun (C) -> receive M -> ?assertEqual({subscribed, C, Sub}, M), eredis_sub:ack_message(Sub) end end, Channels). pubsub_test() -> Pub = c(), Sub = s(), add_channels(Sub, [<<"chan1">>, <<"chan2">>]), ok = eredis_sub:controlling_process(Sub), ?assertEqual({ok, <<"1">>}, eredis:q(Pub, ["PUBLISH", chan1, msg])), receive {message, _, _, _} = M -> ?assertEqual({message, <<"chan1">>, <<"msg">>, Sub}, M) after 10 -> throw(timeout) end, receive Msg -> throw({unexpected_message, Msg}) after 5 -> ok end, eredis_sub:stop(Sub). %% Push size so high, the queue will be used pubsub2_test() -> Pub = c(), Sub = s(), add_channels(Sub, [<<"chan">>]), ok = eredis_sub:controlling_process(Sub), lists:foreach( fun(_) -> Msg = binary:copy(<<"0">>, 2048), ?assertEqual({ok, <<"1">>}, eredis:q(Pub, [publish, chan, Msg])) end, lists:seq(1, 500)), Msgs = recv_all(Sub), ?assertEqual(500, length(Msgs)), eredis_sub:stop(Sub). pubsub_manage_subscribers_test() -> Pub = c(), Sub = s(), add_channels(Sub, [<<"chan">>]), unlink(Sub), Self = self(), ?assertMatch(#state{controlling_process={_, Self}}, get_state(Sub)), S1 = subscriber(Sub), ok = eredis_sub:controlling_process(Sub, S1), #state{controlling_process={_, S1}} = get_state(Sub), S2 = subscriber(Sub), ok = eredis_sub:controlling_process(Sub, S2), #state{controlling_process={_, S2}} = get_state(Sub), eredis:q(Pub, ["PUBLISH", chan, msg1]), S1 ! stop, ok = wait_for_stop(S1), eredis:q(Pub, ["PUBLISH", chan, msg2]), ?assertEqual({message, <<"chan">>, <<"msg1">>, Sub}, wait_for_msg(S2)), ?assertEqual({message, <<"chan">>, <<"msg2">>, Sub}, wait_for_msg(S2)), S2 ! stop, ok = wait_for_stop(S2), Ref = erlang:monitor(process, Sub), receive {'DOWN', Ref, process, Sub, _} -> ok end. pubsub_connect_disconnect_messages_test() -> Pub = c(), Sub = s(), add_channels(Sub, [<<"chan">>]), S = subscriber(Sub), ok = eredis_sub:controlling_process(Sub, S), eredis:q(Pub, ["PUBLISH", chan, msg]), wait_for_msg(S), #state{socket=Sock} = get_state(Sub), gen_tcp:close(Sock), Sub ! {tcp_closed, Sock}, ?assertEqual({eredis_disconnected, Sub}, wait_for_msg(S)), ?assertEqual({eredis_reconnect_attempt, Sub}, wait_for_msg(S)), ?assertEqual({eredis_connected, Sub}, wait_for_msg(S)), eredis_sub:stop(Sub). drop_queue_test() -> Pub = c(), {ok, Sub} = eredis_sub:start_link("127.0.0.1", 6379, "", 100, 10, drop), add_channels(Sub, [<<"foo">>]), ok = eredis_sub:controlling_process(Sub), [eredis:q(Pub, [publish, foo, N]) || N <- lists:seq(1, 12)], receive M1 -> ?assertEqual({message,<<"foo">>,<<"1">>, Sub}, M1) end, receive M2 -> ?assertEqual({dropped, 11}, M2) end, eredis_sub:stop(Sub). crash_queue_test() -> Pub = c(), {ok, Sub} = eredis_sub:start_link("127.0.0.1", 6379, "", 100, 10, exit), add_channels(Sub, [<<"foo">>]), true = unlink(Sub), ok = eredis_sub:controlling_process(Sub), Ref = erlang:monitor(process, Sub), [eredis:q(Pub, [publish, foo, N]) || N <- lists:seq(1, 12)], receive M1 -> ?assertEqual({message,<<"foo">>,<<"1">>, Sub}, M1) end, receive M2 -> ?assertEqual({'DOWN', Ref, process, Sub, max_queue_size}, M2) end. dynamic_channels_test() -> Pub = c(), Sub = s(), ok = eredis_sub:controlling_process(Sub), eredis:q(Pub, [publish, newchan, foo]), receive {message, <<"foo">>, _, _} -> ?assert(false) after 5 -> ok end, %% We do the following twice to show that subscribing to the same channel %% doesn't cause the channel to show up twice lists:foreach(fun(_) -> eredis_sub:subscribe(Sub, [<<"newchan">>, <<"otherchan">>]), receive M1 -> ?assertEqual({subscribed, <<"newchan">>, Sub}, M1) end, eredis_sub:ack_message(Sub), receive M2 -> ?assertEqual({subscribed, <<"otherchan">>, Sub}, M2) end, eredis_sub:ack_message(Sub), {ok, Channels} = eredis_sub:channels(Sub), ?assertEqual(true, lists:member(<<"otherchan">>, Channels)), ?assertEqual(true, lists:member(<<"newchan">>, Channels)), ?assertEqual(2, length(Channels)) end, lists:seq(0, 1)), eredis:q(Pub, [publish, newchan, foo]), ?assertEqual([{message, <<"newchan">>, <<"foo">>, Sub}], recv_all(Sub)), eredis:q(Pub, [publish, otherchan, foo]), ?assertEqual([{message, <<"otherchan">>, <<"foo">>, Sub}], recv_all(Sub)), eredis_sub:unsubscribe(Sub, [<<"otherchan">>]), eredis_sub:ack_message(Sub), receive M3 -> ?assertEqual({unsubscribed, <<"otherchan">>, Sub}, M3) end, ?assertEqual({ok, [<<"newchan">>]}, eredis_sub:channels(Sub)). recv_all(Sub) -> recv_all(Sub, []). recv_all(Sub, Acc) -> receive {message, _, _, _} = InMsg -> eredis_sub:ack_message(Sub), recv_all(Sub, [InMsg | Acc]) after 5 -> lists:reverse(Acc) end. subscriber(Client) -> Test = self(), Pid = spawn(fun () -> subscriber(Client, Test) end), spawn(fun() -> Ref = erlang:monitor(process, Pid), receive {'DOWN', Ref, _, _, _} -> Test ! {stopped, Pid} end end), Pid. subscriber(Client, Test) -> receive stop -> ok; Msg -> Test ! {got_message, self(), Msg}, eredis_sub:ack_message(Client), subscriber(Client, Test) end. wait_for_msg(Subscriber) -> receive {got_message, Subscriber, Msg} -> Msg end. wait_for_stop(Subscriber) -> receive {stopped, Subscriber} -> ok end. get_state(Pid) when is_pid(Pid) -> {status, _, _, [_, _, _, _, State]} = sys:get_status(Pid), get_state(State); get_state([{data, [{"State", State}]} | _]) -> State; get_state([_|Rest]) -> get_state(Rest). % Tests for Pattern Subscribe add_channels_pattern(Sub, Channels) -> ok = eredis_sub:controlling_process(Sub), ok = eredis_sub:psubscribe(Sub, Channels), lists:foreach( fun (C) -> receive M -> ?assertEqual({subscribed, C, Sub}, M), eredis_sub:ack_message(Sub) end end, Channels). pubsub_pattern_test() -> Pub = c(), Sub = s(), add_channels_pattern(Sub, [<<"chan1*">>, <<"chan2*">>]), ok = eredis_sub:controlling_process(Sub), ?assertEqual({ok, <<"1">>}, eredis:q(Pub, ["PUBLISH", <<"chan123">>, <<"msg">>])), receive {pmessage, _Pattern, _Channel, _Message, _} = M -> ?assertEqual({pmessage, <<"chan1*">>,<<"chan123">>, <<"msg">>, Sub}, M) after 10 -> throw(timeout) end, eredis_sub:punsubscribe(Sub, [<<"chan1*">> , <<"chan2*">>]), eredis_sub:ack_message(Sub), eredis_sub:ack_message(Sub), receive {unsubscribed,_,_} = M2 -> ?assertEqual({unsubscribed, <<"chan1*">>, Sub}, M2) end, eredis_sub:ack_message(Sub), receive {unsubscribed,_,_} = M3 -> ?assertEqual({unsubscribed, <<"chan2*">>, Sub}, M3) end, eredis_sub:ack_message(Sub), ?assertEqual({ok, <<"0">>}, eredis:q(Pub, ["PUBLISH", <<"chan123">>, <<"msg">>])), receive Msg -> throw({unexpected_message, Msg}) after 10 -> ok end, eredis_sub:stop(Sub). eredis-1.1.0/test/eredis_tests.erl0000644000232200023220000001451313145103330017507 0ustar debalancedebalance-module(eredis_tests). -include_lib("eunit/include/eunit.hrl"). -include("eredis.hrl"). -import(eredis, [create_multibulk/1]). connect_test() -> ?assertMatch({ok, _}, eredis:start_link("127.0.0.1", 6379)), ?assertMatch({ok, _}, eredis:start_link("localhost", 6379)). get_set_test() -> C = c(), ?assertMatch({ok, _}, eredis:q(C, ["DEL", foo])), ?assertEqual({ok, undefined}, eredis:q(C, ["GET", foo])), ?assertEqual({ok, <<"OK">>}, eredis:q(C, ["SET", foo, bar])), ?assertEqual({ok, <<"bar">>}, eredis:q(C, ["GET", foo])). delete_test() -> C = c(), ?assertMatch({ok, _}, eredis:q(C, ["DEL", foo])), ?assertEqual({ok, <<"OK">>}, eredis:q(C, ["SET", foo, bar])), ?assertEqual({ok, <<"1">>}, eredis:q(C, ["DEL", foo])), ?assertEqual({ok, undefined}, eredis:q(C, ["GET", foo])). mset_mget_test() -> C = c(), Keys = lists:seq(1, 1000), ?assertMatch({ok, _}, eredis:q(C, ["DEL" | Keys])), KeyValuePairs = [[K, K*2] || K <- Keys], ExpectedResult = [list_to_binary(integer_to_list(K * 2)) || K <- Keys], ?assertEqual({ok, <<"OK">>}, eredis:q(C, ["MSET" | lists:flatten(KeyValuePairs)])), ?assertEqual({ok, ExpectedResult}, eredis:q(C, ["MGET" | Keys])), ?assertMatch({ok, _}, eredis:q(C, ["DEL" | Keys])). exec_test() -> C = c(), ?assertMatch({ok, _}, eredis:q(C, ["LPUSH", "k1", "b"])), ?assertMatch({ok, _}, eredis:q(C, ["LPUSH", "k1", "a"])), ?assertMatch({ok, _}, eredis:q(C, ["LPUSH", "k2", "c"])), ?assertEqual({ok, <<"OK">>}, eredis:q(C, ["MULTI"])), ?assertEqual({ok, <<"QUEUED">>}, eredis:q(C, ["LRANGE", "k1", "0", "-1"])), ?assertEqual({ok, <<"QUEUED">>}, eredis:q(C, ["LRANGE", "k2", "0", "-1"])), ExpectedResult = [[<<"a">>, <<"b">>], [<<"c">>]], ?assertEqual({ok, ExpectedResult}, eredis:q(C, ["EXEC"])), ?assertMatch({ok, _}, eredis:q(C, ["DEL", "k1", "k2"])). exec_nil_test() -> C1 = c(), C2 = c(), ?assertEqual({ok, <<"OK">>}, eredis:q(C1, ["WATCH", "x"])), ?assertMatch({ok, _}, eredis:q(C2, ["INCR", "x"])), ?assertEqual({ok, <<"OK">>}, eredis:q(C1, ["MULTI"])), ?assertEqual({ok, <<"QUEUED">>}, eredis:q(C1, ["GET", "x"])), ?assertEqual({ok, undefined}, eredis:q(C1, ["EXEC"])), ?assertMatch({ok, _}, eredis:q(C1, ["DEL", "x"])). pipeline_test() -> C = c(), P1 = [["SET", a, "1"], ["LPUSH", b, "3"], ["LPUSH", b, "2"]], ?assertEqual([{ok, <<"OK">>}, {ok, <<"1">>}, {ok, <<"2">>}], eredis:qp(C, P1)), P2 = [["MULTI"], ["GET", a], ["LRANGE", b, "0", "-1"], ["EXEC"]], ?assertEqual([{ok, <<"OK">>}, {ok, <<"QUEUED">>}, {ok, <<"QUEUED">>}, {ok, [<<"1">>, [<<"2">>, <<"3">>]]}], eredis:qp(C, P2)), ?assertMatch({ok, _}, eredis:q(C, ["DEL", a, b])). pipeline_mixed_test() -> C = c(), P1 = [["LPUSH", c, "1"] || _ <- lists:seq(1, 100)], P2 = [["LPUSH", d, "1"] || _ <- lists:seq(1, 100)], Expect = [{ok, list_to_binary(integer_to_list(I))} || I <- lists:seq(1, 100)], spawn(fun () -> erlang:yield(), ?assertEqual(Expect, eredis:qp(C, P1)) end), spawn(fun () -> ?assertEqual(Expect, eredis:qp(C, P2)) end), timer:sleep(10), ?assertMatch({ok, _}, eredis:q(C, ["DEL", c, d])). q_noreply_test() -> C = c(), ?assertEqual(ok, eredis:q_noreply(C, ["GET", foo])), ?assertEqual(ok, eredis:q_noreply(C, ["SET", foo, bar])), %% Even though q_noreply doesn't wait, it is sent before subsequent requests: ?assertEqual({ok, <<"bar">>}, eredis:q(C, ["GET", foo])). q_async_test() -> C = c(), ?assertEqual({ok, <<"OK">>}, eredis:q(C, ["SET", foo, bar])), ?assertEqual(ok, eredis:q_async(C, ["GET", foo], self())), receive {response, Msg} -> ?assertEqual(Msg, {ok, <<"bar">>}), ?assertMatch({ok, _}, eredis:q(C, ["DEL", foo])) end. c() -> Res = eredis:start_link(), ?assertMatch({ok, _}, Res), {ok, C} = Res, C. c_no_reconnect() -> Res = eredis:start_link("127.0.0.1", 6379, 0, "", no_reconnect), ?assertMatch({ok, _}, Res), {ok, C} = Res, C. multibulk_test_() -> [?_assertEqual(<<"*3\r\n$3\r\nSET\r\n$3\r\nfoo\r\n$3\r\nbar\r\n">>, list_to_binary(create_multibulk(["SET", "foo", "bar"]))), ?_assertEqual(<<"*3\r\n$3\r\nSET\r\n$3\r\nfoo\r\n$3\r\nbar\r\n">>, list_to_binary(create_multibulk(['SET', foo, bar]))), ?_assertEqual(<<"*3\r\n$3\r\nSET\r\n$3\r\nfoo\r\n$3\r\n123\r\n">>, list_to_binary(create_multibulk(['SET', foo, 123]))), ?_assertThrow({cannot_store_floats, 123.5}, list_to_binary(create_multibulk(['SET', foo, 123.5]))) ]. undefined_database_test() -> ?assertMatch({ok,_}, eredis:start_link("localhost", 6379, undefined)). tcp_closed_test() -> C = c(), tcp_closed_rig(C). tcp_closed_no_reconnect_test() -> C = c_no_reconnect(), tcp_closed_rig(C). tcp_closed_rig(C) -> %% fire async requests to add to redis client queue and then trick %% the client into thinking the connection to redis has been %% closed. This behavior can be observed when Redis closes an idle %% connection just as a traffic burst starts. DoSend = fun(tcp_closed) -> C ! {tcp_closed, fake_socket}; (Cmd) -> eredis:q(C, Cmd) end, %% attach an id to each message for later Msgs = [{1, ["GET", "foo"]}, {2, ["GET", "bar"]}, {3, tcp_closed}], Pids = [ remote_query(DoSend, M) || M <- Msgs ], Results = gather_remote_queries(Pids), ?assertEqual({error, tcp_closed}, proplists:get_value(1, Results)), ?assertEqual({error, tcp_closed}, proplists:get_value(2, Results)). remote_query(Fun, {Id, Cmd}) -> Parent = self(), spawn(fun() -> Result = Fun(Cmd), Parent ! {self(), Id, Result} end). gather_remote_queries(Pids) -> gather_remote_queries(Pids, []). gather_remote_queries([], Acc) -> Acc; gather_remote_queries([Pid | Rest], Acc) -> receive {Pid, Id, Result} -> gather_remote_queries(Rest, [{Id, Result} | Acc]) after 10000 -> error({gather_remote_queries, timeout}) end. eredis-1.1.0/rebar.lock0000644000232200023220000000000413145103330015262 0ustar debalancedebalance[]. eredis-1.1.0/LICENSE0000644000232200023220000000205613145103330014333 0ustar debalancedebalanceThe MIT License Copyright (c) 2011 wooga GmbH Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.