Recently a new user of the Readthis caching library inquired about where to force Redis to reconnect after the application booted:
When restarting unicorn with the USR2 signal, a new master is created and workers are forked off. In my config/unicorn.rb, before switching to readthis, I had
Rails.cache.reconnectto reconnect to redis after forking. I believe this was an implementation of the redis-store gem, which you aren’t using here.
How would you suggest I reconnect the unicorn worker to the redis-based cache with readthis? Thanks!
The good news is, after
redis-rb 3.1.0 you don’t need to
manually reconnect your redis clients. In fact, it isn’t desirable to do so! It
was common, historically, to force a Redis reconnect after a Unicorn or Puma
cluster forked off child workers. That was in order to avoid sharing the same
socket between multiple processes, a recipe for unpredictable behavior and
general mayhem. The alternative to manually reconnecting was an error from Redis
warning you about the insanity that would ensue:
Tried to use a connection from a child process without reconnecting. You need to reconnect to Redis after forking or set :inherit_socket to true.
Reconnecting after a child forks is just one of the errors that the Redis client will automatically recover from. This post aims to provide some more context and a whiff of exploration into how the redis client heals itself.
Stealing the Fork Safety Test
For proof of the reconnection claim and a concrete point of reference we’ll
co-opt an example from the
redis-rb test suite. Borrowing from the
require "redis" redis = Redis.new redis.set("foo", 1) child_pid = fork do begin redis.set("foo", 2) rescue Redis::InheritedError exit 127 end end _, status = Process.wait2(child_pid) puts status.exitstatus #=> 0 puts redis.get("foo") #=> "2"
The code snippet starts out by instantiating a client in the parent process. It
then forces a connection to be established by calling
set. The Redis client is
lazy, so it will only establish a connection the first time a command is sent—
without this initial connection before the fork there wouldn’t be any socket
inheritance to test. Immediately after setting the value
1 a child process is
forked, inheriting the parent’s Redis instance, connection and all.
After waiting for the process to return we can see that it exited without a
problem, returning a happy
0 status. Additionally, the child’s
was successful in overwriting
foo with the value
2, so we know everything
worked as expected.
How does the client know to reconnect after a fork? It’s relatively simple. All commands executed by the client are centralized, passing through a chain of base methods. It is within this method chain that common behavior such as logging and connection management are guaranteed. The abbreviated flow of methods looks like:
call -> process -> ensure_connected
redis-rb source is idiomatic, straight forward, and an excellent place to
learn about connections and the Redis command protocol. It is, however, too
verbose to include verbatim in this post, so the code sample has been modified
from its original context for clarity.
ensure_connected method is, predictably, where the reconnection magic
# lib/redis/client.rb#334 def ensure_connected attempts = 0 begin attempts += 1 if connected? unless inherit_socket? || Process.pid == @pid raise InheritedError, INHERITED_MESSAGE end else connect end yield rescue BaseConnectionError disconnect if attempts <= @options[:reconnect_attempts] && @reconnect retry else raise end rescue Exception disconnect raise end end
begin/retry block the connection is verified and the current PID is
compared to the PID from when the connection was established. If the PID is
different then the process has since forked and an
InheritedError is raised.
InheritedError is one of numerous specific connection errors that inherit from
# lib/redis/errors.rb#37 # Raised when the connection was inherited by a child process. class InheritedError < BaseConnectionError end
BaseConnectionError is rescued there is an immediate disconnect, dropping
the connection and clearing the old PID. Provided the reconnect attempt is lower
than the reconnect limit, the block is retried and a new connection is
established. The reconnection mechanism is guarded by tracking the number of
attempts. It is guaranteed not to reconnect infinitely when faced with
persistent connection errors.
By centralizing the execution of commands, the client keeps connection management simple and understandable.
Forget About It
Stop worrying about managing your application’s connections to Redis. Problems with a network outage, unexpectedly closed connection, or an inherited socket error? Not a problem, the Ruby client for Redis has you covered.