Managing Redis Reconnections from Ruby

Recently a new user of the Readthis caching library inquired about where to force Redis to reconnect after the application booted:

When restarting unicorn with the USR2 signal, a new master is created and workers are forked off. In my config/unicorn.rb, before switching to readthis, I had Rails.cache.reconnect to reconnect to redis after forking. I believe this was an implementation of the redis-store gem, which you aren't using here.

How would you suggest I reconnect the unicorn worker to the redis-based cache with readthis? Thanks!

—Justin Downing

The good news is, after redis-rb 3.1.0 you don't need to manually reconnect your redis clients. In fact, it isn't desirable to do so! It was common, historically, to force a Redis reconnect after a Unicorn or Puma cluster forked off child workers. That was in order to avoid sharing the same socket between multiple processes, a recipe for unpredictable behavior and general mayhem. The alternative to manually reconnecting was an error from Redis warning you about the insanity that would ensue:

Tried to use a connection from a child process without reconnecting. You need to reconnect to Redis after forking or set :inherit_socket to true.

Reconnecting after a child forks is just one of the errors that the Redis client will automatically recover from. This post aims to provide some more context and a whiff of exploration into how the redis client heals itself.

Stealing the Fork Safety Test

For proof of the reconnection claim and a concrete point of reference we'll co-opt an example from the redis-rb test suite. Borrowing from the fork_safety_test:

require "redis"

redis = Redis.new
redis.set("foo", 1)

child_pid = fork do
  begin
    redis.set("foo", 2)
  rescue Redis::InheritedError
    exit 127
  end
end

_, status = Process.wait2(child_pid)

puts status.exitstatus #=> 0
puts redis.get("foo")  #=> "2"

The code snippet starts out by instantiating a client in the parent process. It then forces a connection to be established by calling set. The Redis client is lazy, so it will only establish a connection the first time a command is sent— without this initial connection before the fork there wouldn't be any socket inheritance to test. Immediately after setting the value 1 a child process is forked, inheriting the parent's Redis instance, connection and all.

After waiting for the process to return we can see that it exited without a problem, returning a happy 0 status. Additionally, the child's set command was successful in overwriting foo with the value 2, so we know everything worked as expected.

Accomplishing Resiliency

How does the client know to reconnect after a fork? It's relatively simple. All commands executed by the client are centralized, passing through a chain of base methods. It is within this method chain that common behavior such as logging and connection management are guaranteed. The abbreviated flow of methods looks like:

call -> process -> ensure_connected

The redis-rb source is idiomatic, straight forward, and an excellent place to learn about connections and the Redis command protocol. It is, however, too verbose to include verbatim in this post, so the code sample has been modified from its original context for clarity.

The ensure_connected method is, predictably, where the reconnection magic happens:

# lib/redis/client.rb#334

def ensure_connected
  attempts = 0

  begin
    attempts += 1

    if connected?
      unless inherit_socket? || Process.pid == @pid
        raise InheritedError, INHERITED_MESSAGE
      end
    else
      connect
    end

    yield
  rescue BaseConnectionError
    disconnect

    if attempts <= @options[:reconnect_attempts] && @reconnect
      retry
    else
      raise
    end
  rescue Exception
    disconnect
    raise
  end
end

Within a begin/retry block the connection is verified and the current PID is compared to the PID from when the connection was established. If the PID is different then the process has since forked and an InheritedError is raised. InheritedError is one of numerous specific connection errors that inherit from BaseConnectionError:

# lib/redis/errors.rb#37

# Raised when the connection was inherited by a child process.
class InheritedError < BaseConnectionError
end

When BaseConnectionError is rescued there is an immediate disconnect, dropping the connection and clearing the old PID. Provided the reconnect attempt is lower than the reconnect limit, the block is retried and a new connection is established. The reconnection mechanism is guarded by tracking the number of attempts. It is guaranteed not to reconnect infinitely when faced with persistent connection errors.

By centralizing the execution of commands, the client keeps connection management simple and understandable.

Forget About It

Stop worrying about managing your application's connections to Redis. Problems with a network outage, unexpectedly closed connection, or an inherited socket error? Not a problem, the Ruby client for Redis has you covered.