Let's start with some highlights plucked from the CHANGELOG:
Unified Worker Callbacks — We've replaced the perform/2
callback with perform/1
, where the only argument is an Oban.Job
struct.
Along with the backoff/1
callback also accepting a job struct, this unifies the interface for all Oban.Worker
callbacks and helps to eliminate confusion around pattern matching on arguments.
Snooze — You can return {:snooze, seconds}
from a workers's perform/1
callback to reschedule a job some number of seconds in the future.
This is useful for recycling jobs that aren't ready to run yet, e.g. due to rate limiting or temporary errors that might resolve with time.
Discard — Rather than snoozing you can return :discard
from perform/1
to drop the job.
This is useful when a job encounters an error that won't resolve with time, e.g. invalid arguments or a missing record.
Test Helper — The new perform_job/2,3
helper automates validating, normalizing and performing jobs while unit testing.
It bore from catching the same mistakes and "gotchas" in code reviews.
This is the preferred way to unit test workers now.
Local Only Queues — The updated Oban.start_queue/2
function accepts a list of options, including the new local_only
flag, which allows you to dynamically start and stop queues only for the local node.
Crontab Improvements — New support for non-standard expressions such as @daily
, step values with ranges, and fixes for scheduling in a system running multiple Oban instances.
Standard Telemetry — Oban adopted Telemetry from the outset, before some standards had solidified.
Now that conventions have standardized we've switched to the span
convention and enhanced Telemetry events throughout the codebase.
Reliability Fixes — We've eliminated race conditions that allowed duplicate dispatch of new jobs and false positives when enqueuing unique jobs through improvements to the use of transactions and namespacing locks.
The CHANGELOG includes an upgrade guide for breaking changes and examples for select new features.
One other important addition to note is the new plugin system. As more companies use Oban, there is an increasing need for behaviour tailored to certain use cases. Those features require significant development time and can add a lot complexity to the codebase.
The plugin system helps us tackle those problems by moving those time-intensive and complex features to the Oban Pro package, which we will explore in depth soon. In particular, we've made the following changes:
The pruning system housed a lot of complexity to support all of the different use cases. Therefore, it we've simplified it so that it keeps jobs for 60 seconds, making sure Oban by default leaves a small footprint after execution. For those who want flexibility and/or historical data, Oban Web+Pro offers a functional UI to explore all past and on-going jobs alongside a fully featured pruning system
Producer activity is no longer recorded via heartbeats—this functionality powers Oban Web and now, with the plugin system, we were able to move it out
Stop rescuing of orphaned jobs. The rules around restarting jobs can be complex and domain dependent. For instance, automatically retrying a job could accidentally invoice a client twice. Therefore, manual intervention is the safest default. You can do so via the command line or via Oban Web's web interface. Once you see a pattern in your orphaned jobs, you can use Oban Pro's new Lifeline system or automate some of those restarts on your own
We've extracted, rewritten and dramatically improved all of the removed functionality to make it available through the new Oban Pro package.
When we started Oban, we believed it would fill a valuable gap in the Elixir community. We also knew that it would be important to find a sustainable model that would allow us to continue working and improving Oban throughout the years.
Oban adoption has gone well and it is widely used by Elixir powered businesses across an array of industries. Initially we launched Oban Web, which was our first experiment into finding a viable business model for Oban.
Ideally Oban Web would have been enough to sustain Oban's growth, but the feedback we got made it clear that we needed more. While Oban Web helped companies identify trends and act on their queues, their teams also wanted to be able to convert those trends back into complex business rules which are deeply integrated into their queuing system.
Given the effort required to implement and revamp those features, we've decided to give them a new home as Oban Pro. Oban Pro brings all of the missing pieces to Oban Web, now renamed to Oban Web, for the same price as before. Existing customers are upgraded to Oban Web+Pro for free. Oban Pro complements Oban Web with the following plugin powered features:
Flexible Historical Data — The new DynamicPruner
in Oban Pro allows you to specify either a maximum age or a maximum length and provide custom rules for specific queues, workers and job states.
This works great with unique jobs and Oban Web, allowing you to explore and introspect active and historic jobs from your browser
Lifeline — Rules for rescuing orphaned jobs. While you can tell Oban how much time your jobs have to shutdown; misconfiguration, bugs and system crashes may leave some jobs stuck on the “executing” state. With Oban Web+Pro, you can visualize those jobs and restart them from the Web and define rules to periodically restart them
Auto-Reprioritization — Companies that make extensive use of Oban's priority system may find themselves in a position where low priority jobs are not executed, especially during high-traffic and spikes. This feature prevents queue starvation by automatically adjusting priorities to ensure all jobs are eventually processed
In addition we're launching with an official Batch
worker.
The worker links the execution of many jobs as a group and runs optional callbacks after the group of jobs execute.
This allows your application to coordinate the execution of tens, hundreds or thousands of jobs in parallel.
It is an abstraction on top of the standard Oban.Worker
and a dramatic improvement over the old batch recipe.
Learn more about how you can put it to work for you in the official batch worker guide.
To sum up, Oban Pro is a collection of plugins, workers and extensions that improve Oban's reliability even further and make difficult workflows easier. Everything provided by Oban Pro builds on Oban OSS. There isn't anything hidden or any monkey-patching because Elixir keeps us honest. I hope you will give Oban Web+Pro a try and join other companies investing in Oban's future!
Going well beyond a simple renaming, Oban Web is an overhaul of the former UI. The product is entirely redesigned and rebuilt for extensibility, flexibility, clarity and speed. A few of the highlights:
No More Configuration — Oban Web piggybacks on the Oban configuration
No More Migrations — The update and search functionality no longer requires any database migrations
Router Integration — A new oban_dashboard
macro simplifies Phoenix router integration and makes it possible to mount multiple dashboards in the same application
Customizable Refresh — Change how frequently the dashboard refreshes, or pause updates entirely
Bulk Actions — Select multiple jobs and act on them in bulk to cancel, retry, discard or delete them all together
Queue Pausing — Pause and resume queues globally directly from the queue side panel
Queue Scaling — Scale queues up or down globally from the queue side panel
It's still early days, but we've released an alpha of Oban Web 2.0.0 which is compatible with Oban 2.0+ and bundles with Oban Pro. The upgrade process is minimal and mostly involves deleting code, so give it a try! This leads us to the next topic: where to find documentation on all of these changes.
Oban has thorough module documentation and an extensive README, but it lacked guides and walkthroughs beyond the posts of this blog. As of 2.0 there are a few initial guides and with thanks to support from the community, the recipes published on this blog are now available as guides (and updated for 2.0).
What's more, both Oban Web+Pro hook into the new guides structure to present docs side-by-side with Oban on hexdocs. There are docs on installation, troubleshooting, full product changelogs, and extensive guides for each feature.
Check out the Oban Pro docs and Oban Web docs.
As hinted earlier, Oban Web+Pro is available through a single license. If you already have an Oban Web license you now get Oban Pro as well!
Not only are the products available through a single license, there aren't any changes to the pricing tiers either. The only thing that is changing around license subscriptions is the removal of a trial period. Due to abuse by a few unsavory individuals we can no longer offer a seven day trial. Oban Web+Pro is a monthly subscription rather than an annual fee, so there is minimal financial risk to trying it out for a month.
The interest and adoption of Oban has been truly overwhelming. The feedback has been amazingly positive, which is a testament to how respectful and supportive the Elixir community is. Thank you for providing the drive to keep maintaining and improving Oban. There is so much more to do; our future is wide open!
Special thanks to Jesse Cooke, Milton Mazzarri and José Valim for reviewing this post.
]]>Running every job queue on every node isn't always ideal. Imagine that your application has some CPU intensive jobs that you'd prefer not to run on nodes that serve web requests. Perhaps you start temporary nodes that are only meant to insert jobs but should never execute any. Fortunately, we can control this by configuring certain node types, or even single nodes, to run only a subset of queues.
One notorious type of CPU intensive work is video processing. When our application is transcoding multiple videos simultaneously it is a major drain on system resources and may impact response times. To avoid this we can run dedicated worker nodes that don't serve any web requests and handle all of the transcoding.
While it's possible to separate our system into web
and worker
apps within an umbrella, that wouldn't allow us to dynamically change queues at runtime.
Let's look at an environment variable based method for dynamically configuring queues at runtime.
Within config.exs
our application is configured to run three queues: default
, media
and events
:
config :my_app, Oban,
repo: MyApp.Repo,
queues: [default: 15, media: 10, events: 25]
We will use an OBAN_QUEUES
environment variable to override the queues at runtime.
For illustration purposes the queue parsing all happens within the application module, but it would work equally well in releases.exs
.
defmodule MyApp.Application do
@moduledoc false
use Application
def start(_type, _args) do
children = [
MyApp.Repo,
MyApp.Endpoint,
{Oban, oban_opts()}
]
Supervisor.start_link(children, strategy: :one_for_one, name: MyApp.Supervisor)
end
defp oban_opts do
env_queues = System.get_env("OBAN_QUEUES")
:my_app
|> Application.get_env(Oban)
|> Keyword.update(:queues, [], &queues(env_queues, &1))
end
defp queues("*", defaults), do: defaults
defp queues(nil, defaults), do: defaults
defp queues(false, _), do: false
defp queues(values, _defaults) when is_binary(values) do
values
|> String.split(" ", trim: true)
|> Enum.map(&String.split(&1, ",", trim: true))
|> Keyword.new(fn [queue, limit] ->
{String.to_existing_atom(queue), String.to_integer(limit)}
end)
end
end
The queues
function's first three clauses ensure that we can fall back to the queues specified in our configuration (or false
, for testing).
The fourth clause is much more involved, and that is where the environment parsing happens.
It expects the OBAN_QUEUES
value to be a string formatted as queue,limit
pairs and separated by spaces.
For example, to run only the default
and media
queues with a limit of 10 and 5 respectively, you would pass the string default,5 media,10
.
Note that the parsing clause has a couple of safety mechanisms to ensure that only real queues are specified:
default,3
)queue
string to an existing atom, hopefully preventing typos that would start a random queue (i.e. defalt
)In development (or when using mix
rather than releases) we can specify the environment variable inline:
OBAN_QUEUES="default,10 media,5" mix phx.server # default: 10, media: 5
We can also explicitly opt in to running all of the configured queues:
OBAN_QUEUES="*" mix phx.server # default: 15, media: 10, events: 25
Finally, without OBAN_QUEUES
set at all it will implicitly fall back to the configured queues:
mix phx.server # default: 15, media: 10, events: 25
This environment variable based solution is more flexible than running separate umbrella apps because we can reconfigure at any time. In a limited environment, like staging, we can run all the queues on a single node using the exact same code we use in production. In the future, if other workers start to utilize too much CPU or RAM we can shift them to the worker node without any code changes.
This recipe was prompted by an inquiry on the Oban issue tracker.
Reporting job errors by sending notifications to an external service is essential to maintaining application health. While reporting is essential, noisy reports for flaky jobs can become a distraction that gets ignored. Sometimes we expect that a job will error a few times. That could be because the job relies on an external service that is flaky, because it is prone to race conditions, or because the world is a crazy place. Regardless of why a job fails, reporting every failure may be undesirable.
One solution for reducing noisy error notifications is to start reporting only after a job has failed several times. Oban uses Telemetry to make reporting errors and exceptions a simple matter of attaching a handler function. In this example we will extend Honeybadger reporting from the Oban.Telemetry documentation, but account for the number of processing attempts.
To start, we'll define a Reportable
protocol with a single reportable?/2
function:
defprotocol MyApp.Reportable do
@fallback_to_any true
def reportable?(worker, attempt)
end
defimpl MyApp.Reportable, for: Any do
def reportable?(_worker, _attempt), do: true
end
The Reportable
protocol has a default implementation which always returns true
, meaning it reports all errors.
Our application has a FlakyWorker
that's known to fail a few times before succeeding.
We don't want to see a report until after a job has failed three times, so we'll add an implementation of Reportable
within the worker module:
defmodule MyApp.FlakyWorker do
use Oban.Worker
defimpl MyApp.Reportable do
@threshold 3
def reportable?(_worker, attempt), do: attempt > @threshold
end
@impl true
def perform(%{email: email}) do
MyApp.ExternalService.deliver(email)
end
end
The final step is to call reportable?/2
from our application's error reporter, passing in the worker module and the attempt number:
defmodule MyApp.ErrorReporter do
alias MyApp.Reportable
def handle_event(_, _, %{attempt: attempt, worker: worker} = meta, _) do
if Reportable.reportable?(worker, attempt)
context = Map.take(meta, [:id, :args, :queue, :worker])
Honeybadger.notify(meta.error, context, meta.stack)
end
end
end
Attach the failure handler somewhere in your application.ex
module:
:telemetry.attach("oban-errors", [:oban, :failure], &ErrorReporter.handle_event/4, nil)
With the failure handler attached you will start getting error reports only after the third error.
If a service is especially flaky you may find that Oban's default backoff strategy is too fast.
By defining a custom backoff
function on the FlakyWorker
we can set a linear delay before retries:
# inside of MyApp.FlakyWorker
@impl true
def backoff(attempt, base_amount \\ 60) do
attempt * base_amount
end
Now the first retry is scheduled 60s
later, the second 120s
later, and so on.
Elixir's powerful primitives of behaviours, protocols and event handling make flexible error reporting seamless and extendible.
While our Reportable
protocol only considered the number of attempts, this same mechanism is suitable for filtering by any other meta
value.
Explore the event metadata that Oban provides for job failures to see how you can configure reporting by by worker, queue, or even specific arguments.
In the previous post we looked at tracking the progress of a single job as it executes. What about tracking the progress of tens, hundreds or thousands of jobs as they execute? In that situation we want to monitor the jobs as a group—execute them in parallel and then enqueue a callback when all the jobs are finished. At least one popular background job processor calls these groups "batches", and so we'll adopt that term here as we build it out with Oban.
Admins on our site send weekly batch emails to a large mailing list to let users know new content is available. Naturally the system sends emails in parallel in the background. Delivery can take many hours and we want to notify our admins when the batch is complete. This is an admittedly simple use case, but it is just complex enough to benefit from a batching flow.
At a high level, the worker flow looks like this:
batch_size
, which we'll use later to decide whether all jobs have completed.perform/2
clause matching a batch_id
key.
This clause will handle the real work that the job is meant to do, and afterwards it will start a separate process to check whether the batch is complete.
Since executed jobs are stored in the database with a completed
state we can evaluate whether this was the final job in the batch.Here is the worker module with both the primary and callback clauses of perform/2
:
defmodule MyApp.BatchEmailWorker do
use Oban.Worker, queue: :batch, unique: [period: 60]
import Ecto.Query
@final_check_delay 50
@impl true
def perform(%{"email" => email, "batch_id" => batch_id, "batch_size" => batch_size}, _job) do
MyApp.Mailer.weekly_update(email)
Task.start(fn ->
Process.sleep(@final_check_delay)
if final_batch_job?(batch_id, batch_size) do
%{"status" => "complete", "batch_id" => batch_id}
|> new()
|> Oban.insert()
end
end)
end
def perform(%{"status" => "complete", "batch_id" => batch_id}, _job) do
MyApp.Mailer.notify_admin("Batch #{batch_id} is complete!")
end
end
Within the first perform/2
clause we deliver a weekly update email and then start a separate task to check whether this is the final job.
The task is not linked to the job and it uses a short sleep to give enough time for the job to be marked complete; the goal is to prevent race conditions where no calback is ever enqueued.
The final_batch_job?/2
function is wrapper around a fairly involved Ecto query:
defp final_batch_job?(batch_id, batch_size) do
Oban.Job
|> where([j], j.state not in ["available", "executing", "scheduled"])
|> where([j], j.queue == "batch")
|> where([j], fragment("?->>'batch_id' = ?", j.args, ^batch_id))
|> where([j], not fragment("? \\? 'status'", j.args))
|> select([j], count(j.id) >= ^batch_size)
|> MyApp.Repo.one()
end
This private predicate function uses the Oban.Job struct to query the oban_jobs
table for other completed jobs in the batch.
Within the query we use a fragment containing the indecipherable ->>
operator, a native PostgreSQL jsonb operator that keys into the args
column and filters down to jobs in the same batch.
The equally indecipherable existence operator (\\?
), which must be double escaped within a fragment, helps to ensure that we aren't creating duplicate callback jobs.
When the number of completed or discarded jobs matches our expected batch size we know that the batch is complete!
It's worth mentioning at this point that by default there aren't any indexes on the args
column, so this query won't be super snappy with a lot of completed jobs laying around.
If you plan on integrating batches into your workflow, and you want to ensure that callback jobs are absolutely unique, you should add a unique index on batch_id
, and possibly one for the status
argument.
To kick off our batch job we generate a batch_id
and a iterate through a list of emails:
batch_id = "email-blast-#{DateTime.to_unix(DateTime.utc_now())}"
batch_size = length(emails)
for email <- emails do
%{email: email, batch_id: batch_id, batch_size: batch_size}
|> Oban.BatchEmailWorker.new()
|> Oban.insert!()
end
This batching technique is possible without any other tables or tracking mechanisms because Oban's jobs are retained in the database after execution. They're stored right along with your other production data, which opens them up to querying and manipulating as needed. Batching isn't built into Oban because between queries and pattern matching you have everything you need to build complex batch pipelines.
One final note: querying for completed batches all hinges on how aggressive your pruning configuration is. If you're pruning completed jobs after a few minutes or a few hours then there is a good chance that your batch won't ever complete. Be sure that you tune your pruning so that there is enough headroom for batches to finish.
Most applications provide some way to generate an artifact—something that may take the server a long time to accomplish. If it takes several minutes to render a video, crunch some numbers or generate an export, users may be left wondering whether your application is working. Providing periodic updates to end users assures them that the work is being done and keeps the application feeling responsive.
Reporting progress is something that any background job processor with unlimited execution time can do! Naturally, we'll look at an example built on Oban.
Users of our site can export a zip of all the files they have uploaded. A zip file (no, not a tar, our users don't have neck-beards) is generated on the fly, when the user requests it. Lazily generating archives is great for our server's utilization, but it means that users may wait a while when there are many files. Fortunately, we know how many files will be included in the zip and we can use that information to send progress reports! We will compute the archive's percent complete as each file is added and push a message to the user.
In the forum question that prompted this post the work was done externally by a port process. Working with ports is well outside the scope of this post, so I've modified it for the sake of simplicity. The result is slightly contrived as it puts both processes within the same module, which isn't necessary if the only goal is to broadcast progress. This post is ultimately about coordinating processes to report progress from a background job, so that's what we'll focus on (everything else will be rather hand-wavy).
Our worker, the creatively titled ZippingWorker
, handles both building the archive and reporting progress to the client.
Showing the entire module at once felt distracting, so we'll start with only the module definition and the perform/2
function:
defmodule MyApp.ZippingWorker do
use Oban.Worker, queue: :exports, max_attempts: 1
def perform(%{"channel" => channel, "paths" => paths}, _job) do
build_zip(paths)
await_zip(channel)
end
end
The function accepts a channel name and a list of file paths, which it immediately passes on to the private build_zip/1
:
defp build_zip(paths) do
job_pid = self()
Task.async(fn ->
zip_path = MyApp.Zipper.new()
paths
|> Enum.with_index(1)
|> Enum.each(fn {path, index} ->
:ok = MyApp.Ziper.add_file(zip_path, path)
send(job_pid, {:progress, trunc(index / length(paths) * 100)})
end)
send(job_pid, {:complete, zip_path})
end)
end
The function grabs the current pid, which belongs to the job, and kicks off an async task to handle the zipping.
With a few calls to a fictional Zipper
module the task works through each file path, adding it to the zip.
After adding a file the task sends a :progress
message with the percent complete back to the job.
Finally, when the zip finishes, the task sends a :complete
message with a path to the archive.
The async call spawns a separate process and returns immediately.
In order for the task to finish building the zip we need to wait on it.
Typically we'd use Task.await/2
, but we'll use a custom receive loop to track the task's progress:
defp await_zip(channel) do
receive do
{:progress, percent} ->
MyApp.Endpoint.broadcast(channel, "zip:progress", percent)
await_zip()
{:complete, zip_path} ->
MyApp.Endpoint.broadcast(channel, "zip:complete", zip_path)
after
30_000 ->
MyApp.Endpoint.broadcast(channel, "zip:failed", "zipping failed")
raise RuntimeError, "no progress after 30s"
end
end
The receive loop blocks execution while it waits for :progress
or :complete
messages.
When a message comes in it broadcasts to the provided channel and the client receives an update (this example uses Phoenix channels, but any other pubsub type mechanism would work).
As a safety mechanism we have an after
clause that will timeout after 30 seconds of inactivity.
If the receive block times out we notify the client and raise an error, failing the job.
Reporting progress asynchronously works in Oban because anything that blocks a worker's perform/2
function will keep the job executing.
Jobs aren't executed inside of a transaction, which alleviates any limitations on how long a job can run.
This technique is suitable for any single long running job where an end user is waiting on the results. Next time we'll look at combining multiple jobs into a single output by creating batch jobs.
A common variant of recursive jobs are "scheduled jobs", where the goal is for a job to repeat indefinitely with a fixed amount of time between executions.
The part that makes it "reliable" is the guarantee that we'll keep retrying the job's business logic when the job retries, but we'll only schedule the next occurrence once.
In order to achieve this guarantee we'll make use of a recent change in Oban that allows the perform
function to receive a complete Oban.Job
struct.
Time for illustrative example!
When a new user signs up to use our site we need to start sending them daily digest emails. We want to deliver the emails around the same time a user signed up every, repeating every 24 hours. It is important that we don't spam them with duplicate emails, so we ensure that the next email is only scheduled on our first attempt.
defmodule MyApp.ScheduledWorker do
use Oban.Worker, queue: :scheduled, max_attempts: 10
@one_day 60 * 60 * 24
@impl true
def perform(%Oban.Job{attempt: 1, args: args}) do
args
|> new(schedule_in: @one_day)
|> Oban.insert!()
perform(args)
end
def perform(%{"email" => email}) do
MyApp.Mailer.deliver_email(email)
end
end
You'll notice that the first perform/1
clause only matches a job struct on the first attempt.
When it matches, the first clause schedules the next iteration immediately, before attempting to delver the email.
Any subsequent retries fall through to the second perform/1
clause, which only attempts to deliver the email again.
Combined, the clauses get us close to at-most-once semantics for scheduling, and at-least-once semantics for delivery.
The interesting thing that is happening here is that perform/1
can handle either an Oban.Job
struct, or the args map directly.
This is possible because of a "before compile" module hook in the Oban.Worker
module.
Below is a simplified version of the worker module with extraneous code removed to emphasize the @before_compile
hook:
defmacro __before_compile__(_env) do
quote do
def perform(%Job{args: args}), do: perform(args)
end
end
defmacro __using__(opts) do
quote location: :keep do
@before_compile Oban.Worker
end
end
When your module uses Oban.Worker
it includes the args extraction clause in the compiled module before your definition of perform/1
.
For example, if your worker defines a perform
clause to work with an email address there would be two compiled clauses:
def perform(%{email: email}), do: work_with_email(email)
def perform(%Job{args: args}), do: perform(args)
The additional clause ensures that your perform can accept either a struct or the args map interchangeably.
Delivering around the same time using cron-style scheduling would need extra book-keeping to check when a user signed up, and then only deliver to those users that signed up within that window of time. The recursive scheduling approach is more accurate and entirely self contained—when and if the digest interval changes the scheduling will pick it up automatically once our code deploys.
Next time, for something completely different, we'll see how to report progress back to our users as a slow job executes.
An extensive discussion on the Oban issue tracker prompted this example along with the underlying feature that made it possible.
This recipe is now a pack of white lies!
The gist of the recipe is still intact, but the examples and the before_compile
details aren't accurate.
This post prompted an issue on the tracker that suggested replacing the args dance in perform/1
with a consistent perform/2
function instead.
The new perform/2
always accepts an args map as the first argument and the complete job struct as the second.
Here is the worker example from above, slightly modified to use perform/2
:
defmodule MyApp.ScheduledWorker do
use Oban.Worker, queue: :scheduled, max_attempts: 10
@one_day 60 * 60 * 24
@impl true
def perform(%{"email" => email}, %{attempt: 1} = job) do
args
|> new(schedule_in: @one_day)
|> Oban.insert!()
MyApp.Mailer.deliver_email(email)
end
def perform(%{"email" => email}, _job) do
MyApp.Mailer.deliver_email(email)
end
end
The upcoming 0.7.0
release will include the perform/2
changes.
Recursive jobs, like recursive functions, call themselves after they have have executed. Except unlike recursive functions, where recursion happens in a tight loop, a recursive job enqueues a new version of itself and may add a slight delay to alleviate pressure on the queue.
Recursive jobs are a great way to backfill large amounts of data where a database migration or a mix task may not be suitable. Here are a few reasons that a recursive job may be better suited for backfilling data:
Let's explore recursive jobs with a use case that builds on several of those reasons.
Consider a worker that queries an external service to determine what timezone a user resides in. The external service has a rate limit and the response time is unpredictable. We have a lot of users in our database missing timezone information, and we need to backfill.
Our application has an existing TimezoneWorker
that accepts a user's id
, makes an external request and then updates the user's timezone.
We can modify the worker to handle backfilling by adding a new clause to perform/1
.
The new clause explicitly checks for a backfill
argument and will enqueue the next job after it executes:
defmodule MyApp.TimezoneWorker do
use Oban.Worker
import Ecto.Query
@backfill_delay 1
def perform(%{id: id, backfill: true}) do
with :ok <- perform(%{id: id}),
next_id when is_integer(next_id) <- fetch_next(id) do
%{id: next_id, backfill: true}
|> new(schedule_in: @backfill_delay)
|> MyApp.Repo.insert!()
end
end
def perform(%{id: id}) do
update_timezone(id)
end
defp fetch_next(current_id) do
MyApp.User
|> where([u], is_nil(u.timezone))
|> order_by(asc: :id)
|> limit(1)
|> select([u], u.id)
|> MyApp.Repo.one()
end
end
There is a lot happening in the worker module, so let's unpack it a little bit.
perform/1
, the first only matches when a job is marked as backfill: true
, the second does the actual work of updating the timezone.fetch_next/1
to look for the id of the next user without a timezone.With the new perform/1
clause in place and our code deployed we can kick off the recursive backfill.
Assuming the id
of the first user is 1
, you can start the job from an iex
console:
iex> %{id: 1, backfill: true} |> MyApp.TimezoneWorker.new() |> MyApp.Repo.insert()
Now the jobs will chug along at a steady rate of one per second until the backfill is complete (or something fails). If there are any errors the backfill will pause until the failing job completes: especially useful for jobs relying on flaky external services. Finally, when there aren't any more user's without a timezone, the backfill is complete and recursion will stop.
This was a relatively simple example, and hopefully it illustrates the power and flexibility of recursive jobs.
Recursive jobs are a general pattern and aren't specific to Oban.
In fact, aside from the use Oban.Worker
directive there isn't anything specific to Oban in the recipe!
In the next recipe we'll look at a specialized use case for recursive jobs: infinite recursion for scheduled jobs.
On the Oban issue tracker and in the Elixir forum I'm frequently asked how to accomplish certain tasks. While some tasks, like the aforementioned after-sign-up email delivery are straight forward, other jobs are tricky to coordinate and may require knowledge of how Oban composes. This recipe, and the others to follow, apply to background job processing in general, but this post will look at accomplishing a task using Oban.
Enough preamble, on to the recipe!
Preventing duplicate jobs ensures that the same background work isn't done multiple times.
For instance, perhaps you have a job that emails a link whenever a form is submitted.
If the form is submitted twice you don't want to insert a second job while the first is still pending.
Note that "pending" means jobs that are scheduled
or available
to execute, it does not apply to jobs that have already completed.
It is certainly possible to enforce uniqueness within a window of time, but we won't focus on that for this example.
With Oban, unlike most background job libraries, you have complete control over how jobs are inserted into the database. That control gives us a few options where the trade-off is between strong guarantees and convenience.
The first solution is the most robust, as it leverages PostgreSQL's uniqueness guarantees. Namely, we can use partial indexes to scope the uniqueness to a particular worker.
We start by adding an index for the worker that we want to enforce uniqueness for.
Add the following declaration to a new Ecto migration, where MyApp.Worker
is the name of the worker you want to enforce uniqueness for:
create index(
:oban_jobs,
[:worker, :args],
unique: true,
where: "worker = 'MyApp.Worker' AND state IN ('available', 'scheduled')"
)
The composite index is unique, but the where
clause ensures that the index only applies to new jobs.
With the index defined we can make use of Ecto's ON CONFLICT support when inserting a job.
Using ON CONFLICT DO NOTHING
tells PostgreSQL to try and insert the new row, but that it's alright if insertion fails:
%{email: "somebody@example.com"}
|> MyApp.Worker.new()
|> MyApp.Repo.insert(on_conflict: :nothing)
This solution is efficient and has transactional guarantees, but it isn't particularly convenient. Every time we want to add a unique constraint, or modify an existing constraint to take other fields into account, we need to run a database migration. There is an application based alternative that doesn't have such strong guarantees, but is far more flexible.
The second solution is to define a helper function within your application's Repo
module.
The helper checks the database for jobs before trying to insert, and aborts insertion if anything is found:
def insert_unique(changeset, opts \\ []) do
worker = Ecto.Changeset.get_change(changeset, :worker)
args = Ecto.Changeset.get_change(changeset, :args)
Oban.Job
|> where([j], j.state in ~w(available scheduled))
|> get_by(worker: worker, args: args)
|> case do
nil -> insert(changeset, opts)
_job -> {:ignored, changeset}
end
end
Now, where you would have used Repo.insert/1
before you can use Repo.insert_unique/2
instead:
%{email: "somebody@example.com"}
|> MyApp.Worker.new()
|> MyApp.Repo.insert_unique()
The helper can be used freely for any worker without the need to deploy new indexes to your database. Unlike the previous solution, this one has no transactional guarantees and doesn't have any indexes to work off of, so it may be slower.
Supporting unique jobs purely within your own application shows how much direct control you have over queueing behavior. The forum post this recipe is based on is over a month old now, and subsequent discussion on the issue tracker proved that this is a generally desirable feature. The use case for unique jobs is broad enough that it makes a great candidate for inclusion directly within Oban. When unique job support makes its way into Oban I'll update this post with a third official technique.
Official unique job support has landed in master and will be included in the upcoming 0.7.0
release.
The implementation somewhat matches the "Insert Helper" solution above, but more dynamic and configurable.
The example given above could be achieved like this, at the job level:
%{email: "somebody@example.com"}
|> MyApp.Worker.new()
|> Oban.insert(unique: [states: [:available, :scheduled]])
Or, it could be declared at the worker level directly:
defmodule MyApp.Worker do
use Oban.Worker, unique: [states: [:available, :scheduled]]
end
There are more features and a lot more documentation available in the Oban
documentation!
If you're curious how support was added you can take a look at the pull request on GitHub.
There are at least three discrete strategies to solve the issue of expiring targeted data. Which one to use depends on how the keys have been composed, how the data was generated, and exactly what has become stale.
When the cached data is referencing a database record, and has a key that is
based on the timestamp of a record, you can touch
it to bust the cache. The
next time the data is fetched the key will have expired and you'll get fresh
data. For more in depth information on key composition see essentials of cache
expiration.
With a narrow collection of records touching is easy and targeted. Large
collections, hybrid caches of multiple models or unbounded range (like every
record in the database) are not well suited to purging via touching. When the
situation is right you can use methods like touch
or update_all
in Rails to
bump the timestamp on one or more records:
child.touch # touch a single record
parent.children.update_all(updated_at: Time.now) # touch all records
Occasionally the data in the cache is fresh, but you need a different view of it. Maybe an API client requires a new field, fewer fields, or more associated records. In this case there isn't any point in touching the records. Views and serializers need to be updated, which is your opportunity to bundle the expiration. This is a job for targeted versioning.
Note the word targeted is being used. It is possible to uniformly version
the entire cache by updating the namespace. Much like changing an API from v1
to v2
, you prepend the cache with a version. Targeted versioning is similar,
but the version change is scoped to the view or serializer in question. For
caching within a Rails view this is as simple as composing the key from an array
rather than just the model. For example:
cache [model, 'v2'] do
# fragment to cache
end
cache [model, 'v3'] do
# new fragment to cache
end
Recently, on a client project, a situation arose where a large section of the cache had to be purged, but neither touching nor versioning would work.
Their application caches large trees of API data, many parts of which contain
embedded user data. The embedded user data includes a few avatar URLs, all of
which were securely checked out from Amazon S3 and have an expiration. A
background job keeps the URLs refreshed, but that hadn't been accounted for in
the cache. The end result was a lot of 403 Forbidden
requests when the browser
tried to load the embedded expired avatars.
Touching won't help here, because the user record isn't cached directly, and they aren't part of the cache key for the parent record. Versioning isn't well suited either, as the fields don't need to change, the underlying data is out of sync. That's a lot of wind up for what I'm about to suggest: delete the exact keys that have expired.
Avoid purging the entire cache by using a targeted tool like
delete_matched, as provided by ActiveSupport::Cache
. Most of the
available caches support matching using regular expressions, though some only
support globbing.
Rails.cache.delete_matched("posts/9[0-1]*")
Note: Until very recently my readthis cache for Redis didn't
support delete_matched
due to concerns about performance and the evil keys
command. The eventual implementation uses SCAN
and is entirely safe to
use with gigantic databases. The aforementioned client was using Readthis for
caching, driving the need for delete_matched
to be implemented.
Caching is key to a highly performant application, but stale data can be insidious. Without targeted expiration we start to reach for blunt tools and expire too broadly. All of the expiration strategies presented here are simple, and they come up often in a production system. Recognize the situation and choose the right strategy for the job.
]]>Rails comes with caching built in. It is available anywhere you can reference
Rails
, and as the cache
method within every controller and view. You don't
have to look far to access the cache. The real work is deciding what to put in
the cache, and how to keep it fresh. A stale cache gives users the sense that
the system is slow or broken, and we don't want that. This is where expiration
comes in. How do we ensure old data is expired?
Broadly, there are only three approaches to cache expiration. Each has overlapping use cases and varying granularity:
MemoryStore
only retains values as long as
the parent process is alive. Once the process stops the memory holding cached
values is released. For applications with data that changes less frequently
than the application restarts this is a viable option.Regardless of how values are expired they are always referenced by a key. It all comes down to keys. The fundamental cache operation is fetch, which checks the cache store for the existence of a key and returns the value it finds. If the key isn't found it then generates the value, writes it to the store, and then returns the new value.
The way cache keys are composed is critical to how effectively values are expired. Compose a key too broadly and the cache is busted more often than it should be, making it inefficient. A well composed key, or set of keys, smoothes everything out. Let's look at some recipes for cache key composition, and which situations they work for.
Cache keys are built from unique segments which change when the data they reference changes. Be aware that the precise structure of the key isn't important. The order of segments and how they are separated is inconsequential, so long as they combine into a unique value.
:class/:id -> Post/1
The most basic cache key structure possible, only a model name and the id
. The
parts represent the bare information needed to retrieve a cached value. This
type of key can only be expired by time or a full purge, making it of limited
value in production systems.
:class/:id-:timestamp -> Post/1-1468239686
A timestamped key appends a model's updated_at
timestamp in milliseconds. When
the model is updated or touched, the cache is expired. This is what is
generated for ActiveRecord
models out of the box. How records are touched, and
whether they touch associated records, is critical to proper cache
expiration. In the world of ActiveRecord
it's alright to touch yourself, touch
your friends, and touch your friend's friends.
:class_plural/:last_updated/:count -> Posts/1468239686/8
When all of the values in a collection can be cached together you have to consider more than the ids. Instead, a collection's cache key combines the timestamp of the most recently updated model in the collection along with the collection length. If any model within the collection is updated or one is deleted the cache is expired.
:class/:id/:view/:checksum -> Posts/1/SomeView/a8b56bb
This includes the name of a view that generated the value and a checksum of the view at the time the value was created. This guarantees that when new fields or markup is added to the view that the cache will be expired.
:class/:id/:role -> Post/1/staff
The role of the current user, or "scope", is appended. In this example, staff see different values than regular users. Appending the scope prevents sensitive data from leaking to regular users. It also guarantees that both values can be cached and served up independently.
:class/:id/:user_id -> Post/1/123
Appending the user_id
generates a new cache entry for each user. Generally
this is undesirable—the cache can't be shared at all, defeating most of the
reason you are trying to cache things. However, in a system with a limited
number of users where data is extremely expensive to generate, this is a viable
option.
Caching is critical to high performance Rails applications. The most fundamental part of effective caching is expiration, and that hinges almost exclusively on cache keys. Understanding how to compose keys for different use cases, manage key based expirations, and keep keys finely scoped is all it takes to master caching. Expiration is often maligned as being hard, but there isn't much to it. A sliver of knowledge will get you a long way.
]]>On most days they could only let Sidekiq run for a few hours at a time before the system would break down into constant errors. Some of errors compounded so badly it ground the web server to a halt, taking down the entire site. In an attempt at exerting control, all onboarding jobs were started manually rather than automatically. There was no trust in the system, and no obvious way to scale their way out of the situation.
After brief investigation it was clear that there were fundamental problems with the way jobs were enqueued, distributed, and limited. What follows are the high level changes that were implemented, each presented as an observation and a solution.
Various categories of jobs were bucketed into different queues, but the queues weren't weighted. This forced each queue to drain completely before jobs in the next queue were started. Queues are processed in the order they are listed, so piling thousands of slow running jobs onto a queue in the middle guarantees inefficient processing. A mass of the same slow job running simultaneously just exacerbates resource contention and prevents jobs in the subsequent queues from ever starting.
Declare equal weights for each of the queues so that jobs are plucked randomly between them. That forces fast jobs from the default queue to run alongside slow jobs. Spreading busy jobs between queues becomes even more important when there are limits on how many of each job type can run concurrently.
Force random queue priorities:
sidekiq -q google,1 -q facebook,1 -q linkedin,1 -q twitter,1 -q default,1
Learn more about advanced queuing in the Sidekiq Wiki.
The application's database drivers for Neo4j used HTTPS as a transport.
Instead of connection pooling, it executed every request across a single
long-standing NetHTTPPersistent
connection. When concurrent jobs deadlocked or
threw errors the thread stopped waiting for a response but kept holding the
connection. With one or two errors in quick succession the connection would
recover, but with rapid fire errors the connection problems quickly choked out
the database.
The ultimate solution would be to use a more efficient transport than HTTP, and to institute connection pooling. However, that would require upstream library changes and would take far longer than the client had. The temporary fix was to lean on processes for concurrency rather than threads. With fewer threads per process there was less of a drain on the single connection and it was easy to keep it healthy.
On a platform like Heroku it is simply a matter of scaling the number of worker
instances. On a host that uses Upstart to manage Sidekiq workers it is as simple
as bumping NUM_WORKERS
within the workers conf. Even better, with
Enterprise you can make use of managed multi process using swarm.
env COUNT=4
exec bundle exec sidekiqswarm -e production
Many of the same type of job ran concurrently and contended for the same database or network resources. In an attempt to prevent clobbering the output of each job all of the records were being pessimistically locked, leading to database deadlocks. Some deadlocks would resolve themselves, but any that didn't slowed the queue to a crawl, caused unpredictable errors, and put an extra burden on the throttling constructs.
Impose limits on the number of concurrent jobs that can run at once. With a single process and strictly segregated jobs it is trivial to control the concurrency by limiting a queue. But what happens when there is more work than a single process can handle (or a poorly behaving database driver forces you to parallelize with processes)? Now there are as many queues as there are processes, and any limits that were being enforced has scaled right along with them.
The proper solution is to use a distributed concurrency construct like the throttling available from Sidekiq Enterprise. The throttle enforces job concurrency limits across all threads, processes, and hosts; ensuring that at most N jobs can run at a time. Be aware that throttling operates at the job level, not at the queue level. For example, with a throttle that limits one job at a time and concurrency set to 25 Sidekiq will still start 25 of the same job type, but each job will wait for the lock to release and they will run sequentially. That makes it crucial to balance queues evenly so that multiple job types are enqueued simultaneously.
Configure a concurrent rate limiter that only permits 2 concurrent jobs, with generous timeouts:
LINKEDIN_THROTTLE = Sidekiq::Limiter.concurrent(
'linkedin',
2,
wait_timeout: 10,
lock_timeout: 60
)
def perform
LINKEDIN_THROTTLE.within_limit do
# Talk to LinkedIn
end
end
See the details of concurrent rate limiting at the Wiki.
Happily the workers are now plowing through 300,000+
jobs a day without any
downtime or hiccups.
None of these changes by themselves were enough to get the system running smoothly. It required a healthy amount of defensive coding, bug fixes, and configuration tuning to smooth out the platform. It may have been possible without the industrial strength features offered by Enterprise, but it would have required a lot of plugins and some hand rolled throttling. I didn't even mention using batches, unique jobs or time based rate limiting—used properly, Pro/Enterprise save tremendous amounts of developer time.
No, I don't receive kickbacks on Enterprise sales!
]]>For months I'd been watching Skylight performance metrics for a couple of critical API endpoints. The response times weren't great, moreover they were highly unpredictable. The endpoint had some intensive caching, but it fell flat whenever the cache wasn't warm. When the cache was warm it was still plagued by massive object allocations and frequent GC pauses. These are essential API endpoints, serving hundreds of thousands of requests a day. It had to get better.
I've been down this road before with a library called Perforated. The idea behind Perforated is simple, only cache the parts of a collection that need to be recomputed and stitch the serialized values back together. To that end, Perforated worked very well. What Perforated lacked were some crucial optimizations to reduce initial object allocation and provide flexibility. The architecture of Perforated wasn't composable enough for optimizations to be added on. It was simply too hard to isolate and instrument each part of the serialization process. It begged for a rewrite.
The successor to Perforated is called Knuckles. If that name is confusing, just know that "Sonic" was already taken. It extends the functionality of Perforated by adding crucial features like cache customization, full instrumentation, and an integrated view module. Thanks to rigorous profiling and benchmarking it also crushes on performance.
Setting a few goals from the outset helped make design decisions and kept the project focused. The library is meant to be as fast and lightweight as possible, which pushed back on feature creep. Here are some of the highest impact decisions along with the results they yielded.
Personalization is a caching roadblock. In a typical system you can't cache a payload when the content is customized for the current requester. That results in an untenable one-cache-entry-per-user situation, which isn't useful.
Knuckles breaks the cache process down into discrete stages of a functional
pipeline. Each stage aids in reducing down to a final serialized payload. Stages
can be removed from the pipeline, or new ones can be added. For example, to
handle the personalization conundrum you simply insert an enhancer
stage that
augments the payload with the current user's information. If a resource is being
served to users with differing privileges then the customizer step can prune
sensitive information before it is served up.
As an example, imagine rendering content for staff and regular users. The only difference is that staff can see more fields. In this situation, you would cache everything and then prune the final payload when the request isn't from staff:
module Knuckles
module Enhancers
module StaffEnhancer
STAFF_ONLY = %w[bookmarks notes tags].freeze
def self.call(rendered, options)
scope = options[:scope]
unless scope.staff?
rendered.delete_if { |key, _| STAFF_ONLY.include?(key.to_s) }
end
rendered
end
end
end
end
Now the endpoint can be fully cached and serve multiple roles efficiently. Using this technique, or the opposite wherein personal content is appended to the payload, there are no limitations on how personalized cached content can be.
Rampant object allocations are a massive performance killer for any application. Unrestricted object creation puts a strain on the garbage collector, hurting random unrelated requests. A key to the design of Knuckles was promoting patterns where fewer objects were allocated. Just how many fewer objects? Take a look at the chart below. These numbers are for the same sizable endpoint running in production, fully cached.
The retained value for Knuckles is 136 objects, so low that it isn't even on the chart. These drastic reductions in object allocation stem from three places:
Having spent the past half a year writing Elixir I've come to strongly favor
explicit code over DSLs. That's why the serializer that comes with Knuckles,
called a view
only uses three methods to construct serialized data.
PostView = Module.new do
extend Knuckles::View
def self.root
:posts
end
def self.data(post, _)
{id: post.id, title: post.title, tag_ids: post.tags.map(&:id)}
end
def self.relations(post, _)
{tags: has_many(post.tags, TagView)}
end
end
All of the data structures, keys, and values must be stated explicitly. There isn't any surprising pluralization, or obscure incantations to sideload an association rather than embed it. Views are very simple and designed to stay out of your way.
There is a built in stage for ActiveModelSerializers
compatibility, if you're
coming to Knuckles from an existing system. In fact, that's how it was rolled
out to production initially.
It's been several months since Knuckles hit production. I can proudly say that the 95th percentile response times jumped 4-5x, with warm requests coming back in ~31ms or less. Undoubtedly there were stumbling points while folding Knuckles in, but the final transition was seamless.
If you're running an app with lagging API endpoints, or endpoints you wish you could cache, give Knuckles a try.
]]>The PubSub pattern is a fundamental tool for distributed computing. This pair of articles cover the basics of PubSub, using the pattern with Redis in Ruby, and finally how to build an alternative to Rail's ActionCable with Elixir.
Alternative Service Communication Using PubSub Surrogate WebSockets Through Elixir
]]>This is a reinterpretation of Folding Window Functions Into Rails, rewritten and adapted from ActiveRecord to Ecto 2.0. The results were unexpected...
Perhaps you've heard of window functions in PostgreSQL, but you aren't quite sure what they are or how to use them. On the surface they seem esoteric and their use-cases are ambiguous. Something concrete would really help cement when window functions are the right tool for the job. That's precisely what we'll explore in this post:
You've recently finished shipping a suite of features for an application that helps travelers book golf trips. Things are looking good, and a request comes in from your client:
Our application started by being the go-to place to find golf trips, and our users love it. Some of the resorts that list trips with us also offer some non-golf events, such as tennis, badminton, and pickleball. When we begin listing other trips it would be great to highlight our user's favorite trips for each category. Can you do that for us?
—Anonymous Client
Why, of course you can do that!
The application lets potential traveler's flag trips they are interested in as favorites, providing a reliable metric that we can use to rank trips.
With the simple addition of a category
for each trip we can also filter or group trips together.
This seems straight forward enough...
A look at the Trip
schema reveals that it currently has these relevant fields: name
, category
, and favorites
.
defmodule Triptastic.Trip do
use Ecto.Schema
@categories ~w(golf tennis badminton pickleball)
schema "trips" do
field :name, :string
field :category, :string
field :favorites, :integer, default: 0
end
def categories, do: @categories
end
Instead of listing all of the top ranked trips we'll only show the top two trips in each category. Some tests will help verify that we're getting the expected results.
defmodule Triptastic.TripRepoTest do
use ExUnit.Case
alias Triptastic.{Repo, Trip}
setup do
:ok = Ecto.Adapters.SQL.Sandbox.checkout(Triptastic.Repo)
trips = for category <- Trip.categories, favorites <- 0..5 do
%{name: "#{category}-#{favorites}",
category: category,
favorites: favorites}
end
Repo.insert_all(Trip, trips)
:ok
end
test "grouping top trips by category" do
trips = Trip |> Repo.all() |> Trip.popular_by_category()
assert length(trips) == 8
assert Enum.all?(trips, &(&1.favorites > 2))
end
end
The test seeds the test database with trips across four categories with a varying number of favorites.
The popular_by_category/2
function expects a list of trips and returns the most popular two from each category.
Initially we'll approach this with pure naive Elixir.
All of the trips are loaded into memory, grouped by category, ranked according to the number of favorites, and then the requested per
amount is taken off of the top.
Do note that sorting is comprised of both favorites and name, which is necessary to force deterministic sorting in the likely event that trips are equally popular.
# Defined within the Triptastic.Trip module shown above
def popular_by_category(trips, per \\ 2) do
trips
|> Enum.group_by(&(&1.category))
|> Enum.flat_map(&(popular_in_subset(&1, per)))
end
defp popular_in_subset({_category, trips}, per) do
trips
|> Enum.sort_by(&([-&1.favorites, &1.name]))
|> Enum.take(per)
end
As a wizened developer you immediately recognize that loading every trip into memory simply to retrieve eight results is rather inefficient.
It makes fine use of the Enum
module and some piping, but it isn't suitable for production usage.
Between various sub-selects, GROUP BY
with aggregates and multiple queries, there are many ways to manipulate the trips data in SQL.
One advanced feature of PostgreSQL that is particularly adept at solving this categorization problem are window functions.
Directly from the documentation:
A window function performs a calculation across a set of table rows that are somehow related to the current row.
The key part of the phrase is the power of calculating across related rows.
In our case, the rows are related by category, and the calculation being performed is ordering them within those categories.
In the realm of window functions this is handled with an OVER
clause.
There are additional expressions for fine tuning the window, but for now we can achieve all we need with PARTITION BY
and ORDER BY
expressions.
Dropping into psql
, let's see how to partition the data set by category:
SELECT category, favorites, row_number() OVER (PARTITION BY category) FROM trips;
category | favorites | row_number
------------+-----------+-------------
badminton | 0 | 1
badminton | 1 | 2
badminton | 2 | 3
badminton | 3 | 4
golf | 0 | 1
golf | 1 | 2
The row_number
is a window function that calculates number of the current row within its partition.
Row number becomes crucial when the partitioned data is then ordered:
SELECT category, favorites, row_number() OVER (
PARTITION BY category ORDER BY favorites DESC
) FROM trips;
category | favorites | row_number
------------+-----------+------------
badminton | 3 | 1
badminton | 2 | 2
badminton | 1 | 3
badminton | 0 | 4
golf | 3 | 1
golf | 2 | 2
All that remains is limiting the results to the top ranked rows and our query matches the expected output.
At this time there aren't any constructs for OVER
built into Ecto 2.0 and it doesn't support arbitrary FROM
clauses.
The only way to utilize window functions is with the raw Ecto.Adapters.SQL.query
function.
Using the from
macro from Ecto.Query
with a sub-select would be preferable to working with a raw string, but we aren't there yet.
We'll make a new test that is very similar to the last, but which expects a Postgrex.Result
struct instead.
The Result
struct wraps a list of raw rows with all of the trip data.
test "grouping top trips by category using windows" do
{:ok, result} = Trip.popular_over_category()
assert result.num_rows == 8
assert Enum.all?(result.rows, &(Enum.at(&1, 3) >= 2))
end
Now the popular_over_category/1
function must be defined to construct a SQL query:
def popular_over_category(per \\ 2) do
query = """
SELECT * FROM (SELECT *, row_number() OVER (
PARTITION BY category
ORDER BY favorites DESC, name ASC
) FROM trips) AS t WHERE t.row_number <= $1::integer;
"""
Ecto.Adapters.SQL.query(Triptastic.Repo, query, [per])
end
The query string uses a subquery to build up trips partitioned by category.
The where
clauses filters out any trips with a row_number
below the desired threshold, and only the top favorites in each category are returned.
With the change in place the new test is now passing!
Inspecting the test results, with the help of some formatting, yields:
category | favorites | row_number
------------+-----------+------------
badminton | 3 | 1
badminton | 2 | 2
golf | 3 | 1
golf | 2 | 2
pickleball | 3 | 1
pickleball | 2 | 2
tennis | 3 | 1
tennis | 2 | 2
Those are precisely the results we're looking for!
Here is where the presumptions behind this article fall apart and the BEAM blows my mind. The original version of this article was written about window queries in Rails. In those benchmarks the window function was 539.3x faster than the naive version. Naturally, I was excited to see how well the Elixir/Ecto variant would perform in comparison.
This benchmarking test has a lot of boilerplate just to set up the sandbox and insert an arbitrary number of trips into the database.
An outer for
comprehension builds up a sequence of tests with an increasing number of trips for comparison.
All tests are run in separate processes via Task.async |> Task.await
, which introduces the slight complication of sharing sandboxed connection ownership.
Note that the test caps out at 20,000 trips because any more breask Repo.insert_all
, and that is plenty for a comparison.
defmodule Triptastic.TripBenchmarkTest do
use ExUnit.Case
alias Triptastic.{Repo, Trip}
setup do
:ok = Ecto.Adapters.SQL.Sandbox.checkout(Repo)
end
def ms(parent, fun) do
Task.async(fn ->
Ecto.Adapters.SQL.Sandbox.allow(Repo, parent, self())
begin = :os.timestamp
fun.()
finish = :os.timestamp
:timer.now_diff(finish, begin) / 1000
end) |> Task.await
end
for num <- [100, 500, 5_000, 10_000, 20_000] do
@tag num: num
test "compare memory and windows for #{num} trips", %{num: num} do
categories = Stream.cycle(Trip.categories)
trips = for _ <- 1..num do
cat = hd(Enum.take(categories, 1))
fav = trunc(:rand.uniform() * 10)
%{name: "#{cat}-#{fav}", category: cat, favorites: fav}
end
Repo.insert_all(Trip, trips)
mem = ms(self(), fn -> Trip |> Repo.all |> Trip.popular_by_category end)
win = ms(self(), fn -> Trip.popular_over_category end))
wij = ms(self(), fn -> Trip.popular_over_category_joined |> Repo.all end)
IO.puts "| #{num} | #{mem} | #{win} | #{wij} |"
end
end
end
With a small number of trips the performance difference is negligible. As the number of trips increases the cost of loading that many records into memory simply to filter them out does start to add up. Even with 20,000 records being slurped in for manipulation, the naive strategy is only 2x slower. For now, if you are working in Ecto, you can rest assured that the performance of naive queries is good enough not to worry about fiddling with raw SQL.
The simple application used for testing can be found in triptastic on GitHub.
Edit: The benchmark test and chart now includes a hybrid approach where the OVER
sub-select is performed in a join.
This was suggested by Jośe Valim as a way to avoid SQL queries, and provides better query interop with comparable performance.
The fact is, caching makes systems more complicated. Expiration and eviction strategies require planning, foresight, and maintenance. Caching adds additional dependencies in the form of database(s) that store all the cached data. It adds more libraries dedicated to caching and communication with said external cache. Finally, it makes personalizing content unwieldy, rarely worth the extra effort. All this extra effort is only necessary when a language can't do the heavy lifting for you.
Performance without any caching at all has always been possible. Even for a high traffic site serving dynamic content with sizable payloads, it is entirely attainable. All that is required is a language that is powerful enough to make it possible. That language needs to be inherently fast, concurrent, absent of stop-the-world garbage collection pauses, and tolerant in the face of errors. That language is also likely to be compiled, and not an interpreted scripting language. Sadly, none of those attributes describe Ruby (MRI), and as a result every production fortified Ruby application must resort to a menagerie of caching for any hope of performance.
Ruby isn't alone in the caching conundrum, it is a common pitfall of all the dynamic languages commonly used on the web. However, the majority of my caching experience has been focused on fortifying Rails applications, so I'm calling Ruby out.
For years I've been following the development of Elixir and using it for hobby projects. Only recently have I gotten the opportunity to build production systems with it. Now I'm completely spoiled. While I can espouse praise for the language, functional programming, the beauty of pattern matching, and the brilliance of the BEAM all day...that probably won't be convincing. Instead, I'll share a few benchmarks that emphasize the performance gulf between Ruby systems and an Elixir system.
Synthetic benchmarks are a poor measure of anything in the real world. So, let's not pretend this is a scientific comparison. Instead, I'll compare the performance of two systems in production serving up equivalent content. To be fair, the content isn't identical, but that's because the Elixir/Phoenix version can be customized without fear of breaking caching. The true configuration of each application is non-trivial, and the code is confidential, so I can only share an overview of each.
We will be testing is a typical API endpoint that uses token based authentication and returns JSON side-loaded associations. Both applications are using Postgres 9.4, and both are hosted on Heroku, but there is a difference in the servers they run on. The Rails application is running on two Performance-M dynos while the Phoenix application is running on a single 1x Production dyno.
Ruby/Rails
Elixir/Phoenix
SELECT *
The following chart plots the response time in milliseconds when hitting the API endpoint five times. Be warned, these requests were made to production instances with abnormally large data-sets, so the values are fairly noisy. The gray line is Rails, the blue is Phoenix.
These response times are not characteristic of either system, they are at the extreme upper limit. Even so, serving up 2.5x the records with 4.5x the data, without any caching, the Phoenix API response times are 1.5x-2.5x faster.
For several years I focused my effort on squeezing performance out of caching and serialization in the Ruby world. The libraries I've built have been benchmarked and micro-tuned to attain what felt like blazing fast response times. On top of the work put into the libraries there was substantial overhead in constructing APIs to work within the confines of caching. As it turns out, those response times weren't so blazing fast after all.
]]>Sidekiq is a background job processor for Ruby that’s relied on by thousands of Rails apps. It’s regularly used to process slow or resource intensive work in the background. The results and operations of background jobs are often critical to a business and its users...
Keep reading Know Your Sidekiq Testing Rights
]]>We'll start with a vanilla Rails app, with simple authentication and a single
endpoint defined for posts. Posts are top level objects that also have an author
and associated comments. The published
scope limits the response to only
include published posts, and it also pre-loads the associated records. The
endpoint naively generates a brand new JSON payload for every request:
class PostsController < ApplicationController
def index
posts = current_user.posts.published
render json: posts.to_json(include: %i[author comments])
end
end
Now we benchmark the results of assaulting this endpoint with siege. Siege saturates the endpoint with repeated concurrent requests and measures performance statistics. The same requests will be used to benchmark all subsequent modifications, with slight modifications to headers where necessary. All tests are run in production mode to mimic a real production environment.
siege -c 10 -r 10 -b 127.0.0.1:3000/posts
Transactions: 100 hits
Availability: 100.00 %
Elapsed time: 10.89 secs
Data transferred: 14.54 MB
Response time: 1.05 secs
Transaction rate: 9.18 trans/sec
Throughput: 1.34 MB/sec
Concurrency: 9.60
Successful transactions: 100
Failed transactions: 0
Longest transaction: 1.43
Shortest transaction: 0.34
With the initial benchmark in hand our endpoint is ready for layering on some defenses.
The outer layer of defense is HTTP Caching. HTTP caching is the outer
wall, catching hints about what clients have seen and responds intelligently.
If a client has already seen a resource and has retained a reference to the
Etag
or Last-Modified
header, then our application can quickly respond with a
304 Not Modified
. The response won't contain the resource, allowing the
application to do less work overall.
Support for If-Match
and If-Modified-Since
, the reciprocals to Etag
and
Last-Modified
, respectively, are built into Rails. Controllers have
fresh_when
, fresh?
, and stale?
methods to make HTTP caching simple. Here
we're adding a stale?
check to the index action:
class PostsController < ApplicationController
def index
posts = current_user.posts.published
if stale?(last_modified: posts.latest.updated_at)
render json: posts.to_json(include: %i[author comments])
end
end
end
Now we can re-run siege with the If-Modified-Since
header using the latest
post's timestamp.
siege -c 10 -r 10 -b -H 'If-Modified-Since: Mon, 19 Oct 2015 14:13:50 GMT' 127.0.0.1:3000/posts
Transactions: 100 hits
Availability: 100.00 %
Elapsed time: 2.82 secs
Data transferred: 0.00 MB
Response time: 0.27 secs
Transaction rate: 35.46 trans/sec
Throughput: 0.00 MB/sec
Concurrency: 9.65
Successful transactions: 100
Failed transactions: 0
Longest transaction: 0.33
Shortest transaction: 0.06
The data transfer has plummeted to 0.00 MB
and the elapsed time is 2.82 secs
. This is only possible if the client has already cached the data though.
What if the client doesn't have anything cached locally?
When the endpoint is a shared resource that the current client hasn't seen yet it is still possible to return an entire cached payload. This is action caching in Rails. Unlike HTTP caching, where the resource is cached by the client, action caching requires the server to cache the full payload. This is where a cache like Redis or Memcached comes into play.
Action caching is applicable to any resource that is shared between multiple
users without any customization applied. A cache
convenience method is
available within every controller, providing a shortcut to the cache#fetch
method and automatically scoped to the controller.
class PostsController < ApplicationController
def index
posts = current_user.posts.published
if stale?(last_modified: posts.latest.updated_at)
render json: serialized_posts(posts)
end
end
private
def serialized_posts(posts)
cache(posts) do
posts.to_json(include: %i[author comments])
end
end
end
Now the test can be re-run, this time without any HTTP cache headers.
siege -c 10 -r 10 -b 127.0.0.1:3000/posts
Transactions: 100 hits
Availability: 100.00 %
Elapsed time: 0.61 secs
Data transferred: 14.54 MB
Response time: 0.06 secs
Transaction rate: 163.93 trans/sec
Throughput: 23.84 MB/sec
Concurrency: 9.69
Successful transactions: 100
Failed transactions: 0
Longest transaction: 0.09
Shortest transaction: 0.03
Believe it or not the response times are even faster. Total elapsed time is down
to 0.61 secs
, but we are back to 14.54 MB
of data transfer. For a high speed
connection the data transfer isn't a problem, but what about mobile devices?
Reducing the size of an application's responses isn't quite caching, but a door on a chain doesn't exactly sound like defense either. Computing responses can be slow, and the effect of sending large data sets over the wire (particularly to a mobile device) can have a more pronounced effect on speed.
Summary representations as can be used to broadly represent resources instead of more detailed representations. Listing a collection only includes a subset of the attributes for that resource. Some attributes or sub-resources are computationally expensive to provide, and not needed at a high level. To obtain those attributes the client must fetch a separate detailed representation. Each of these representations can be cached individually, or you may opt to cache the summary representation and layer on the less frequently accessed detailed representation.
In our example the posts are being sent along with the author and all of the
comments. Chances are that the client doesn't need all of the comments up front,
so let's eliminate those from the index payload. Note, using a serializer
library like ActiveModelSerializers is recommended over passing options
to to_json
, but we're aiming for clarity.
class PostsController < ApplicationController
def index
posts = current_user.posts.published
if stale?(last_modified: posts.latest.updated_at)
render json: serialized_posts(posts)
end
end
private
def serialized_posts(posts)
cache(posts) do
posts.to_json(include: %i[author])
end
end
end
Only the include
block has been changed to omit comments.
Transactions: 100 hits
Availability: 100.00 %
Elapsed time: 0.62 secs
Data transferred: 4.17 MB
Response time: 0.06 secs
Transaction rate: 161.29 trans/sec
Throughput: 6.72 MB/sec
Concurrency: 9.74
Successful transactions: 100
Failed transactions: 0
Longest transaction: 0.08
Shortest transaction: 0.03
The elapsed time is slightly lower, at 0.62 secs
, but the real win is that the
data transfered is down from 14.52 MB
to 4.17 MB
. Over a non-local
connection that will have a significant impact.
So far our endpoint has been considered unchanging. All of a user's posts are cached together in a giant blob of JSON. Typically, resources that are part of a collection don't expire all at once. It is probable that most cached posts are fresh, with only a few stale entries. Perforated caching is the technique of fetching all fresh records from the cache and only generating stale records. Not only does this reduce the time our application spends serializing JSON, it also reduces the stale blobs laying around in the cache waiting to expire.
This behavior is built into ActiveModelSerializers
, provided caching has been
enabled. We'll simulate it with a private method inside the controller for now.
class PostsController < ApplicationController
def index
posts = current_user.posts.published
if stale?(etag: posts, last_modified: posts.latest.updated_at)
render json: serialized_posts(posts)
end
end
private
def serialized_posts(posts)
posts.map do |post|
cache(post) { post.to_json(include: %i[author]) }
end.join(',')
end
end
Without sporadically expiring requested objects, you'll see a 100% hit rate for every request after the first. That makes testing the endpoint a little misleading, as it will appear slower than action caching and you can't see the benefit of fine grained caching.
Transactions: 100 hits
Availability: 100.00 %
Elapsed time: 1.84 secs
Data transferred: 4.17 MB
Response time: 0.18 secs
Transaction rate: 54.35 trans/sec
Throughput: 2.26 MB/sec
Concurrency: 9.71
Successful transactions: 100
Failed transactions: 0
Longest transaction: 0.25
Shortest transaction: 0.08
As expected the results are slower, down to 1.84 secs
for the full run. That
is still nearly 10x
faster than the original time of 10.89secs
though, and
it comes with added resiliency against complete cache invalidation.
Personalized payloads can't be cached in their entirety. Usually some, if not most, of the payload can be cached and the personalized data is generated during the request. This situation arises when a cache key would be so specific that you're generating a cache entry for every user. That's useless and wasteful, not clever at all!
This is a variant of fragment caching, with a twist. Typically fragment caching sees each record serialized and fetches a portion of the results to avoid expensive computations. Well, serializing the record is the expensive computation!
We want to pull the post out of the cache and inject a personalized attribute directly. In this case, want to indicate whether the client has ever "read" this post before. The client could be anybody, but the rest of the response is the same for everybody.
class PostsController < ApplicationController
def index
posts = current_user.posts.published
if stale?(last_modified: posts.latest.updated_at)
render json: serialized_posts(posts)
end
end
private
def serialized_posts(posts)
posts.map do |post|
cached = cache(post) { post.as_json(include: %i[author]) }
cached[:read_before] = current_user.read_before?(post)
end.to_json
end
end
Transactions: 100 hits
Availability: 100.00 %
Elapsed time: 4.08 secs
Data transferred: 0.03 MB
Response time: 0.40 secs
Transaction rate: 24.51 trans/sec
Throughput: 0.01 MB/sec
Concurrency: 9.73
Successful transactions: 100
Failed transactions: 0
Longest transaction: 0.45
Shortest transaction: 0.20
Now the post had to be cached with as_json
instead of to_json
, so that we
could modify it as a hash and avoid re-serializing it during the request. Even
so, the total runtime only grew to 4.08 secs
, merely 38% of the original naive
runtime. Still, avoid personalizing cached endpoints whenever possible. It is
drastically slower, and introduces a lot of additional complexity.
Strategic caching can get you a long way, but be sure to employ additional defenses against ballooning payloads and malicious requests. Practices like rate limiting and resource pagination are invaluable for maintaining predictable response times. Relying on a single layer of defense is a performance headache waiting to happen. Each layer of caching takes a little more effort than the previous layer, but a couple of layers working together will get you started.
Note: External layers like a reverse-proxy cache can check the contents
of the Cache-Control
header to fully respond to requests for public resources,
without even touching your application layer. However, this post focused on what
your application can do itself, so there weren't any details about that.
All of the examples in this post are written with Readthis in mind, but
the technique will work for any instrumented library. In fact, because the event
names are standardized as cache_{{operation}}.active_support
, these techniques
will work for any ActiveSupport
cache.
With standard Rails production logging your request timing is comprised of three
values: duration
, view
, and db
. For basic applications without any
external services or caching the combined view
and db
timing is sufficient.
Once your application integrates additional databases such as Memcached,
ElasticSearch, or Redis the standard log output won't tell the whole story.
Below is sample output of a log without any additional timing information. Note that it is formatted using Lograge and has been edited for clarity.
method=GET path=/api/posts status=200 duration=55.73 view=27.06 db=21.51
All of the render timing is bundled into the view. Any external requests, including cache or search, are included in the timing. That simply won't do! Let's get started unpacking those timings.
The linchpin of Rails instrumentation is Notifications
. If you recall from
Instrumenting Your Cache With Notifications, it is the module that enables
hooking into any instrumented block to receive timing information. Here is an
how you would write out the timing of every cache event:
ActiveSupport::Notifications.subscribe(/cache_/) do |*args|
event = ActiveSupport::Notifications::Event.new(*args)
puts "Cache Event Duration: #{event.duration}"
end
That example merely writes out the duration to STDOUT
. Instead, we want to
forward that event to a module that will aggregate the timing for multiple
events.
Remember, the subscribe
block will be called for every cache event, many of
which can trigger during a single request. Aggregated timings can be recorded
with a small purpose-built module, so long as you're mindful of thread safety.
For multi-threaded servers such as Puma it is crucial that the recording events
is limited to the thread handling the request. Modules are constants, and
constants are global, so we need to use Thread.current
to store timing values.
# config/initializers/readthis.rb
module Readthis
module Instrumentation
module Runtime
extend self
def runtime=(value)
Thread.current['readthis_runtime'] = value
end
def runtime
Thread.current['readthis_runtime'] ||= 0
end
def reset_runtime
new_runtime, self.runtime = runtime, 0
new_runtime
end
def append_runtime(event)
self.runtime += event.duration
end
end
end
end
With this module available it is trivial to append new runtimes for cache
events. Instead of writing out the event duration within the subscribe block,
the append_runtime
method is called with the event.
ActiveSupport::Notifications.subscribe(/cache_/) do |*args|
event = ActiveSupport::Notifications::Event.new(*args)
Readthis::Instrumentation::Runtime.append_runtime(event)
end
Augmenting the controller rendering lifecycle isn't at all invasive.
ActionController
provides a method explicitly for the purpose of fixing
erroneous view runtimes. The cleanup_view_runtime
hook is called around
rendering, offering a place to remove cache runtime from the full view runtime.
# config/initializers/readthis.rb
module Readthis
module Instrumentation
module ControllerRuntime
attr_internal :readthis_runtime
def cleanup_view_runtime
before_render = reset_runtime
runtime = super
after_render = reset_runtime
self.readthis_runtime = before_render + after_render
runtime - after_render
end
def append_info_to_payload(payload)
super
payload[:readthis_runtime] = (readthis_runtime || 0) + reset_runtime
end
def reset_runtime
Readthis::Instrumentation::Runtime.reset_runtime
end
end
end
end
Another hook, append_info_to_payload
, provides a way to inject
additional runtime information into the event payload. Above we do just that,
append the readthis_runtime
and reset it for the next request.
The on_load
hook from ActiveSupport
is the final piece in gluing everything
together. We can pass a block that will be executed within ActionController
after it has finished loading. It is within that block that we include our
ControllerRuntime
module and subscribe to cache notifications.
ActiveSupport.on_load(:action_controller) do
include Readthis::Instrumentation::ControllerRuntime
ActiveSupport::Notifications.subscribe(/cache_/) do |*args|
event = ActiveSupport::Notifications::Event.new(*args)
Readthis::Instrumentation::Runtime.append_runtime(event)
end
end
While it is possible to format Rails' built in logger, we'll use the compact
formatting provided by Lograge. Not only is it compact and easy to read, but it
is simple to modify at configuration time through custom_options
. Lograge
uses the same instrumentation subscribers as Rails' own logging internals. The
logging event is passed through with accumulated timings in the payload, which
can be accessed directly.
MyApp.Application.configure do |config|
config.lograge.custom_options = lambda do |event|
{ cache: event.payload[:readthis_runtime] }
end
end
With this addition the log output will look like this now:
method=GET path=/api/posts status=200 duration=55.73 view=15.06 db=21.51 cache=11.92
I initially underestimated the effort involved in piping instrumentation data into logs. The classes are well documented, cleanly abstracted, and compose well—but it takes a while to understand how to glue everything together. Understanding the tools available is the hard part, once you get past that the effort involved is minimal.
Custom instrumentation and logging should be provided for mature libraries. If it isn't available, ask for it, or better yet, submit a pull request. Enhanced logging tools will be rolled into Readthis soon, so augmenting logs with cache timings is as simple as requiring a module.
]]>Many of the gems intended to work with Rails come with built in support for hooking into Notifications. Readthis, a Redis based caching library, is among the gems that have support built in. Provided ActiveSupport is available Readthis will use Notifications to instrument every cache operation as a separate event. Because instrumentation is so flexible, Readthis doesn't have any additional hooks for debugging or configuration for a logger. Here we'll at a few powerful ways to utilize instrumentation in your app.
The Notifications module has excellent documentation that is recommended reading for any Rails developer. However, as a precursor to the use cases we're about to explore, here are the most important parts:
start
and finish
times.To instrument a bit of code all that's required is wrapping it in an
instrument
block:
ActiveSupport::Notifications.instrument('event.namespace', payload) do
do_something_worth_measuring
end
By itself the instrument
block won't emit any events. However, once some other
part of your application has subscribed to the event.namespace
event then
metrics will start to be reported. The most straight forward use of
instrumentation is tracking metrics, so let's start there.
Measuring performance is the primary use case for instrumenting with notifications, making it a natural starting point. Any service that aggregates metrics over time, such as statsd, is perfect for collecting remotely. All that is required is to subscribe to a pattern that matches any cache operation and forward the timing measurements to the statsd instance:
require 'statsd'
ActiveSupport::Notifications.subscribe(/cache_.*\.active_support/) do |name, start, finish, _, _|
Statsd.measure(name, finish - start)
end
Every time a cache operation is called a new measurement will be emitted. The service aggregating the metrics will collapse them within the resolution window, typically a matter of seconds. Using a visualization tool such as Librato or Graphite the measurements can then be averaged, have standard deviation tracked, etc.
There are echelons of logging and log integration within a production application. Logging can range from robust production level logging to simple debug statements stuffed into development. Hooking into Rails for production metrics logging is too complex a topic for this post, instead we'll look at outputting simpler debugging logs.
There are a wide assortment of loggers in the Rails space, but we'll work with
the vanilla Rails logger here. It makes use of ActiveSupport::LogSubscriber
to
register and measure runtimes for database and view performance. The APIs used
for logging are wide open, and can be used to include custom values.
module Readthis
class LogSubscriber < ActiveSupport::LogSubscriber
# LogSubscriber wraps payloads in an Event object, which has convenience
# methods like `#duration`
def cache_read(event)
payload = event.payload
debug "Readthis: #{payload[:name]} (#{event.duration}) #{payload[:key]}"
end
alias_method :cache_write, :cache_read
end
end
Readthis::LogSubscriber.attach_to :active_support
The log subscriber must be "attached" to a particular namespace. With
notifications the namespace is appended to the end of the event name, for
example the cache_read
event is namespaced as cache_read.active_support
.
When a read
or write
event is emitted you'll have entries like this output
to the logs:
Readthis: read (1.74) model/1/12345678
Within applications it is common to use fetch
to retrieve values. Any direct
call to cache
within a template is really calling Rails.cache.fetch
. The
benefit of using fetch
over read
is that it accepts a block and will use the
result of the block to write a value to the cache if the read is a "cache miss".
Here is a simple example of using fetch
:
cache.fetch('special-info') do
'Expensive info to be cached'
end
If the key 'special-info'
has been cached it will be returned immediately from
a cache read
, if it is missing it will be written with a write
. It is
desirable to have a warm cache with a majority of fetches resulting in reads
rather than writes. Notifications can be used to track the hit rate of fetch
operations by comparing raw reads to raw writes. Here is a simplistic class that
uses sets to maintain a list of all keys for read and write operations:
require 'set'
module Readthis
class HitRateInstrumenter
attr_reader :reads, :writes
def initialize
@reads = Set.new
@writes = Set.new
subscribe('cache_read.active_support', @reads)
subscribe('cache_write.active_support', @writes)
end
def hit_rate
1 - (reads.intersection(writes).length.to_f / reads.length)
end
def reset
@reads.clear
@writes.clear
end
private
def subscribe(pattern, set)
ActiveSupport::Notifications.subscribe(pattern) do |_, _, _, _, payload|
set.add(payload[:key])
end
end
end
end
When the class is initialized it subscribes to the events that it cares about and simply stores the cache key in each set. Later we can compute the hit rate percentage by comparing the number of keys that are read to the number that are written.
cache = Readthis::Cache.new
instrumenter = Readthis::HitRateInstrumenter.new
cache.fetch('a') { true }
cache.fetch('b') { true }
cache.fetch('a') { true }
instrumenter.hit_rate #=> 0.5
Notifications are integral to the modularity and re-usability of Rails internals, and a powerful abstraction to have at your disposal. Reach for them whenever you need to measure, log, or track events within an application.
]]>This isn't such a hypothetical situation. It happens with Redis servers often enough. Redis has a broad set of tools specifically to enable backing up, restoring, and migrating data. In most situations that set of tools is perfectly adequate. However, sometimes there are constraints imposed by various providers that make migration much trickier.
Think of this post as a verbose migration constraint solver. We'll start with the ideal means of migration and gradually add constraints. Eventually, we'll arrive at the last resort and see how to work with it.
The most reliable and preferred way to handle a migration is with master/slave
replication using SLAVEOF
. All that is required is configuring the
destination instance as a slave of your current production instance. Once the
new database has caught up to production your application's Redis clients can be
directed to the new database and the server can be promoted to master.
SLAVEOF {SOURCE_HOST} {SOURCE_PORT}
INFO replication
, it will say that the
link is up and the last sync was 0 or more milliseconds agoSLAVEOF NO ONE
Use of SLAVEOF
isn't allowed.
Lacking access to SLAVEOF
eliminates replication outright. Any managed Redis
provider worth their salt supplies replication with failover, making it
impossible for you to control the master/slave status of your instance. Master
servers can't be put into slave mode without killing the sentinels or
other HA system controlling them, which certainly isn't possible with a
cloud service.
When replication isn't an option you can fall back to transferring the database
dump persisted to disk. Even databases that aren't configured for periodic
RDB persistence can trigger a dump to disk with either SAVE
or
BGSAVE
, the asynchronous version. With the database dumped to disk it
can be transfered to the new server or uploaded with a data migration tool.
One major caveat around snapshot loading is the exact timing that is used. Snapshots must be loaded when the Redis server is shut down. On reboot the Redis instance will load data from the dump. Any server that is configured to perform RDB persistence will automatically save to disk on shutdown. That means you must shut down the destination instance before overwriting the snapshot or it will be overwritten automatically during shutdown.
BGSAVE
scp
or rsync
/var/lib/redis/dump.rdb
No access or mechanism to upload a RDB snapshot.
They don't have any way to upload a RDB dump file. Understandable, any cloud service provider doesn't want to grant you access to the underlying file-system. Still, some providers have interfaces that allow you to load a dump into the new instance.
When bulk loading and replication aren't available you can try to migrate one
key at a time with MIGRATE
. The MIGRATE
command atomically transfers a key
from a source database to a destination database. By default this is a
destructive action unless the COPY
option is passed, it must be handled with
care when pointed at a production instance.
Migration isn't automatic, it requires a script to iterate over all of the keys
and call MIGRATE
for every individual key.
SCAN
, or KEYS
for smaller data setsMIGRATE host port key destination-db timeout COPY
for each keyAuthorization is enabled (thankfully), preventing MIGRATE
.
MIGRATE
can't be used if the target server requires authorization. There have
been pull requests to allow authorization for MIGRATE
commands, but it
hasn't been merged in.
Maybe one could use DUMP/RESTORE - why would one want to do that though?
— HippieLogLog (@itamarhaber)
When you eliminate all of the traditional, supported, recommended means of
performing data migration what are you left with? The answer is DUMP
and
RESTORE
. Internally MIGRATE
uses DUMP
on the source database and
RESTORE
on the target database. Those commands are available to any Redis
client, no special privileges are necessary.
SCAN
, or KEYS
for smaller data setsDUMP
for each source key, which returns a serialized representation of
the valueTTL
for each source key to record the expirationRESTORE key ttl value
on the destination for each keyAll of the same scripting work necessary to use MIGRATE
is needed for dumping
and restoring, along with the additional work of setting the TTL for each key
that is restored. This should be your absolute last resort, let's hope you're
never in the situation to actually use it.
View other enterprise systems with suspicion and distrust—any of them can stab you in the back.
Michael T. Nygard—Release It!
Inevitably applications need to do actual time consuming, highly coordinated work. As engineers we know not to handle such hard work during a request, it needs to be pushed into the background. Often that work can be performed locally inside of application code, or purely within the database; but eventually external systems will come into play. When our applications start coordinating work with external services we can really start to lean on our background processor for isolation from others systems outside of our control.
Let's set up a scenario, something common and digestible, and work through how to break it up at the boundaries. This (relatively) concrete example will demonstrate when and how to make services transactional through isolation.
Our application accepts multimedia uploads, including videos. Perhaps we've found that handling uploads is fraught with timeouts and connection issues, so instead the mobile apps upload videos directly to S3. The mobile app then alerts the server that a video is ready and the server sets off to start processing the video. We spare no expense processing the video, and so numerous external services are utilized. Processing is comprised of several steps:
Each one of those tasks require interfacing with an external service, and the failure of one task shouldn't have any effect on the others. Each task must be wrapped in an independent unit of work, a background job. The job manager will make sure the work is done in a transactional manner, handling retries in the event of errors.
A transaction is an abstract unit of work processed by the system. This is not the same as a database transaction. A single unit of work might encompass many database transactions.
Michael T. Nygard—Release It!
If the video processing was a series of interactions with an ACID compliant database, all of the operations could be wrapped in a transaction, or set of transactions. If any of the processing steps were to fail all of the changes could be rolled back and retried again later. This behavior is fundamental to eliminating duplicate entries and orphaned data.
Here is a paraphrased example illustrating how the steps in our video processing task would operate if we could wrap them in a database transaction:
def process_video(video_id)
video = Video.find(video_id)
Video.transaction do
perform_cloud_copy!(video)
perform_transcoding!(video)
perform_transcription!(video)
end
end
Sadly, services over the internet don't provide any such transactional behavior, so we need to approximate it ourselves. We can compensate for a lack of transactional safety by breaking tasks into discrete background jobs.
Because it is amazingly fast and utterly reliable, we'll use Sidekiq for our examples. However, the same principles hold true for any background processing library that automatically retries failing jobs—most ActiveJob compliant queues will do the trick.
The processing sequence starts with a worker that copies the remote file and then kicks off the other jobs.
class VideoCopyWorker
include Sidekiq::Worker
def perform(video_id, object_path)
video = Video.find(video_id)
perform_copy!(video, object_path)
enqueue_processing(video)
end
private
def perform_copy!(video, object_path)
CloudCopier.new(video).copy!(object_path)
end
def enqueue_processing(video)
VideoTranscodeWorker.perform_async(video.id)
VideoTranscribeWorker.perform_async(video.id)
end
end
After the object is successfully copied, the transcode and transcription workers
are enqueued to process the video. If the cloud_copy!
fails it will raise an
exception, aborting the job and triggering a retry a little bit later. A failed
cloud copy also prevents the other workers from being enqueued. At a later
point, when the cloud_copy!
is successful the secondary jobs will be enqueued.
The workers are wrapped safely in individual jobs. This encapsulation is essential to prevent duplicate work and prevent unwanted side effects. To paint a clearer picture here are pseudo examples of the transaction and transcription workers:
class VideoTranscodeWorker
include Sidekiq::Worker
def perform(video_id)
video = Video.find(video_id)
Transcoder.new(video).create!
end
end
class VideoTranscribeWorker
include Sidekiq::Worker
def perform(video_id)
video = Video.find(video_id)
Transcriber.new(video).create!
end
end
The implementation of Transcoder
and Transcriber
are intentionally vague to
keep the focus on job encapsulation rather than actual service integration.
It is important to keep each job idempotent, meaning the job can be called
repeatedly but will only perform the actual work once. In order to keep the
VideoCopyWorker
job idempotent there needs to be a check for whether the video
has been copied yet:
def perform(video_id, object_path)
video = Video.find(video_id)
unless video.copied?
perform_copy!(video, object_path)
enqueue_processing(video)
end
end
Each of the workers needs a similar guard to ensure it is idempotent.
Splitting work that coordinates with external systems into independent jobs is simple, straight forward, and a reliable way to give your system more resiliency. Just as you split classes up by responsibility and minimize communication between objects, break work apart around integration points with other systems. Isolate external integrations like it is going out of style (which it's not). Don't trust other systems with your sites reliability. External may mean another process, a different host, or a service provided by another company, it's all the same to your system.
]]>When restarting unicorn with the USR2 signal, a new master is created and workers are forked off. In my config/unicorn.rb, before switching to readthis, I had
Rails.cache.reconnect
to reconnect to redis after forking. I believe this was an implementation of the redis-store gem, which you aren't using here.How would you suggest I reconnect the unicorn worker to the redis-based cache with readthis? Thanks!
The good news is, after redis-rb 3.1.0
you don't need to
manually reconnect your redis clients. In fact, it isn't desirable to do so! It
was common, historically, to force a Redis reconnect after a Unicorn or Puma
cluster forked off child workers. That was in order to avoid sharing the same
socket between multiple processes, a recipe for unpredictable behavior and
general mayhem. The alternative to manually reconnecting was an error from Redis
warning you about the insanity that would ensue:
Tried to use a connection from a child process without reconnecting. You need to reconnect to Redis after forking or set :inherit_socket to true.
Reconnecting after a child forks is just one of the errors that the Redis client will automatically recover from. This post aims to provide some more context and a whiff of exploration into how the redis client heals itself.
For proof of the reconnection claim and a concrete point of reference we'll
co-opt an example from the redis-rb
test suite. Borrowing from the
fork_safety_test
:
require "redis"
redis = Redis.new
redis.set("foo", 1)
child_pid = fork do
begin
redis.set("foo", 2)
rescue Redis::InheritedError
exit 127
end
end
_, status = Process.wait2(child_pid)
puts status.exitstatus #=> 0
puts redis.get("foo") #=> "2"
The code snippet starts out by instantiating a client in the parent process. It
then forces a connection to be established by calling set
. The Redis client is
lazy, so it will only establish a connection the first time a command is sent—
without this initial connection before the fork there wouldn't be any socket
inheritance to test. Immediately after setting the value 1
a child process is
forked, inheriting the parent's Redis instance, connection and all.
After waiting for the process to return we can see that it exited without a
problem, returning a happy 0
status. Additionally, the child's set
command
was successful in overwriting foo
with the value 2
, so we know everything
worked as expected.
How does the client know to reconnect after a fork? It's relatively simple. All commands executed by the client are centralized, passing through a chain of base methods. It is within this method chain that common behavior such as logging and connection management are guaranteed. The abbreviated flow of methods looks like:
call -> process -> ensure_connected
The redis-rb
source is idiomatic, straight forward, and an excellent place to
learn about connections and the Redis command protocol. It is, however, too
verbose to include verbatim in this post, so the code sample has been modified
from its original context for clarity.
The ensure_connected
method is, predictably, where the reconnection magic
happens:
# lib/redis/client.rb#334
def ensure_connected
attempts = 0
begin
attempts += 1
if connected?
unless inherit_socket? || Process.pid == @pid
raise InheritedError, INHERITED_MESSAGE
end
else
connect
end
yield
rescue BaseConnectionError
disconnect
if attempts <= @options[:reconnect_attempts] && @reconnect
retry
else
raise
end
rescue Exception
disconnect
raise
end
end
Within a begin/retry
block the connection is verified and the current PID is
compared to the PID from when the connection was established. If the PID is
different then the process has since forked and an InheritedError
is raised.
InheritedError
is one of numerous specific connection errors that inherit from
BaseConnectionError
:
# lib/redis/errors.rb#37
# Raised when the connection was inherited by a child process.
class InheritedError < BaseConnectionError
end
When BaseConnectionError
is rescued there is an immediate disconnect, dropping
the connection and clearing the old PID. Provided the reconnect attempt is lower
than the reconnect limit, the block is retried and a new connection is
established. The reconnection mechanism is guarded by tracking the number of
attempts. It is guaranteed not to reconnect infinitely when faced with
persistent connection errors.
By centralizing the execution of commands, the client keeps connection management simple and understandable.
Stop worrying about managing your application's connections to Redis. Problems with a network outage, unexpectedly closed connection, or an inherited socket error? Not a problem, the Ruby client for Redis has you covered.
]]>Pipelining commands to minimize I/O, optimizing storage on in the database and minimizing persistence overhead can get you pretty far toward caching bliss. However, there are still some more speed improvements we can squeeze out. It's time to look at applying Lua scripting to caching operations.
Redis has provided Lua for scripting since version 2.6.0
. Typically it
is used for transactional semantics and more advanced queuing behavior. Today we
are going to look at a data driven use-case for executing Lua scripts in Redis.
The possibility to manipulate large volumes of data on the server opens up
avenues of performance that are normally prevented by the I/O heavy nature of
Redis:
Most Redis work-flows tend to be I/O bound and not CPU bound. Even when you see the CPU at 100% it is likely to be all about protocol handling. This is almost impossible to avoid as Redis commands are much faster than dealing with I/O. With scripting we can put at much better use our bandwidth and CPU power.
Wouldn't it be awesome to construct fully prepared API responses directly from the server? Doing so means fewer commands to execute, fewer I/O operations, less bytes transfered, fewer objects allocated and less work done on the client. Let's see how it can be done.
There are other primers on Lua integration, so we'll stay focused on why offloading work to Lua is desirable, and how it can be used to speed up data heavy tasks, rather than the semantics of the language or its integration into Redis. However, as an extremely specific introduction to Lua scripting for our use case, here is a high level overview of scripting with Lua in Redis.
1
. For example, the first key passed can be
accessed as KEYS[1]
, instead of KEYS[0]
.Executing scripts on the server is as simple as using the EVAL
command, which
can be used from the command line like so:
redis-cli EVAL 'return redis.call("KEYS", "*")' 0
Throughout the rest of this post the examples will all be in Ruby; partially because we're directly comparing Ruby client performance against Lua on the server, but also because it's much easier to send dynamic keys.
Continuing from the post on hashed caching, we'll investigate the process
of reconstructing a blog post and all of its associated records. For this
example it is assumed that all of the serialized records are stored within
fields where the names are posts/1
, authors/1
, etc. The goal is to
efficiently reconstruct the data from multiple hashes into a single payload
where all of the posts, authors, and comments are grouped under common keys,
like so:
{ "authors": [], "comments": [], "posts": [] }
The sample data we're working with and the output format are greatly simplified from actual application data. The purpose of this exercise is to explore the performance possibilities of using Lua, so the reconstruction process is what truly matters. Building up json-api compliant payloads would be a great exercise, maybe some other time.
The starting point is a ruby script that performs the following high level steps:
MULTI
block, fetch the full contents of each hash for each post
via HGETALL
.# Boilerplate, configuration and data seeding has been left out to emphasize
# the relevant bits of code.
hashes = REDIS.multi do
('posts/0'..'posts/30').map { |key| REDIS.hgetall(key) }
end
array_backed_hash = Hash.new { |hash, key| hash[key] = [] }
payload = hashes.each_with_object(array_backed_hash) do |hash, memo|
hash.each do |key, val|
root, _ = key.split('/')
memo[root] << val
end
end
puts payload.length #=> 3
puts payload.keys.sort #=> ['authors', 'comments', 'posts']
puts payload['posts'].length #=> 30
Now, with a working reference in place, we can start translating to Lua and offloading the work. The first step is simply testing that we can pass keys along to the server and execute the commands we expect:
keys = ('posts/1'..'posts/30').to_a
REDIS.eval('return KEYS[1], KEYS[2]', keys) #=> ['posts/1', 'posts/2']
That did it, keys were passed in and are available in the KEYS
table. Next
we'll iterate over all of the keys, fetching the contents of each hash. There
are a few syntactical jumps here, for example usage of local
, for
, and
ipairs
, but nothing tricky is going on. It is all variable declaration and
looping over the KEYS
table:
script = <<-LUA
local payload = {}
for _, key in ipairs(KEYS) do
payload[key] = key
end
return cjson.encode(payload)
LUA
REDIS.eval(script, keys) #=> {"posts\/1":"posts\/1",...
Note the call to cjson.encode
for the return value. Without encoding the
return value as a string the table will be returned as nil
, rather
unintuitively. The cjson
module is indispensable for client/script interop.
Commands can be executed within scripts through redis.call
. Using call
the
script can use the HGETALL
command for each key to build up the payload.
local payload = {}
for _, key in ipairs(KEYS) do
local hash = redis.call('HGETALL', key)
end
return cjson.encode(payload)
The final step is to loop over each field/value pair in the hash in order to construct our desired payload. This is largely a mechanical translation of the Ruby enumeration we saw earlier. Only the Lua script is being shown here—it's much more readable with some syntax highlighting.
local payload = {}
for _, key in ipairs(KEYS) do
local hash = redis.call('HGETALL', key)
for index = 1, #hash, 2 do
local field = hash[index]
local data = hash[index + 1]
local root = string.gsub(field, '(%a)([/\]%d*)', '%1')
if type(payload[root]) == "table" then
table.insert(payload[root], data)
else
payload[root] = {data}
end
end
end
return cjson.encode(payload)
There you have it, the fully translated construction script moved to Lua! The entire purpose of this exercise is to squeeze out more performance from our Redis cache. Naturally, it's time to do some benchmarking!
Here the script is being loaded once from an external file through SCRIPT LOAD
and then referenced with EVALSHA
to avoid the overhead of repeatedly sending
the same script to the server.
SHA = REDIS.script(:load, IO.read('payload.lua'))
def construct_ruby
# see above
end
def construct_lua
REDIS.evalsha(SHA, ('posts/1'..'posts/30').to_a)
end
Benchmark.ips do |x|
x.report('ruby') { construct_ruby }
x.report('lua') { construct_lua }
end
The results are impressive, checking iterations per second for pipelined Ruby and scripted Lua:
Lua is your performance go-to whenever you want to minimize round trips, guarantee atomicity, or process large swaths of data without slurping it into memory. It is a perfect fit for reliable queues, atomic scheduling, and custom analytics. It can also be indispensable for processing large data sets without slurping all of the data back to the client.
A few parting words of caution. Scripts are evaluated atomically, which in the
world of Redis means that no other script or Redis command will be executed in
parallel. It's guaranteed by the single threaded "stop the world" approach.
Consequently, EVAL
has one major limitation—scripts must be small and fast to
prevent blocking other clients.
If you spend some time browsing through Redis documentation you'll quickly stumble upon references to "intelligent caching". Within the context of Redis, intelligent caching refers to leveraging the native data types rather than storing all values as strings. There are numerous examples of this out in the wild, some using ordered lists, some using sets, and a notable example using hashes.
The inspiration for this post came from the memory optimization documentation for Redis itself:
Small hashes are encoded in a very small space, so you should try representing your data using hashes every time it is possible. For instance if you have objects representing users in a web application, instead of using different keys for name, surname, email, password, use a single hash with all the required fields.
Typically Redis string storage costs more memory than Memcached, but it doesn't have to be that way. Memory consumption, write and read performance can be boosted by using the optimized storage of the hash data type. Wrapping data in hashes can be semantically cleaner, much more memory efficient, faster to write, and faster to retrieve.
Smaller and faster always sound great in theory, let's see if it proves to be true.
Before delving into specific use cases, we'll set up some benchmarking and measurement scripts to confirm or deny the following hypothesis: bundling values into hashes is more memory efficient and performant than discrete string storage.
The benchmark, written in Ruby only for the sake of simplicity, performs the following steps:
require 'redis'
require 'benchmark'
require 'securerandom'
REDIS = Redis.new(url: 'redis://localhost:6379/11')
REDIS.flushdb
def write_string(obj, index)
obj.each do |key, data|
REDIS.set("string-#{key}-#{index}", data)
end
end
def write_hash(obj, index)
fields = obj.flat_map { |field, value| [field, value] }
REDIS.hmset("hash-#{index}", *fields)
end
values = (0..1_000).to_a.map do |n|
data = SecureRandom.hex(100)
(0..100).to_a.each_with_object({}) do |i, memo|
memo["field-#{i}"] = data
end
end
Benchmark.bm do |x|
x.report('write_string') do
REDIS.multi do
values.each_with_index { |value, index| write_string(value, index) }
end
end
x.report('write_hash') do
REDIS.multi do
values.each_with_index { |value, index| write_hash(value, index) }
end
end
end
The results for one iteration in seconds (lower is better):
With some slight modifications, the same script can be used to measure memory
consumption, simply by checking Redis#info(:memory)
for each strategy.
# ...speed script from above
REDIS.flushdb
write_strings
puts "String: #{REDIS.info(:memory)['used_memory_human']}"
REDIS.flushdb
REDIS.config(:set, 'hash-max-ziplist-value', '64')
write_hashes
puts "Hash: #{REDIS.info(:memory)['used_memory_human']}"
REDIS.flushdb
REDIS.config(:set, 'hash-max-ziplist-value', '1024')
write_hashes
puts "Tuned: #{REDIS.info(:memory)['used_memory_human']}"
The results, in megabytes, are consistent across multiple iterations (lower is better):
This demonstrates a sizable difference between string storage, hash-based storage, and tuned hash-based storage (more on that later). With this sample data, the memory savings are nearly 30%. Those are pretty huge savings! Note that there is a slight trade off between tuned hash entry size and insertion time.
Again, with some slight modifications, the writing and memory benchmark can also
be used to measure read speed. There isn't any appreciable performance
difference for HGETALL
between hash entry size, so only one data-point is
included.
# ...speed script from above
string_keys = (0..1_000).to_a.flat_map do |n|
(0..100).to_a.map { |i| "string-#{n}-#{i}" }
end
hash_keys = (0..1_000).to_a.map do |n|
"hash-#{n}"
end
Benchmark.bm do |x|
x.report('read_string') do
REDIS.multi do
string_keys.map { |key| REDIS.get(key) }
end
end
x.report('read_hash') do
REDIS.multi do
hash_keys.map { |key| REDIS.hgetall(key) }
end
end
end
The results for one iteration in seconds (lower is better):
Ignoring the fact that reading over 10,000 items in a single MULTI
command is
pretty slow, you can see that hash fetching is 26% faster. This is intuitive.
Fetching one hundredth as many keys should be faster.
As expected, the documentation was right. With a little tuning, the hash-based approach can be smaller and faster. There are some caveats though. Let's explore them with a practical example.
The original idea, as demonstrated, uses a sharding scheme to bucket models into hashes based on a model's unique key. This has a wonderful sense of symmetry and predictability, but doesn't do anything to ensure that the models are related to each other. The cache only has a finite amount of space and we want old values to be evicted, so it is desirable to keep all fields within a hash related. Additionally, there are performance benefits to naively fetching each hash in its entirety.
For example, imagine an API endpoint that serves up JSON data for blog posts.
Each post would include the author, some comments, and tags. Naturally the
serialized JSON would be cached in order to boost response times. Typically each
post and the associated data would be stored separately, as strings, available
at keys such as posts/:id/:timestamp
. Instead, with hash based caching, all of
the serialized values are stored inside of a single hash referenced by the
post's key at the top level.
When requests come in, a post is retrieved from the database, the cache
key generated in the format of posts/:id/:children/:timestamp
, and if the
cache is fresh there is only a single fetch necessary. Field invalidation for
associated children (authors, comments, etc.), field additions (new comments,
new tags), or field removal (deleted comments) are simply dealt with by using
the number of children and a timestamp within the cache key.
Previously I mentioned that there was a significant difference in storage that
was achieved by "tuning." Through the Redis config file, or with the CONFIG
command, the storage semantics of most data structures can be configured. In
this case the hash-max-zipmap-*
values are most important.
Hashes, Lists, Sets composed of just integers, and Sorted Sets, when smaller than a given number of elements, and up to a maximum element size, are encoded in a very memory efficient way that uses up to 10 times less memory.
As long as the hash-max-zipmap-value
size is larger than the maximum value
being stored in a hash then storage will be optimized. For caching it is
recommended that you use a particularly aggressive value, at least 2048-4096b.
Fields within a hash can't have individual expiration, only the hash key itself. Bundling related fields together allows the entire hash to be treated as a single unit, eventually falling out of memory entirely when it is no longer needed. This means that it is preferable to use Redis as an LRU cache. We're already doing that though, right?
Bundling serialized data for associated models is just one way to utilize the native hash type for intelligent caching. Custom caching schemes require more explicit and purposeful schemes for organizing data, but can be far more rewarding than naive key/value storage.
]]>Unlike Memcached, which is multi-threaded, Redis only runs a single thread per process. Considering the brute speed of Redis a single process seems like plenty for many workloads. That's until your platform traffic starts rising, background jobs are firing continuously, pub/sub channels are relaying thousands of payloads over the network and the cache is being hit continuously. Each request to Redis is blocking, which can throw off the timing of background jobs or be an outright bottleneck for a set of load balanced servers.
Configure multiple separate instances of Redis to alleviate pressure on a single process. Separate instances by workload: one for background jobs, another for pub/sub and another dedicated to caching. Don't rely on partitioning data into multiple Redis databases! Each of those databases is still backed by a single process so all of the same caveats apply.
/0
, /1
, /2
) to partition workloads.You should care about persistence and replication, two features only available in Redis. Even if your goal is to build a cache it helps that after an upgrade or a reboot your data are [sic] still there.
Each Redis instance has its own configuration file and can be tuned according to the use-case. Caching servers, for example, can be configured to use RDB persistence to periodically save a single backup instead of AOF persistence logs. By only taking periodic snapshots of the database RDB maximizes performance at the expense of up-to-the-second consistency. For a hybrid Redis instance that may be storing business critical background jobs data consistency is paramount. With a cache it is alright to lose some data in the event of a disaster, after reboot most of the cache will be warm and intact.
stop-writes-on-bgsave-error
to no
to prevent all writes from
failing when snapshotting fails. This requires proper monitoring and alerts
for failures, which you are doing anyhow, right?Once you have a Redis instance dedicated to caching you can start to optimize memory management in ways that don't make sense for a hybrid database. When ephemeral and long-lived data is co-mingled it is imperative that ephemeral keys have a TTL and Redis is free to clean up expired keys.
Redis can manage memory in a variety of ways. The management
policies vary from never evicting keys (noeviction
) to randomly evicting a key
when memory is full (allkeys-random
). Hybridized databases typically use
volatile-*
policies, which require the presence of expiration values or they
behave identically to noeviction
. There is another policy that works better
for cache data, allkeys-lru
. The allkeys-lru
policy attempts to remove the
less recently used (LRU) keys first in order to make space for the new data
added.
It is also worth to note that setting an expire to a key costs memory, so using a policy like
allkeys-lru
is more memory efficient since there is no need to set an expire for the key to be evicted under memory pressure.
Redis uses an approximated LRU algorithm instead of an exact algorithm. What
this means is that you can conserve memory in favor of inaccuracy by tuning the
number of samples to check with each eviction. Set maxmemory-samples
to a low
level, say around 5, for "good enough" eviction with a low memory footprint.
Lastly, and most importantly, set a maxmemory
limit to a comfortable amount of
RAM. Without a limit Redis can't function properly as a LRU cache and will start
replying with errors when memory consuming commands start failing.
maxmemory
limit.allkeys-lru
policies for dedicated cache instances. Let Redis manage
key eviction by itself.expire
for keys, it adds additional memory overhead per key.Because of Redis data structures, the usual pattern used with memcached of destroying objects when the cache is invalidated, to recreate it from the DB later, is a primitive way of using Redis.
Only storing serialized HTML or JSON as strings, the standard way of caching for web applications, doesn't fully utilize Redis as a cache. One of the great strengths of Redis over Memcached is the rich set of data structures available. Ordered lists, structured hashes, and sorted sets are particularly useful caching tools only available through Redis. Caching is more than stuffing everything into strings.
Let's look at the Hash type for a specific example.
Small hashes are encoded in a very small space, so you should try representing your data using hashes every time it is possible. For instance if you have objects representing users in a web application, instead of using different keys for name, surname, email, password, use a single hash with all the required fields.
Instead of storing objects as a serialized string you can store the object as fields and values available through a single key. Using a Hash saves web servers the work of fetching an entire serialized value, de-serializing it, updating it, re-serializing it, and finally writing it back to the cache. Eliminating that flow for every minor update pushes the work into Redis and out of your applications, where it is supposed to be.
list
, set
, zset
,
hash
).string
type for structured data, reach for a hash
.Happy optimizing. Go forth and cache!
]]>Redis is blazingly fast, amazingly versatile and its use is virtually ubiquitous among Rails apps. Typically it's being leveraged for background job processing, pub/sub, request rate limiting, and all manner of other ad-hoc tasks that require persistence and speed. Unfortunately, its adoption as a cache has lagged in the shadow of Memcached, the longstanding in-memory caching alternative. That may be due to lingering views on what Redis's strengths are, but I believe it comes down to a lack of great libraries. That's precisely what led to writing Readthis, an extremely fast caching library for Ruby and backed by Redis.
Before diving into project goals and implementation details let's look at a
chart comparing the performance of multi
cache operations across varying cache
libraries. Multi, or pipelined, read/write operations are particularly valuable
for caching with API requests, and an excellent example of where Readthis's
performance excels:
The multi benchmark can be found in the Readthis repository.
The only store faster than Readthis is ActiveSupport's in memory storage, which isn't persisted to a database at all. Throughout the rest of this post we'll look at the high level goals that made this performance possible, and examine some of the specific steps taken to achieve it.
Writing a new implementation of existing software begins with setting high level goals. These goals establish how the library will be differentiated from the alternatives and provide some metrics of success. As there was already a Redis backed cache available in redis-store, and an extremely popular Memcached library in dalli, setting the initial goals was quite straight forward.
ActiveSupport
beast while still supporting integration points with Rails apps.redis-activesupport
but the pull requests languished for months while
compatibility drifted away from releases of Rails.Once the initial library structure was in place a small suite of benchmark scripts were created to measure performance and memory usage. As features were added or enhanced the scripts were used to identify performance bottlenecks, while also ensuring there weren't any performance regressions.
The initial benchmark results can be broken down into three distinct bottlenecks: round trips to Redis, marshaling cached data and cache entry creation. Though there were also other micro-optimizations that presented themselves, these three areas provided the most obvious gains.
Redis is extremely fast, but no amount of speed can compensate for wasting time
with repeated calls back and forth between an application and the database. The
round-trip back and forth wastes a lot of CPU time and instantiates additional
objects that will need to be garbage collected eventually. Redis provides
pipelining via the MULTI
command for exactly this situation.
Readthis uses MULTI
to complete data setting and retrieval with as few
transactions as possible. Primarily this benefits "bulk" operations such as
read_multi
, fetch_multi
, or the Readthis specific write_multi
. For fetch
operations where values are written only when they can't be retrieved, reading
and writing of all values is always performed with two commands, no matter how
many entries are being retrieved.
The most significant gains to pipelining and round-trip performance came through the use of hiredis. Hiredis is a Redis adapter written in C that drastically speeds up the parsing of bulk replies.
require 'bundler'
Bundler.setup
require 'benchmark/ips'
require 'readthis'
REDIS_URL = 'redis://localhost:6379/11'
native = Readthis::Cache.new(REDIS_URL, driver: :ruby, expires_in: 60)
hiredis = Readthis::Cache.new(REDIS_URL, driver: :hiredis, expires_in: 60)
('a'..'z').each { |key| native.write(key, key * 1024) }
Benchmark.ips do |x|
x.report('native:read-multi') { native.read_multi(*('a'..'z')) }
x.report('hiredis:read-multi') { hiredis.read_multi(*('a'..'z')) }
x.compare!
end
Once you eliminate time spent retrieving data over the wire it becomes clear that most of the wall time is spent marshaling data back and forth between strings and native Ruby objects. Even when a value being cached is already a string it is still marshaled as a Ruby string:
Marshal.dump('ruby') #=> "\x04\bI\"\truby\x06:\x06ET"
For some caching use cases, such as storing JSON payloads, it simply isn't
necessary to load stored strings back into Ruby objects. This insight provided
an opportunity to make the marshaller plug-able, and even bypass serialization
entirely, yielding a significant performance boost. In some implementations,
such as Dalli's, a raw
option may be set to bypass entry serialization as
well, but the option is checked on every read or write and doesn't provide any
additional flexibility.
Let's look at the script used to measure marshal performance. It illustrates
that configuring the marshaller is as simple as passing an option during
construction. Any object that responds to both dump
and load
may be used.
require 'bundler'
Bundler.setup
require 'benchmark/ips'
require 'json'
require 'oj'
require 'readthis'
require 'readthis/passthrough'
REDIS_URL = 'redis://localhost:6379/11'
OPTIONS = { compressed: false }
readthis_pass = Readthis::Cache.new(REDIS_URL, OPTIONS.merge(marshal: Readthis::Passthrough))
readthis_oj = Readthis::Cache.new(REDIS_URL, OPTIONS.merge(marshal: Oj))
readthis_json = Readthis::Cache.new(REDIS_URL, OPTIONS.merge(marshal: JSON))
readthis_ruby = Readthis::Cache.new(REDIS_URL, OPTIONS.merge(marshal: Marshal))
HASH = ('a'..'z').each_with_object({}) { |key, memo| memo[key] = key }
Benchmark.ips do |x|
x.report('pass:hash:dump') { readthis_pass.write('pass', HASH) }
x.report('oj:hash:dump') { readthis_oj.write('oj', HASH) }
x.report('json:hash:dump') { readthis_json.write('json', HASH) }
x.report('ruby:hash:dump') { readthis_ruby.write('ruby', HASH) }
x.compare!
end
Benchmark.ips do |x|
x.report('pass:hash:load') { readthis_pass.read('pass') }
x.report('oj:hash:load') { readthis_oj.read('oj') }
x.report('json:hash:load') { readthis_json.read('json') }
x.report('ruby:hash:load') { readthis_ruby.read('ruby') }
x.compare!
end
The results, in prettified chart form:
This benchmark demonstrates the relative difference between serialization modules when working with a small hash. Performance varies for other primitives, such as strings, but the pass-through module is always fastest for load operations. This makes sense as there aren't any additional allocations being made, the string that is read back from Redis is returned directly.
When you can get away with it, which is any time you're only working with
strings, the pass-through module provides an enormous boost in load performance.
Otherwise, if you are only working with basic types (strings, arrays, numbers,
booleans, hashes) then there are gains to be made with Oj, particularly
during dump
operations.
All of the caches built off of ActiveSupport::Cache
rely on the Entry
class
for wrapping values. The Entry
class provides a base for serialization,
compression, and expiration tracking. Every time a new value is read or written
to the store a new entry is initialized for the value.
When working with Redis not all of the entry class's functionality is necessary.
For instance, some stores, such as FileStore
or MemoryStore
require
per-entry expirations to evict stale cache entries. Redis has built in support
for expiration and can avoid wrapping individual entries. By not wrapping each
cache entry Readthis can use pure methods and avoid instantiating additional
objects.
In synthetic benchmarks the performance gains are negligible (and it makes for a very boring chart), but there is a direct reduction in the number of objects allocated. That savings adds up across thousands of requests, aiding in fewer GC pauses.
Ignoring the implementation details between Redis and Memcached there isn't anything preventing other caches from benefiting from these techniques. Everybody benefits from healthy competition. There is always room for improvement and I hope to see Readthis pushed further.
Use Redis for your next project and give Readthis a try!
]]>Fire up a PostgreSQL REPL connected to your development environment and get to work on a rough signup funnel.
psql widgets_development
Your platform lets account owners create and launch new embeddable widgets. An important measure of success is how many new accounts then go on to create and launch one or more widgets. You write up the funnel bundling account owners into monthly cohorts:
WITH widgeters AS (
SELECT id,
date_trunc('month', created_at) AS created_at,
NULLIF(widgets_count, 0) AS widgets_count,
(SELECT NULLIF(COUNT(*), 0)
FROM widgets
WHERE widgets.account_id = accounts.id
AND widgets.launched_at IS NOT NULL) AS launched_count
FROM accounts
AND created_at > current_timestamp - '1 year'::interval
)
SELECT created_at,
COUNT(*) AS signed_up,
COUNT(widgets_count) AS made_widget,
COUNT(launched_count) AS launched_widget
FROM widgeters
GROUP BY created_at
ORDER BY created_at;
That works. The data looks right to you. Now it is time to export it. No
problem. Modify the query by wrapping it in a COPY
statement:
COPY (
-- previous query
) TO '/tmp/cohorts.csv' WITH CSV HEADER;
That outputs a perfectly formatted CSV onto the local file system. After taking
a quick glance at the output you notice that your development data isn't
remotely accurate. It is mangled and out of date. Grabbing data from production
would be much more useful. Again, no problem, turn the query into a .sql
file
that can be run against the remote database.
psql "postgres://username:password@widgets-server/widgets_production" -f cohorts.sql
Hmm, PostgreSQL didn't like that. It informs you that only root
users can
export to the file system. You would have to scp
the file back to your local
machine anyhow. Fortunately, the COPY
command can also output to STDOUT
.
Update the output in cohorts.sql
:
COPY (
-- previous query
) TO STDOUT WITH CSV HEADER
Now the results can be piped into a local file directly:
psql "postgres://username:password@widgets-server/widgets_production" \
-f cohorts.sql \
> cohorts-production.csv
As expected, operations liked the export. Now they want another one, but they
want to be able to generate it themselves right from the admin section.
Assuming the server is running Ruby (a safe assumption for the time being), we
may be tempted to reach for the CSV library. After all, it is right there in the
standard library. However, that may require rewriting our perfectly functional
query in ActiveRecord
. It would also involve pulling all of the data back from
the server and instantiating objects for every column and row. There is a better
way!
Every database connection adapter has the means to execute a SQL query and read
back the results. Ruby's pg
adapter is no exception. With a very thin wrapper
around a database connection you can generate exports from the same query file
right on the server.
First, the wrapper class:
class PostgresCSVWriter
attr_reader :adapter
def initialize(adapter = ActiveRecord::Base)
@adapter = adapter
end
def connection
adapter.connection.instance_variable_get('@connection')
end
def write_rows(query, io: '')
connection.copy_data(build_copy_query(query)) do
while row = connection.get_copy_data
io << row
end
end
end
private
def build_copy_query(query)
%(COPY (#{query}) TO STDOUT WITH DELIMITER ',' CSV HEADER)
end
end
Please note that while the query itself is interpolated, it is not escaped, and is therefore vulnerable to injection attacks. This is not suitable for user generated queries.
With the base writer in place you can now write an exporter that injects an
existing .sql
query and hands back a string suitable for streaming back to the
client.
require 'postgres_csv_writer'
class PostgresCSVExporter
def export(filename)
writer.write_rows(query(filename))
end
def query(filename)
IO.read(Rails.root.join('queries', filename))
end
private
def writer
PostgresCSVWriter.new
end
end
Exports are now as simple as PostgresCSVExporter.new.export('cohorts.sql')
.
Every export lives in its own .sql
file which can be edited and executed
natively. For larger exports, this approach will build up multi-megabyte
strings, which is probably undesirable. In those situations you may reach for
Tempfile
and send the file contents rather than a large string.
Now you can keep all of your queries written in completely portable SQL. The queries can be accessed from any other tech stack without the need for an ORM or an intermediate representation. Let the team know their exports are now just one click away.
]]>Recently I led a team that overhauled a client side experience to handle a continuous flow of data from everybody participating in the project. Speed, consistency and reliability were critical for the new implementation's success. Naturally we turned to Pusher, but we didn't get what we were looking for.
Of course we tried the existing industry standard solution first. Configuration and setup was simple enough, but it quickly fell over with even a modest workload. What follows are only a few of the problem areas that we encountered.
Broadcasting events would often timeout after 5 seconds. More accurately, "often" means up to 30% of the time. To combat unpredictability and latency, events were broadcast in the background and automatically retried. Automatic retries may assuage timeouts over time, but they introduce rampant race conditions. Imagine a scenario where an event that added some data fails to send but the subsequent event that removes that data sends immediately?
Any payload over a seemingly arbitrary threshold of 10 kilobytes could not be delivered. It was quite common that a JSON payload included a lot of text, lengthy URLs, or numerous associations for sideloading. Engineering solutions to this problem such as compressing data or only sending a delta are possible, but neither are foolproof and introduce more complexity.
All developers are prone to bouts of NIH Syndrome. Surely our team can implement a websocket solution ourselves?
Websockets and MRI simply don't play well together. Support for Rack Hijack is spotty and only works with certain servers. Even with hijack support working you won't scale a threaded server like Puma up to thousands of concurrent connections. The Faye project and related libraries provide excellent tooling around websockets, but it won't work with Unicorn and provides no abstractions or instrumentation at all.
Jumping to another stack, such as Node.js or Erlang, is tricky enough by itself. On top of the issues with building out a relay you need to support additional servers, additional deployments, some sort of pub/sub or message broker. That is a lot of added complexity to distract your team from building your primary product.
Websockets enforce security policies. Yes, it is a bad idea to send insecure data from a secure client, fortunately it isn't even possible. That means the real-time server needs to handle SSL connections, adding another layer of complexity. Node isn't natively able to handle secure connections. That leaves a solution like stunnel or nginx to terminate SSL, making configuration even more complex. Additionally cross domain policies mandate a wildcard certificate or additional CORS setup.
Without additional engineering effort all messages within the system are zipping around within a black box. There isn't any instrumentation on connections or performance. Tracking connection activity and messaging is just as important as monitoring HTTP traffic. Now it's time to get statsd involved too!
Building and maintaining your own solution is unquestionably the most expensive way to tackle the issue. The cost of a single developer (one who has worked on this exact problem before and knows precisely what to build) greatly exceeds subscription fees to an outside service for years. No suitable service or stack existed when I went through all of these steps.
That is why we're introducing Snö, a reliable platform for websites and apps that need real time messaging. Please take a look. If you like what you see sign up for the waitlist, we'll let you know how it progresses.
]]>Each production server should only have one job, be it running a load balancer, serving up static pages or working as a database. Realistically that isn't always the case, staging servers are a notable exemption, but it is an attainable goal. Every component in the stack should rely on the init system to maintain a steady state. Chances are the load balancer, reverse proxy cache, NoSQL server, SQL server or configuration registry is already being managed by an init system. The application should too.
This post assumes you are deploying to Ubuntu, though the same principles apply to nearly any other *nix system. The current service management system for Ubuntu is Upstart, though it is being phased out in favor of the controversial RedHat driven systemd. Regardless, Upstart is included in Ubuntu 14.04 LTS, so it will be around for at least another four years.
The Upstart Cookbook is your best friend when crafting upstart configuration files. Don't be intimidated by the cookbook's massive length. While searching around for specific details you'll learn of other useful features that you didn't even know existed.
The least common denominator for any web application is the server, so that is what we will look at setting up as a service. Below is a configuration file for running the Puma web server as a service. Most of the details are common to any upstart script, and in fact much of this configuration is straight out of the example from the Puma repository:
#!upstart
description "Puma Server"
setuid deploy
setgid deploy
env HOME=/home/deploy
reload signal USR1
normal exit 0 TERM
respawn
respawn limit 3 30
start on runlevel [2345]
stop on runlevel [06]
script
cd /var/www/app/current
exec bin/puma -C config/puma.rb -b 'unix:///var/run/puma.sock?umask=0111'
end script
post_script exec rm -f /var/run/puma.sock
There are a couple of important changes and additions to the configuration that I'll point out, as they are crucial for service maintainability.
setuid deploy
setgid deploy
First, drop down to a less priveleged user for the sake of security. This is a
very helpful feature built into more recent versions of Upstart. Your service
simply should not need to run as root. Some sudo
level commands are necessary
for service control, but they should be enabled within sudoers
, as we'll look
at later.
reload signal USR1
normal exit 0 TERM
Use an alternate reload signal. The standard signal emitted to the process is
HUP
, which tells a process to reload its configuration file. Puma, like some
of the other web servers, can perform a full code reload and hot restart when
sent a particular signal. Here we are hijacking the upstart reload
event to
send Puma the USR1
signal, triggering a phased restart. Part
of the phased restart process involves sending the TERM
signal, which we tell
upstart to ignore. Without the normal exit
directive Upstart would consider
the Puma process down after one reload.
respawn
respawn limit 3 30
Add a respawning directive. It will try to restart the job up to 3 times within a 30 second window if it fails for some reason. More often than not, the service simply isn't coming back. It's nice to have a backup.
start on runlevel [2345]
Automatic start is one of the strongest selling points for using an init system for an application. If the VM is mysteriously rebooted by your hosting provider, which is guaranteed to happen at some point, it will be brought right back up when the VM boots.
exec bin/puma -C config/puma.rb -b 'unix:///var/run/puma.sock?umask=0111'
The final line of the script
block determines which process will be tracked by
upstart. While that may seem obvious, there are some gotchas to be aware of. By
default upstart pipes STDOUT
and STDERR
to /var/log/upstart/puma.log
,
which is convenient. If you decide that you'd prefer to log directly to syslog
you may be tempted to add a pipe:
exec bin/puma ... | logger -t puma
However, that causes upstart to track the logger process's PID instead of
Puma's, preventing any further control of the Puma process by upstart. As you
would soon discover, attempts to sudo stop puma
would only stop the logger
process and leave a zombie Puma process running in the background. Tracking the
proper PID is also crucial for the next stage of managing applications as
services, service monitoring.
By placing the configuration file in the proper location we can use service
commands to control the server process. Write the file to /etc/init/puma.conf
.
All configuration files go into etc/init/
, and the service becomes available
as whatever the file is named.
With the configuration in place the server can start up:
sudo service puma start
Even though the process will be ran as the deploy
user the service must be
controlled with sudo
. This can be problematic when using a deployment tool
like Capistrano, which doesn't officially support running commands as sudo
. In
order for all of the necessary job control to be available during deployment you
will need to configure the deploy
user with proper sudoer
permissions.
Playing with passwordless sudo
can be dangerous, so only add an exemption for
controlling the puma process directly:
sudo echo "deploy ALL = (root) NOPASSWD: /sbin/start puma, /sbin/stop puma, /sbin/restart puma, /sbin/reload puma" >> /etc/sudoers
The various service commands (start, stop, restart, and reload) are all aliased
into /sbin
. This makes the passwordless commands slightly more readable, but
is functionally equivalent to the service {name} {action}
version.
Now the service is up and the init system will ensure it comes back up if the system crashes, or even if the process itself crashes. But what happens if the process itself misbehaves or starts syphoning too many resources? There are tools for just that situation, of course.
Utilities for monitoring a server and the services on that server are essential to maintaining the health of a system. Many systems in the Ruby world have relied on tools like God or Bluepill to monitor and control application state. Those particular tools have a couple of large drawbacks though. Notably they require a Ruby runtime, which reduces portability and sacrifices stability when version management is involved. More importantly, instead of working with an existing init system they duplicate the functionality.
A recently released monitoring tool called Inspeqtor addresses both of the aforementioned issues. It is distributed as a small self-contained binary that itself is managed by an init system. However, it doesn't get into the business of trying to control services directly. Instead, it leverages the init system and very concise configuration files to help the system manage services directly. Installation is simple and works with the existing package manager.
Continuing on with the goal of keeping the system up, self-healing, and allowing
the init system to do our work for us here is an example configuration file for
Puma. It is targeting the Puma service specifically, and would be placed in
/etc/inspeqtor/services.d/puma.inq
:
check service puma
if cpu:total_user > 90% then alert
if memory:total_rss > 2g then alert, reload
That outlines, in very plain language, how Inspeqtor will monitor the service.
It will find the init system that is managing the process and periodically
perform some analysis on it. It performs simple status checks, such as whether
the service is even up currently, and can alert you if the service goes down.
Deeper introspection into resource usage is also possible, as shown in the
example above. Experience tells us that a Ruby web server will suffer memory
bloat over time and we'll want to track it. When the memory passes a threshold
Inspeqtor will take action. In this case it will tell Upstart to reload puma
(the same as running service puma reload
) and it will send an alert to any of
the configured channels such as email or Slack.
Some services, such as Sidekiq workers for example, may not have such
strident requirements on uptime or may not have any notion of "phased restart".
In that case the config can use restart
in place of reload
.
Make the most of the tools that are available to you. Some of them, such as Upstart, can be leveraged to great effect with a tiny bit of configuration and some outside monitoring. Converting a system from a set of custom deployment recipies that manage logs, sockets and pid files to one that manages and maintains itself will be vastly more stable and predictable.
]]>Apps sometimes store config as constants in the code. This is a violation of twelve-factor, which requires strict separation of config from code. Config varies substantially across deploys, code does not.
In the Ruby world the Dotenv library makes it simple to dynamically
load configuration from values stored in local .env
files. Early in the
loading process the file is read and each key value pair is loaded into Ruby's
hash-like ENV
object. A common, and simple, example of using environment
variables is storing the URL, credentials, and configuration for a database
connection:
DATABASE_URL=postgres://username:password@localhost:port/database?pool=16
The details of the connection are confidential and shouldn't be checked into
source control. An .env
file can be managed independently of the source code
and transferred to the web server securely, even as part of the deploy process.
This method of providing database configuration is so common that Rails will
check the ENV
for a DATABASE_URL
when it boots. This built in usage of
environment variables is great, but there are some caveats.
Those familiar with Heroku know that when you change an environment variable on Heroku, no matter how small, the application will be restarted. In the land of Heroku a restart means creating all new containers for your application, starting them up, and finally routing traffic to them once they have loaded. Complete shutdown and startup is consistent, but has noticeable lag when compared to the hot reloading available in Unicorn or Puma.
Unicorn servers achieve concurrency by running one or more workers, each
controlled from a single master process. The master process listens for Unix
signals such as TERM
, QUIT
, or USR2
and manages the pool of
workers accordingly. For example, when the master receives a USR2
signal it
forks new worker instances with the most recent version of the code and begins
directing connections to the new instances. This is called a phased restart.
Starting a new Unicorn master forks it from the current process, typically a shell of some kind. During the forking process it inherits the shell's environment variables. After the process is forked it loses any reference to the shell's environment, so any further changes to the environment will be ignored. This separation prevents chaos between different processes, but it also creates a hiccup when we want to update the configuration for a long running process like Unicorn.
Updating environment variables from a configuration file can be performed at any
time with ENV.update
. Calling update will add or replace any existing keys
with the new values, but only within the current process. In order to have the
updated ENV
cascade down to the workers actually handling requests we have to
call update
before the workers are forked. It is very common to perform some
setup around the exec/fork life cycle, so servers provide life cycle hooks. Here
is an example of how to update within a Unicorn config:
require 'dotenv'
before_exec do
ENV.update Dotenv::Environment.new('.env')
end
Or, alternately, within a Puma config:
require 'dotenv'
on_worker_boot do
ENV.update Dotenv::Environment.new('.env')
end
With the configuration hooks in place you can safely update a .env
file at any
time, issue a restart, and change configuration on the fly.
As this is a common problem for social websites there are a number of existing examples for how to calculate the hotness or ranking of posts. We're going to evaluate a hybrid version of some of these algorithms, and see how they need to be modified to get the most out of our database.
The popularity of a post can be calculated rather simply. In our example it is the sum of the comments and likes counts. Expressed in Ruby:
def popularity(post)
post.comments_count + post.likes_count
end
That will get you a raw score, which is useful if you are sorting items by
popularity alone. If you are combining popularity with another measurement, such
as recentness, you will want a means of weighting the output. By weighting the
output with a multiplier, we can compose multiple values more evenly. Here is
the popularity
method rewritten to accept a pure number and apply the weight:
def popularity(count, weight: 3)
count * weight
end
popularity(1) # 3
popularity(25) # 75
The new popularity
method is simplistic, but documents the code better than a
group of magic numbers. The need for weighting will become more apparent after
we look at the other important factor in computing a post's trending arc,
recentness.
Recentness is a measure of how old something is, relative to an absolute point in time. There are two opposing methods for tracking recentness: decreasing the value over time or increasing the value over time. In the ranking algorithm used on Hacker News posts decrease their "recentness" value over time. Reddit, on the other hand, perpetually increases the score for new posts. Decreasing recentness over time seems intuitive, but we'll favor the perpetual increase approach for reasons that will be apparent later.
Here is a basic recentness method in Ruby:
SYSTEM_EPOCH = 1.day.ago.to_i
SECOND_DIVISOR = 3600
def recentness(timestamp, epoch: SYSTEM_EPOCH, divisor: SECOND_DIVISOR)
seconds = timestamp.to_i - epoch
(seconds / divisor).to_i
end
recentness(1.hour.ago) # 23
recentness(12.hours.ago) # 12
recentness(1.day.ago) # 0
Using an absolute epoch
of just a day ago makes the increments very clear.
Each new post will have a higher score than an older post, staggered by one hour
windows.
Combining the recentness and the popularity yields a composite trending score Let's try it out purely in Ruby land, ignoring any pesky database details:
Post = Struct.new(:id, :created_at, :likes_count, :comments_count)
posts = [
Post.new(1, 1.hour.ago, 1, 1),
Post.new(2, 2.days.ago, 7, 1),
Post.new(3, 9.hours.ago, 2, 5),
Post.new(4, 6.days.ago, 11, 3),
Post.new(5, 2.weeks.ago, 58, 92),
Post.new(6, 1.week.ago, 12, 7)
]
sorted = posts.map do |post|
pop = popularity(post.likes_count + post.comments_count)
rec = recentness(post.created_at, epoch: 1.month.ago.to_i)
[pop + rec, post.id]
end.sort_by(&:first)
sorted.reverse # [[608, 6], [617, 4], [695, 2], [724, 1], [731, 3], [833, 5]]
That looks about right. The really popular post from two weeks ago is hanging around at the top, but newer and less popular posts are up there as well. This example demonstrates how the ranking should work, but it glosses over one crucial aspect of how applications really operate. The entire history of posts aren't sitting around the server in memory. They are stored in a database where our Ruby methods won't be of any use. Loading thousands, or millions, of posts into memory to sort them is ludicrous. So, how can we move our ranking to the database?
All of the mainstream SQL databases (MySQL, SQL Server, Oracle, Postgres) have
the notion of custom procedures, or functions. Postgres, what we'll be focusing
on here, has particularly great support for defining custom functions. It allows
custom functions to be written in SQL
, C
, an internal representation, or any
number of user-defined procedural languages such as JavaScript
.
Postgres functions can be defined in isolation and labeled with
a hint for the query optimizer about the behavior of the function. The behaviors
range from VOLATILE
, in which there may be side-effects and no optimizations
can be made, to IMMUTABLE
where the function will always return the same
output given the same input. Beyond an academic love of "functional purity"
there is one killer benefit to using IMMUTABLE
functions: they can be used for
indexing. That feature can also be a constraint when dealing with notions of
time, as we'll see shortly.
First, let's translate the popularity method from Ruby. Connect to a database
with the psql
client execute the following:
CREATE FUNCTION popularity(count integer, weight integer default 3) RETURNS integer AS $$
SELECT count * weight
$$ LANGUAGE SQL IMMUTABLE;
The declaration is a bit more verbose than the Ruby version. It requires that you specify the input types, but it even allows default values. We can easily verify the output matches the Ruby version:
SELECT popularity(1); -- 3
SELECT popularity(25); -- 75
The recentness function proves to be a little trickier. Postgres has a lot of
facilities for manipulating timestamps and it can take a while to
find the invocation that gets the exact value you need. In this case we are
trying to mimic Ruby's Time#to_i
method. The EXTRACT
function, when combined
with EPOCH
, does just that. It converts the timestamp into an integer. Here is
the translated recentness function:
CREATE FUNCTION recentness(stamp timestamp, sys_epoch integer default 1388380757) RETURNS integer AS $$
SELECT ((EXTRACT(EPOCH FROM stamp) - sys_epoch) / 3600)::integer
$$ LANGUAGE SQL IMMUTABLE;
Plugging in the same output shows that it matches:
SELECT recentness((now() - interval '1 hour')::timestamp); -- 23
SELECT recentness((now() - interval '12 hours')::timestamp); -- 12
All that is left is to combine the two scores into a single value:
CREATE FUNCTION ranking(counts integer, stamp timestamp, weight integer) RETURNS integer AS $$
SELECT popularity(counts, weight) + recentness(stamp)
$$ LANGUAGE SQL IMMUTABLE;
Imagine a posts
table that is similar to the Ruby Struct that was used above:
| Column | Type |
+----------------+-----------+
| id | integer |
| title | text |
| comments_count | timestamp |
| likes_count | timestamp |
| created_at | timestamp |
| updated_at | timestamp |
We can rank the trending posts easily with an ORDER BY
clause that uses the
trending function:
SELECT * FROM posts
ORDER BY ranking(
comments_count + likes_count,
created_at::timestamp
) DESC LIMIT 30;
As expected the ordering works! Trending ranking has been successfully moved to
the database. There is a little matter of performance, however. On my local
machine with a table of roughly 210,000 posts this query takes ~289.049 ms—not
speedy by any measure. For comparison, running a similar query that orders only
by id
takes ~0.428 ms. That is over 390x; faster. We can still do better.
The first step to improving any query is understanding what the query planner is
doing with it. All it takes is prepending an EXPLAIN
clause to our original
query:
EXPLAIN SELECT * FROM posts -- ...
The output clearly identifies an insurmountable cost in the sorting phase:
-> Sort (cost=69179.55..69689.03 rows=203790 width=229)
This is where indexes come to the rescue. There are three separate columns from each row used to compute the trending score. Adding an index to any column would help when sorting by columns individually or in tandem, but it doesn't help when sorting by an aggregate—such as a function. Postgres will always need to compute the trending score for every row in order to sort them all.
From the index ordering documentation:
An important special case is ORDER BY in combination with LIMIT n: an explicit sort will have to process all the data to identify the first n rows, but if there is an index matching the ORDER BY, the first n rows can be retrieved directly, without scanning the remainder at all.
Clearly we need an index that matches the ORDER BY
statement.
Earlier I mentioned that there was an important property of the IMMUTABLE
function attribute. Because an immutable function is guaranteed not to alter any
external values it can safely be used to generate an index. Back in the psql
console try adding a new index:
CREATE INDEX index_posts_on_ranking
ON posts (ranking(comments_count + likes_count, created_at::timestamp) DESC);
It doesn't work! In fact, it throws an error:
ERROR: 42P17: functions in index expression must be marked IMMUTABLE
The error is raised despite our marking the ranking function and its sub
functions as IMMUTABLE
. What gives? Recall that earlier I mentioned that there
was a problem with immutability and timestamps. Time is a fluid concept, and is
naturally subject to change. Running the function at different times, in
different timezones, or even on different servers will yield a different result.
Any function that deals with a timestamp can not truly be immutable.
There is a simple workaround to the timestamp blocker. The integer value of a timestamp is just an increasing counter, ticking off each microsecond. The database itself has another form of increasing counter, the primary key. We can replace the timestamp with the primary key for the "recentness" calculation:
DROP FUNCTION recentness(timestamp, integer);
DROP FUNCTION ranking(integer, timestamp);
CREATE FUNCTION ranking(id integer, counts integer, weight integer) RETURNS integer AS $$
SELECT id + popularity(counts, weight)
$$ LANGUAGE SQL IMMUTABLE;
Now try adding the index again:
CREATE INDEX index_posts_on_ranking
ON posts (ranking(id, comments_count + likes_count, 3) DESC);
The index is accepted. Now we can try it out again:
SELECT * FROM posts
ORDER BY ranking(id, comments_count + likes_count) DESC LIMIT 30;
The results come back in ~0.442ms, just as fast as the id
only ordering. The
final ranking function is entirely trivial. New posts will slowly fall off as
new posts are added and get boosted by social activity. It has the exact effect
we aimed for and is hundreds of times faster! Granted, the simplicity comes at
the cost of being unable to fine tune the results—nothing that a little
weighting or logarithmic clamping can't fix. That is an exercise left up to the
reader.
require 'delegate'
Model = Struct.new(:id, :title)
class Presenter < SimpleDelegator
def slug
title.downcase.gsub(/\s+/, '-')
end
end
model = Model.new(100, 'Presenters Rule')
presenter = Presenter.new(model)
presenter.title # 'Presenters Rule'
presenter.slug # 'presenters-rule'
Implementing the presenter pattern in Ruby is almost free because of the
Delegate module from the standard library. It is a fairly thin wrapper
around Ruby's dynamic method dispatch, method_missing
. Whenever an unknown
method is sent to a delegate it dynamically checks whether the object it is
delegating to has that method and calls that instead. In the instance above the
delegator is being elevated to a presenter by manipulating the data
slightly.
What about languages that don't have method_missing
? One such language is
JavaScript. There isn't any standard tracked method for handling method
calls dynamically. Fortunately, JavaScript is highly malleable, allowing for
dynamic assignment instead. Let's take a shot at a JavaScript presenter:
var Presenter = function(model) {
this.model = model;
for (key in model) {
if (model.hasOwnProperty(key) && !this.hasOwnProperty(key)) {
this[key] = model[key]
}
}
}
Presenter.prototype.slug = function() {
return this.title.toLowerCase().replace(/\s+/, '-');
};
model = { id: 100, title: 'Presenters Are Fun!' };
presenter = new Presenter(model);
presenter.title // 'Presenters Are Fun'
presenter.slug() // 'presenters-are-fun'
That was a bit more work wasn't it? Not only that, it has some major pitfalls.
First, there is the issue of uniform access principal. In Ruby every
message sent to an object with .
is a method call, whether that particular
method returns a static value or is a proper method definition. That isn't the
case in JavaScript. Calling a method on an object with .
will always yield the
value of that object. That means if the value of an object is a function you'll
get a [Function]
reference back, not the evaluated function. This is evident
in the call to presenter.slug()
above—it required the trailing ()
to invoke
the function, whereas the call to presenter.title
did not.
There is also the issue of duplicating all of the data from the model onto the presenter. For trivial applications or models with only a few attributes duplication isn't much of an issue. When you have hundreds or thousands of models with nested objects or sizable data the duplication is entirely wasteful. In addition, as soon as the model's data changes the presenter will be out of sync. Referencing the example above:
console.log(model.id, presenter.id); // 100, 100
model.id = 101;
console.log(model.id, presenter.id); // 101, 100
It turns out that ES6 Harmony proposes a clean solution to our presenter
problem. As of Firefox 18.0, Chrome 24.0 there is a new Proxy API, allowing
objects to be created and have properties computed at runtime. This is an ideal
tool for a presenter. Here is a simple example of how the Proxy
object
behaves:
var handler = {
get: function(target, name) {
return name in target ? target[name] : 'missing';
}
}
var data = { id: 100 },
proxy = new Proxy(data, handler);
console.log(proxy.id); // 100
console.log(proxy.title); // 'missing'
Three objects are interacting together here: a data object, a handler, and the
proxy itself. There is a wealth of what are called traps
available to the
handler object. The example above uses the get
trap to determine how to
respond to property access. This is exactly what we need for a proper presenter!
var Presenter = {
present: function(model) {
return new Proxy(model, this.handler);
},
handler: {
get: function(target, name) {
var value = name in target ? target[name] : this[name];
return typeof value == 'function' ? value(target) : value;
},
slug: function(target) {
return target.title.toLowerCase().replace(/\s+/, '-');
}
}
};
var model = { id: 100, title: 'Proxy Presenter' },
presenter = Presenter.present(model);
console.log(presenter.id, presenter.slug); // 100, proxy-presenter
model.title = 'Dynamic Presenter';
console.log(presenter.slug); // dynamic-presenter
This version holds all of the benefits of a presenter with none of the drawbacks enumerated before.
get
trap uniformly handles values from the model and methods
from the presenter.Unfortunately, as with any new web technology, there is the adoption hurdle.
Proxy isn't available in many browser's, even in Chrome without explicitly
enabling javascript harmony. Until the shiny future where the vast majority
of browsers support Proxy
you will need to provide a hybridized version using
feature flags. That is precisely what I'll be doing for my MVP needs.
Just recently I revamped Fragmenter, a multipart uploading library that handles storing and reassembling binary data. Fragmenter is designed to work with any web framework, but the most likely targets are Rails applications. Even with such a strong imperative to integration test I still didn't want to test against an entire Rails application.
Simply loading a fresh install of the current Rails (4.0.0 at the time of writing) installs 44 gems, using 33 MB of space, and takes ~1.05 seconds to load:
$ bundle list | wc -l
# 44
$ du -h vendor/ruby/2.0.0 | tail -n 1
# 40M vendor/ruby/2.0.0
$ for i in {1..10}; do time rails r ''; done 2>&1 |\
awk '{ sum += $4 } END { print sum / NR }'
# 1.05
Fragmenter provides two modules for mixing in to classes within an app, one for
models
and one for controllers
. The modules are insular and only rely on
services that Fragmenter provides, they have no reliance on Rails or Railties.
The decision to keep Fragmenter decoupled from Rails was made for ease of use
with other web frameworks, i.e. Sinatra. Decoupling gives the added benefit
of integrating against the most minimal API possible: Any class that can handle
Rack requests and responses.
class UploadsController < ApplicationController
include Fragmenter::Rails::Controller
end
class Resource < ActiveRecord::Model
include Fragmenter::Rails::Model
end
All of the interaction with Fragmenter's mixins are via HTTP, making it ideal for exercising with a request spec. Using Rack Test makes sending requests to a Rack app and making assertions on the response extremely easy. The standard structure of a request spec looks like:
require 'rack/test'
describe 'A Resource' do
include Rack::Test::Methods
let(:app) do
lambda { [200, {}, 'Success!'] }
end
it 'performs a successful GET request' do
get 'http://example.com'
expect(last_response).to eq(200)
expect(last_response.body).to eq('Success!')
end
end
All that Rack::Test expects is an app
method returning an object that adheres
to the Rack interface. In the example above we have a hardcoded lambda
that will always return the same result. To test Fragmenter functionality we'll
replace the lambda with a Rack compatible class that includes Fragmenter's
controller mixin:
require 'fragmenter/rails/controller'
require 'rack/request'
class UploadsApp
include Fragmenter::Rails::Controller
attr_reader :request, :resource
def initialize(resource)
@resource = resource
end
def call(env)
@request = Rack::Request.new(env)
case request.request_method
when 'GET' then show
when 'PUT' then update
when 'DELETE' then destroy
end
end
end
When a Rails controller handles requests it automatically provides the
request
object. Here we must instantiate the request manually, which is very
straight forward. Each of the HTTP verbs is then mapped directly to the
corresponding mixed in method—acting as a micro RESTful router.
Lets write a spec to actually test the request/response cycle for one of the
UploadApp
methods:
require 'fragmenter'
require 'json'
require 'rack/test'
describe 'Uploading Fragments' do
include Rack::Test::Methods
Resource = Struct.new(:id) do
include Fragmenter::Rails::Model
def rebuild_fragments
fragmenter.rebuild && fragmenter.clean!
end
end
let(:resource) { Resource.new(200) }
let(:app) { UploadsApp.new(resource) }
it 'Stores uploaded fragments' do
header 'Content-Type', 'image/gif'
header 'X-Fragment-Number', '1'
header 'X-Fragment-Total', '2'
put '/', file_data('micro.gif')
expect(last_response.status).to eq(200)
expect(decoded_response).to eq(
'content_type' => 'image/gif',
'fragments' => %w[1],
'total' => '2'
)
header 'X-Fragment-Number', '2'
header 'X-Fragment-Total', '2'
put '/', file_data('micro.gif')
expect(last_response.status).to eq(202)
expect(decoded_response).to eq('fragments' => [])
end
The example simulates uploading two distinct parts of a very small gif
and
sets expectations about the responses it gets back. It looks like there is a
lot more going on here, but all of the methods (header
, put
) are still
provided by Rack::Test. The most notable addition is the Resource
class, a
generic model-like class that includes Fragmenter's model mixin.
Running the spec yields an unexpected error:
Failure/Error: put '/', file_data('micro.gif')
NoMethodError:
undefined method `render' for #<UploadsApp:0x007fb96c1c2168>
The render
method is the missing part of the Rails compatibility puzzle. Each
of the controller actions end with a call to render
with some json and a
status code. Looking through the signature for render it is clear that
only need to implement a small part of the functionality to get Rails
compatibility with the UploadsApp
:
require 'fragmenter'
require 'rack/request'
require 'rack/response'
class UploadsApp
# No change to the rest of the class
private
def render(options)
body = if options[:json]
JSON.dump(options[:json])
else
''
end
Rack::Response.new(body, options[:status], {}).finish do
@uploader = nil
end
end
end
The the compatible render
method in place our specs pass, and very quickly at
that!
Uploading Fragments
Stores uploaded fragments
Finished in 0.01303 seconds
1 example, 0 failures
All of the integration issues exposed by the request spec were between Fragmenter classes and Rack, there weren't any incompatibilities when it was pulled into a full Rails app.
The tradeoff of testing without Rails is that it won't be resistant to changes
in render
, but that has been stable for a long time. The risk is well
worth the savings in setup, boot time, run time, and complexity.
Please note that in reality the spec was written before the UploadApp
implementation. It made more sense to explain the process slightly out of
order.
YYYY-MM-DDThh:mm:ssTZD (eg 1997-07-16T19:20:30+01:00)
If you read slightly farther in the spec you'll see that there is also an enhanced form that includes fractions of a second:
YYYY-MM-DDThh:mm:ss.sTZD (eg 1997-07-16T19:20:30.45+01:00)
Note the decimal near the end of the second format (ss.s
). The decimal allows
sub-second precision within date/time, which is potentially useful. Ruby's
standard Date
and Time
libraries support parsing the higher resolution
format via methods like Date.iso6801. Though they support parsing they don't
output fractions of a second when the #iso8601
method is called on an
instance of Time
:
Time.now.iso8601 #=> "2013-07-22T20:57:21-05:00"
Those of you with a test suite that verifies timestamps may have noticed a change when upgrading to Rails 4. Given a request spec similar to this:
it 'lists an existing post' do
post = Post.create(title: 'Strawberries')
get "http://example.com/api/posts/#{post.id}"
decoded = JSON.parse(last_response.body)
expect(decoded).to eq(
'id' => post.id,
'title' => 'Strawberries',
'created_at' => post.created_at.iso8601,
'updated_at' => post.updated_at.iso8601
)
end
The spec will fail when comparing both timestamps, even though they have been
formatted as iso8601
. The fake diff below attempts to illustrate this:
-"created_at"=>"2013-07-22T21:48:19Z",
+"created_at"=>"2013-07-22T21:48:19.355Z",
The problem stems from an inconsistent overriding of as_json
that only
applies to instances of TimeWithZone
but does not effect the as_json
monkey
patching of Time, Date, and DateTime.
The sub-second resolution change, as small as it is, was enough to break
timestamp parsing within some iOS apps. To prevent backward incompatibilities
I've laid my own monkey patch over TimeWithZone
:
# config/initializers/time_with_zone.rb
module ActiveSupport
class TimeWithZone
def as_json(options = nil)
iso8601
end
end
end
Going forward I'm looking to expose configuration that allows the
resolution to be configured. The default behavior in Rails 4 will remain 3
digits of resolution, but setting the resolution to 0
will remove it
entirely.
If you are serving up complex resources with customizable or user specific attributes you need something more flexible. One solution is composable per-resource caching, this is my experience implementing and enhancing performance.
Even with the current breed of native extension backed serializers the process
of serializing from native objects to a string of JSON can take a hefty
percentage of the server's response time. Caches such as in-memory, Memcached,
or Redis readily store a serialized JSON string or a marshalled object.
Always cache the serialized output of to_json
rather than the native
serialization produced by as_json
. We can see the performance difference with
an isolated benchmark:
require 'active_support/json'
require 'benchmark/ips'
require 'dalli'
client = Dalli::Client.new('localhost', namespace: 'json-bm', compress: true)
object = {
id: 1000,
published: false,
posts: [
{ id: 2000, body: 'Lorem ipsum dolor sit amet, consectetur adipiscing elit. Donec a diam lectus. Sed sit amet ipsum mauris. Maecenas congue ligula ac quam viverra nec consectetur ante hendrerit. Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean ut gravida lorem. Ut turpis felis, pulvinar a semper sed, adipiscing id dolor. Pellentesque auctor nisi id magna consequat sagittis. Curabitur dapibus enim sit amet elit pharetra tincidunt feugiat nisl imperdiet. Ut convallis libero in urna ultrices accumsan. Donec sed odio eros. Donec viverra mi quis quam pulvinar at malesuada arcu rhoncus. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. In rutrum accumsan ultricies. Mauris vitae nisi at sem facilisis semper ac in est.' }
]
}
client.set("object-to-json", object.to_json)
client.set("object-as-json", object.as_json)
GC.disable
Benchmark.ips do |x|
x.report('to_json') { client.get('object-to-json') }
x.report('as_json') { client.get('object-as-json').to_json }
end
Calculating -------------------------------------
to_json 1069 i/100ms
as_json 507 i/100ms
-------------------------------------------------
to_json 10581.7 (±12.0%) i/s - 50243 in 5.039299s
as_json 5089.4 (±0.9%) i/s - 25857 in 5.080955s
Storing and retrieving the string is, unsurprisingly, 2.1x faster than retrieving the marshalled object and stringifying it every time it is retrieved.
This works wonderfully when caching an individual resource, but caching a collection makes this approach tricky. When an entire collection is cached as a single string it locks any cached objects inside where they can't be displayed individually or shared with other collections. To work around this we need a collection caching mechanism that is bit more intelligent.
Typically cached content is retrieved one key at a time. That quickly adds up
to a lot of round trips when you are displaying a lot of cached resources.
Fortunately the caching strategies built into Rails support the ability to read
multiple items from the cache at once using the read_multi
method.
Rails.cache.read_multi 'first-key', 'second-key'
#=> { 'first-key' => '...', 'second-key' => '...' }
If read_multi
doesn't get a hit for a particular key it will eliminate it
from the results hash, leaving a hole in the results:
Rails.cache.read_multi 'first-key', 'unknown-key'
#=> { 'first-key' => '...' }
The missing key/value pairs leave an indication of what content is missing so
that we can patch in the content that we need. Prior to rails 4.1.X you would
need to do this manually, but now Rails provides the very clean fetch_multi
that handles both reading existing keys and writing whatever is missing.
object = { 'first-key' => 123, 'second-key' => 456 }
Rails.cache.fetch_multi('first-key', 'second-key') do |key|
object[key]
end
With a caching strategy like Dalli reading and writing can each be pipelined into a single request. This is hugely efficient and gives us a highly performant way to store and retrieve the elements of a collection we are caching.
Note: Unfortunately at this time the dalli
adapter does not support
fetch_multi
, but I have submitted a pull request which will hopefully
get it included in future versions. The benchmark presented here uses the branch
which implements fetch_multi
:
require 'dalli'
require 'active_support/json'
require 'active_support/cache'
require 'active_support/cache/dalli_store'
require 'benchmark/ips'
client = ActiveSupport::Cache::DalliStore.new('localhost', namespace: 'pipelining-bm')
objects = 30.times.inject({}) do |hash, i|
hash[i.to_s] = { id: i, value: 'abcdefg' }
hash
end
GC.disable
Benchmark.ips do |x|
x.report('fetch') do
objects.each do |key, object|
client.fetch(key) { object[:value] }
end
client.clear
end
x.report('fetch_multi') do
client.fetch_multi(*objects.keys) do |key|
objects[key][:value]
end
client.clear
end
end
Calculating -------------------------------------
fetch 19 i/100ms
fetch_multi 64 i/100ms
-------------------------------------------------
fetch 197.7 (±2.0%) i/s - 1007 in 5.094973s
fetch_multi 638.4 (±1.9%) i/s - 3200 in 5.014111s
Pipelining yields over a 3x performance increase, easily the difference between a 10ms and a 35ms cache retrieval.
The perforated gem implements the storage and pipelining strategies
outlined above. It provides a small wrapper around a collection that will
automatically pipeline reading and writing when the JSON serialization methods
(to_json
, as_json
) are called. There are no constraints on the caching
strategy or serialization libraries you work with, both of these aspects are
configurable (and a fallback fetch_multi
is provided if the current cache
strategy doesn't support it).
require 'perforated'
Perforated.configure do |config|
config.cache = Rails.cache
end
# Custom key construction strategy takes the current_user (scope)'s role in the
# system as an element of the final key.
class KeyStrategy
attr_reader :scope
def initialize(scope)
@scope = scope
end
def expand_cache_key(object, suffix)
args = [object, scope.role, suffix]
ActiveSupport::Cache.expand_cache_key(args)
end
end
Wrap the resource collection when returning the response and caching is used automatically. In this example posts are scoped to what a user has "liked". There would be potential overlap between posts that different users have liked, but they could be composed using what has previously been cached.
class PostsController < ApplicationController
def index
render json: Perforated::Cache.new(posts, strategy).to_json
end
private
def posts
current_user.liked_posts.limit(30)
end
def strategy
KeyStrategy.new(current_user)
end
end
Many apps use ActiveModelSerializers to serialize resources programmatically rather than declaratively. An essentialy part of the serialization strategy is how to handle relationships with other resources. The strategies themself are extremely well defined, but there is only partial support for properly caching and expiring the serialized resources. This is an area I'm actively exploring and I hope to hybridize the "perforated" approach and Rails' "russian doll" caching into a single robust strategy.
In the meantime I hope you'll look into leveraging perforated caching.
]]>