Migrating from Standalone Sidekiq to an ActiveJob Adapter

Wed, 17 Jan 2024 00:00:00 +0000

I paired on this problem with my coworker Hannah Yeates. The examples below are from this pairing session. Thanks for working on this together, Hannah!

I am working on a project right now that uses Sidekiq without the ActiveJob adapter. This means that our Sidekiq jobs look something like this:

class ExampleWorker
  include Sidekiq::Worker

  def perform(arg)
    puts "Hello, the arg was #{arg}"
  end
end

And we call them like this:

ExampleWorker.perform_async("a nice arg")

And in dev they’ll execute like this:

2024-01-17T15:05:31.455Z pid=45599 tid=18gj
  class=TestWorker 
  jid=8d32dcd88b1de2ed3a5419eb 
  INFO: start
Hello, the arg was foo

However, we want to switch over to GoodJob so that we have a transactionally safe background job processor. Our steps for doing this are:

Set up Sidekiq as an ActiveJob adapter instead of using Sidekiq on its own.
Install GoodJob and replace Sidekiq with GoodJob.
Run Sidekiq and GoodJob in parallel so that Sidekiq finishes all the work in Redis.
Shut down Sidekiq and remove it from the application code.

This post is about step 1, making Sidekiq the ActiveJob adapter instead of using it on its own.

Step 1 : Configure ActiveJob

Our first step is to configure the application to use Sidekiq for ActiveJob

# in application.rb
config.active_job.queue_adapter = :sidekiq

Then we created an empty ApplicationJob (more on this later).

# application_job.rb
class ApplicationJob < ActiveJob::Base
  # leave this empty for now
end

Step 2: Convert Workers to ApplicationJob

Next we updated our jobs to inherit from application job:

class ExampleWorker < ApplicationJob # Add this
  # remove this: include Sidekiq::Worker

  # The rest of the code remains the same
end

And finally, we changed all our method calls from ExampleWorker.perform_async("a nice arg") to ExampleWorker.perform_later("a nice arg").

Step 3: See What Breaks

So far, so good! Next, we had a question: What happens when a job is enqueued by Sidekiq but processed by ActiveJob? To emulate this we:

Booted a Rails console locally.
Use the original code to enqueue a job.
Modify the worker to use ApplicationJob.
Boot Sidekiq locally and see what errors we get.

In the real world, this would happen when the new code is deployed and there are jobs in the queue that were put there by Sidekiq. Here’s the first error we got:

WARN: NoMethodError: undefined method `jid=' for #<TestWorker:...>

We resolved this by creating an alias to ActiveJob’s provider_job_id= and tested again. In the end we had three NoMethodError messages to resolve. We updated our ApplicationJob to respond to those methods.

class ApplicationJob
  def jid=(arg)
    provider_job_id = arg
  end

  def jid
    provider_job_id
  end

  def bid=(arg);
    # We are lucky to not need batching, so we let this return `nil`
  end
end

Wrapping Up

Making the switch to ActiveJob was easier than we expected. We now have a backwards compatible way of moving from standalone Sidekiq to ActiveJob. Our next step is to swap out Sidekiq for GoodJob.

If you are planning on using GoodJob, or Solid Queue, check out this article on making the switch once you’re using ActiveJob:

Migrating from Sidekiq to Solid Queue | Kyle Keesling

Sidekiq on enumerator.dev

Migrating from Standalone Sidekiq to an ActiveJob Adapter

Step 1 : Configure ActiveJob

Step 2: Convert Workers to ApplicationJob

Step 3: See What Breaks

Wrapping Up