Sidekiq Basics

Lyon.rb12 feb 2014Kurt Sussman@neophiliac

What is Sidekiq?

Async job manager

Released by Mike Perham in January 2012

Developed for theclymb.com

Drop-in replacement for Resque (almost)

Uses Celluloid gem for concurrency

Resque forks a complete copy of parentSidekiq spawns threads, uses actor model

Async challenges

Adds complexity (redis, sidekiq daemon)

Workers must be thread-safe and idempotent

Timing-related bugs are difficult to reproduce

Thread-safety

Uses all shared variables in an atomic way, unless each is allocated to a specific instance of the worker

Does not call non-thread-safe methods

Does not use APIs or database in a non-atomic way

The G{I,V}L does NOT make Ruby thread-safe

Shared mutable state == riskyRebuild context in the worker; assume nothingConstants aren't!Pay special attention to anything globalENV

Constants

Idempotency

An idempotent operation is one that has no additional effect if it is called more than once with the same input parameters

Sidekiq will retry jobs if there is an error

Retry interval and count are configurable

Why Sidekiq?

Sidekiq is faster, uses fewer resources, and is well-supported

TheClymb claims 100% uptime for sidekiq*

* but they restart sidekiq daemons every day

NameDatabaseThreads

delayed_jobSQL1

Resqueredis1

Sidekiqredismany

For real work, people report that one sidekiq daemon can do the work of 8-25 resque workers in far less memory

Submit a job

Mailer.welcome(@user.email) becomesMailWelcomeWorker.perform_async(@user.id)

Use ID isntead of data for 2 reasons:Less data to marshall/unmarshallGet to test whether it still exists (and needs to be sent) when the job is processed

Schedule a job

#perform_in(interval, *args)

Worker.perform_in(5.days, @user.id)

#perform_at(timestamp, *args)

Worker.perform_at(5.days.from_now, @user.id)

Testing

require 'sidekiq/testing'

Sidekiq::Testing.fake! (the default)

Worker.drain # force processing of queued jobs

Sidekiq::Testing.inline!

Workers are POROs

Testbed

github.com/neophiliac/sidekiq-example

Disclaimer: not a real app!

git diff master sidekiq

Uses foreman to start redis and sidekiq

Includes (some) rspec tests

Further reading...

https://github.com/mperham/sidekiq/wiki/_pages

http://www.mikeperham.com/2014/01/04/the-clymb-2013/

https://github.com/neophiliac/sidekiq-example

http://www.sitepoint.com/comparing-background-processing-libraries-sidekiq/

https://github.com/JustinLove/autoscaler