lyonrb 12feb2014-sidekiq
Post on 16-Apr-2017
867 Views
Preview:
TRANSCRIPT
Sidekiq Basics
Lyon.rb12 feb 2014Kurt Sussman@neophiliac
What is Sidekiq?
Async job manager
Released by Mike Perham in January 2012
Developed for theclymb.com
Drop-in replacement for Resque (almost)
Uses Celluloid gem for concurrency
Resque forks a complete copy of parentSidekiq spawns threads, uses actor model
Async challenges
Adds complexity (redis, sidekiq daemon)
Workers must be thread-safe and idempotent
Timing-related bugs are difficult to reproduce
Thread-safety
Uses all shared variables in an atomic way, unless each is allocated to a specific instance of the worker
Does not call non-thread-safe methods
Does not use APIs or database in a non-atomic way
The G{I,V}L does NOT make Ruby thread-safe
Shared mutable state == riskyRebuild context in the worker; assume nothingConstants aren't!Pay special attention to anything globalENV
Constants
Idempotency
An idempotent operation is one that has no additional effect if it is called more than once with the same input parameters
Sidekiq will retry jobs if there is an error
Retry interval and count are configurable
Why Sidekiq?
Sidekiq is faster, uses fewer resources, and is well-supported
TheClymb claims 100% uptime for sidekiq*
* but they restart sidekiq daemons every day
NameDatabaseThreads
delayed_jobSQL1
Resqueredis1
Sidekiqredismany
For real work, people report that one sidekiq daemon can do the work of 8-25 resque workers in far less memory
Submit a job
Mailer.welcome(@user.email) becomesMailWelcomeWorker.perform_async(@user.id)
Use ID isntead of data for 2 reasons:Less data to marshall/unmarshallGet to test whether it still exists (and needs to be sent) when the job is processed
Schedule a job
#perform_in(interval, *args)
Worker.perform_in(5.days, @user.id)
#perform_at(timestamp, *args)
Worker.perform_at(5.days.from_now, @user.id)
Testing
require 'sidekiq/testing'
Sidekiq::Testing.fake! (the default)
Worker.drain # force processing of queued jobs
Sidekiq::Testing.inline!
Workers are POROs
Testbed
github.com/neophiliac/sidekiq-example
Disclaimer: not a real app!
git diff master sidekiq
Uses foreman to start redis and sidekiq
Includes (some) rspec tests
Further reading...
https://github.com/mperham/sidekiq/wiki/_pages
http://www.mikeperham.com/2014/01/04/the-clymb-2013/
https://github.com/neophiliac/sidekiq-example
http://www.sitepoint.com/comparing-background-processing-libraries-sidekiq/
https://github.com/JustinLove/autoscaler
top related