resque - files.meetup.com awesome.pdf · tangent alert: redis! scalability the gem load balances...
TRANSCRIPT
![Page 1: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/1.jpg)
ResqueHigh-performance asynchronous task queuing and
processing
![Page 2: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/2.jpg)
ResqueHigh-performance asynchronous task queuing and
processing
![Page 3: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/3.jpg)
Experts‘Cause you might build the next GitHub
![Page 4: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/4.jpg)
Three Things
![Page 5: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/5.jpg)
1. BackgroundWhy run background processes at all?
![Page 6: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/6.jpg)
2. Delayed::JobBecause it’s the foundation for Resque
![Page 7: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/7.jpg)
3. ResqueProcess different.
![Page 8: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/8.jpg)
1. Background
Why?
![Page 9: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/9.jpg)
1. Background
It’s all about the ms• Resize an image
• Send an e-mail
• Communicate with a web service
• Geocode a location
![Page 10: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/10.jpg)
1. Background
Do something later than now
![Page 11: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/11.jpg)
1. Background
Webpages.Rails serves webpages.
Memories. You're talkin about memories.
![Page 12: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/12.jpg)
1. Background
1. Receive requests2. Send responses
![Page 13: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/13.jpg)
1. Background
Everything else can go elsewhere
![Page 14: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/14.jpg)
1. Background
Where?• CRON
• Beanstalkd
• Workling
• BackgroundRB
![Page 15: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/15.jpg)
1. Background
• Silent failure
• No status checks
• Unstable processes
WTF my tasks r fail?!1
![Page 16: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/16.jpg)
2. Delayed::Job
To the rescue!
![Page 17: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/17.jpg)
2. Delayed::Job
ActiveRecord::Base’d
• Persistence! At last!
• Ease of use! At last!
![Page 18: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/18.jpg)
2. Delayed::Job
How easy?
![Page 19: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/19.jpg)
2. Delayed::Job
# without delayed_jobNotifier.deliver_signup(@user)
# with delayed_jobNotifier.send_later :deliver_signup, @user
Super easy.
![Page 20: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/20.jpg)
2. Delayed::Job
Where do they go?
![Page 21: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/21.jpg)
2. Delayed::Job
mysql> SELECT * FROM delayed_jobs;*************************** 1. row *************************** id: 1 priority: 0 attempts: 0 handler: --- !ruby/object:WorkerBee <-- Notice the pretty Marshal dumpn: 18
last_error: NULL run_at: 2010-03-10 22:44:30 locked_at: NULL failed_at: NULL locked_by: NULLcreated_at: 2010-03-10 22:44:30updated_at: 2010-03-10 22:44:301 row in set (0.00 sec)
![Page 22: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/22.jpg)
2. Delayed::Job
What do you do with them?
$ script/delayed_job start
![Page 23: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/23.jpg)
2. Delayed::Job
Workers.
• Live in their own thread
• Query the database for jobs
• Perform the long-running taskaway from request/response
![Page 24: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/24.jpg)
The Old Way
![Page 25: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/25.jpg)
The New Way
Delayed::Job
![Page 26: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/26.jpg)
2. Delayed::Job
Totally rad.
![Page 27: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/27.jpg)
3. Resque
Isn’t Delayed::Job enough?
![Page 28: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/28.jpg)
3. Resque
Apparently not.
![Page 29: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/29.jpg)
3. Resque
Resque is some Enterprise Grade shit.
![Page 30: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/30.jpg)
3. Resque
Runs on Redis
![Page 31: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/31.jpg)
Just like Memcached
Redis is a Key-Value Store
Tangent Alert: Redis!
![Page 32: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/32.jpg)
Tangent Alert: Redis!
PersistenceUnlike Memcached
![Page 33: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/33.jpg)
Tangent Alert: Redis!
ScalabilityThe Gem load balances for you
Resque.redis = DistRedis.new( :hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})
![Page 34: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/34.jpg)
Tangent Alert: Redis!
Basically:Fast & Distributed
![Page 35: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/35.jpg)
Also, Rails-independentNo ActiveRecord/ActiveSupport means you can run Resque (on Redis) in Sinatra or any other Ruby app
Tangent Alert: Redis!
![Page 36: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/36.jpg)
3. Resque
• Forked, sandboxed threads
• Failing tasks are tracked
• Kills timeouts
Better Workers
![Page 37: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/37.jpg)
3. Resque
Queueing
fileserver$ QUEUE=zip_files rake resque:work webserver$ QUEUE=cache_images rake resque:work
Separate queues for separate workers
![Page 38: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/38.jpg)
3. Resque
No MarshalingBecause it should be functional
Resque.enqueue(SignupNotifier, @user.id)
![Page 39: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/39.jpg)
3. Resque
No Marshaling
class SignupNotifier def self.perform(user_id) Notifier.deliver_signup(User.find(user_id)) endend
![Page 40: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/40.jpg)
3. Resque
UIThat’s right. It has its own interface.
![Page 41: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/41.jpg)
3. Resque
• Many background tasks
• Distributed task servers
• Stable workers
So... why?
![Page 42: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/42.jpg)
3. Resque
• Complex and expensive
• No scheduling or priority like in D::J
• Asynchronous Redis writes don’t guarantee persistence
Trouble Points
![Page 43: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/43.jpg)
3. Resque
Live coding FTW!Here goes nothing...
![Page 44: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/44.jpg)
ThnaksThanks for taking the time to listen to me blather
about Resque!
![Page 46: Resque - files.meetup.com Awesome.pdf · Tangent Alert: Redis! Scalability The Gem load balances for you Resque.redis = DistRedis.new(:hosts=> %w{192.168.1.22:6379 192.168.1.23:6379})](https://reader036.vdocuments.site/reader036/viewer/2022071013/5fcbefc21c6189578942e3c8/html5/thumbnails/46.jpg)
Bmore on Rails
http://www.meetup.com/bmore-on-rails/
Meetups - 2nd Tuesday#OSHN - 4th Tuesday