riak at ideeli
TRANSCRIPT
What is an ideeli?
Thursday, February 14, 13
Flash Saleshttp://www.flickr.com/photos/moonlightbulb/4064060773/
Thursday, February 14, 13
Flash SalesThursday, February 14, 13
Flash SalesThursday, February 14, 13
Flash SalesThursday, February 14, 13
Flash SalesThursday, February 14, 13
Obligatory Stats
• 2,000,000 visitors/week
• 80,000 front end requests/minute
• 120 production servers
• 300 deploys a year
• 40ms front end response time
Thursday, February 14, 13
Obligatory Riak Stats
• 1.2MM keys
• 4 virtualized nodes
• 4ms response
• 40,000 GETs/min
• 2,000 PUTs/min
Thursday, February 14, 13
The SpikeThursday, February 14, 13
The StackThursday, February 14, 13
New Toys Are Fun(9AM - 5PM)
Thursday, February 14, 13
What Counts1AM - 7AM
Thursday, February 14, 13
K.I.S.S.Keep It Simple Stupid
Thursday, February 14, 13
K.I.S.S.Keep It Simple Stupid
Kan It Sell Shoes?
Thursday, February 14, 13
3 Uses for Riak
• Cache Warming
Thursday, February 14, 13
3 Uses for Riak
• Cache Warming
• Feature Testing (LABS)
Thursday, February 14, 13
3 Uses for Riak
• Cache Warming
• Feature Testing (LABS)
• User Favorites
Thursday, February 14, 13
Cache Warming
Thursday, February 14, 13
The SpikeThursday, February 14, 13
Cache WarmingPseudo-code:
every.30.minutes do all_urls.each do |url| curl "#{url}?force_cache=true" endend
Thursday, February 14, 13
Cache WarmingThursday, February 14, 13
Response timesThursday, February 14, 13
History
Thursday, February 14, 13
memcached
Thursday, February 14, 13
memcached~40ms
Thursday, February 14, 13
memcached5/6 requests ~ 40ms
1/6 requests ~ 3000ms=
~500ms average
Thursday, February 14, 13
Domino Effecthttp://www.flickr.com/photos/soham_pablo/309379628/
Thursday, February 14, 13
Manual InterventionThursday, February 14, 13
membase(couchbase)
Thursday, February 14, 13
RiakAll requests ~40ms
Thursday, February 14, 13
RiakAll requests ~40ms
Thursday, February 14, 13
LABS
• Feature Testing Framework
• Key based on user id
• Value YAML string w/ list of experiments
Thursday, February 14, 13
FavoritesThursday, February 14, 13
FavoritesThursday, February 14, 13
Favorites
• Partially normalized
• user_#{id} → array of productsproduct_#{id} → metadata about product
Thursday, February 14, 13
Rails Integration
• Modified riak-client gem for Rails 2.3
• Load balancing built in
• Multiple primary and secondary servers
• Custom ActiveSupport::Cache store
Thursday, February 14, 13
Rails Integrationmodule ActiveSupport module Cache class RailsStore < Store
def read end
def write end
def delete end endend
config.after_initialize do ActionController::Base.cache_store = Rails.rails_store.fragmentsend
Thursday, February 14, 13
MonitoringThursday, February 14, 13
GraphiteThursday, February 14, 13
New RelicThursday, February 14, 13
Alerting
• Compute random string
• PUT value into Riak
• GET value out of Riak
• alert if value_in != value_out
• alert if operation took too long
Thursday, February 14, 13
Tuning
Thursday, February 14, 13
Bitcask Merges
Thursday, February 14, 13
Disk UsageThursday, February 14, 13
Heavy I/OThursday, February 14, 13
expiry_grace_timeThursday, February 14, 13
Offset Merge WindowsThursday, February 14, 13
Puppetclass ideeli_riak ( $merge_window_start, $merge_window_end ) {... file { '/etc/riak/app.config' : content => template("ideeli_riak/app.config.erb"), }...}
class riak-node { $merge_window_start = inline_template( ‘<%= (scope.function_fqdn_rand([6]).to_i + 20) % 23 %>' )
class { 'ideeli_riak' : merge_window_start => $merge_window_start, merge_window_end => 6 }}
Thursday, February 14, 13
Riak Loves Disk
Thursday, February 14, 13
New Disk ArrayThursday, February 14, 13
New Disk ArrayThursday, February 14, 13
Improvement!Thursday, February 14, 13
We’re Hiringhttp://ideelicareers.com/
Thursday, February 14, 13
Questions?
Thursday, February 14, 13