when rails hits the fan

Proprietary and Confidential

When Rails hits the fan

Eric Saxby@sax @ecdysone @sax

Thursday, June 6, 13

This talk is about:


User traffic increasing weekover week



Databases outgrowing RAM



Avg site response time over same period


Who am I? Why should you care?


■ Application developerMany and various technologies. Worked with Rails for ~5 years.

■ Recent focus has been operationalChef, PostgreSQL, SmartOS, monitoring

■ TDD, BDD, Agile, DevOps, SOA, etcI care about how code is organized, and always want to learn how to do that better.

■ Now I’m at Wanelo


What is Wanelo?


■ Wanelo (“Wah-nee-lo” from Want, Need Love) is a global platform for shopping.



Marketing-free shopping across 100s of thousands of unique stores



Personal feed of products from any store on the internetNo Il8n or l10n... yet!


Technology overview


■ MRI Ruby 1.9.3 & Rails 3.2

■ PostgreSQL 9.2.4, Solr 3.6

■ Joyent Cloud, SmartOSZFS, ARC, raw IO performance, SmartOS, dTrace

■ Circonus, NewRelic, BoundaryMonitoring, graphing, alerting

■ Chef + Opscode

■ Amazon S3 + Fastly CDN

■ statsd, Graphite, nagios


How do you know whenyou are entering the

bad place?

Proprietary and ConfidentialThursday, June 6, 13

Let's go back to this for a sec


What is the bad place?


green: disk reads, red: disk writes on DB server

Is this actually a problem?


What is the bad place?


Can be difficult to predict, using most of thedefault metrics we track

green: disk reads, red: disk writes on DB server


Utilization


> iostat -xM 3 extended device statisticsdevice r/s w/s Mr/s Mw/s wait actv svc_t %w %bsd0 11.3 48.8 0.2 0.5 0.0 0.5 8.2 0 11sd1 13.9 42.2 0.2 0.6 0.0 0.3 5.8 0 11sd2 12.0 24.9 0.2 0.5 0.0 0.2 5.5 0 10sd3 12.3 27.2 0.2 0.5 0.0 0.4 9.6 0 10sd4 7.3 15.3 0.1 0.3 0.0 0.1 6.2 0 6sd5 13.0 14.6 0.2 0.3 0.0 0.1 4.8 0 7sd6 9.6 50.5 0.2 0.5 0.0 0.4 5.9 0 10sd7 7.0 46.5 0.1 0.6 0.0 0.4 7.5 0 11sd8 9.3 33.5 0.1 0.4 0.0 0.3 6.3 0 9sd9 7.6 32.5 0.1 0.4 0.0 0.2 6.1 0 7sd10 6.3 52.1 0.1 0.6 0.0 0.3 6.0 0 10sd11 7.6 50.2 0.1 0.6 0.0 0.5 8.5 0 12


Utilization



Develop spidey senses over time, but problemsare highly anecdotal


Saturation




Both are important!


■ Tracking saturation helps you predict problems

■ Tracking utilization can tell you how to solve that problem

■ Basically, watch every video by Brendan Gregg on YouTubehttp://www.youtube.com/results?search_query=brendan+gregg


http://www.youtube.com/results?search_query=brendan+gregg

http://www.youtube.com/results?search_query=brendan+gregg

Know the limits of your data


Averages can be extremely useful, butdo not give a complete picture


Know the limits of your data


Outliers can cause severe problemseven when average is great


This is why we ❤ PostgreSQL


■ pg_stat_activity

■ pg_stat_user_indexes

■ pg_stat_user_tables

■ pg_stat_statements

■ PostgreSQL gives you tools to monitor it and operate at scale



pg_stat_statements


Ok, so back to thebad place



Hit ratio of File System Cache ondatabase server degrading over time



Average response latencyincreasing over time, ties up CPU resources


Step 1: Cache all the things!


■ Reduce load on DB(s)

■ Reduce rendering time


Cache invalidation?


■ Use model updated_at in the cache key

■ Changing DB record invalidates all caches for that record


What were the goals of caching?


■ Using model attributes as cache keys means you need to fetch DB records

■ At high scale, fetching DB records for every page becomes problematic

■ Cache sweepers are painful, but they're the only thing we've found to be scalable and reliable


Action cache all the things!


■ Cache hits skip all rendering

■ Still able to run before filters, for instance when doing A/B testing

■ Some cached pages can be put behind a CDN

■ Requires page personalization to be added via Ajax


Fragment cache the rest!


■ Fragments can be shared between pages

■ Can reduce rendering time

■ Can remove queries for related records



■ Difficult to remove top level query

■ Joins/eager loading means some related records are still queried

■ How many trips to memcached?



def multi_get_on_collection(objects, cache_options = {}) cache_keys = objects. inject(ActiveSupport::OrderedHash.new) do |key_map, obj| key_map[obj] = cache_options[:cache_key_proc].call obj key_map end

pre_rendered_objects = Rails.cache.read_multi *cache_keys.values

cache_keys.map do |object, cache_key| cached_html = pre_rendered_objects[cache_key]

if cached_html.present? && caching_enabled? cached_html else cache_options[:render_proc].call(object).tap do |fragment| Rails.cache.write cache_key, fragment, cache_options end end endend


MultiGet is your friend


■ Turns 100+ memcached calls into a single request

cache_key = ->(product) do "products_thumb_#{product.id}" endrenderer = ->(product) do render "products/thumb", model: product end

multi_get_on_collection(@products, cache_key_proc: cache_key, render_proc: renderer, expires_in: 6.hours).join("\n").html_safe



Increasing your cache hit ratio meansless queries against your database



device r/s w/s Mr/s Mw/s wait actv svc_t %w %b

sd1 384.0 1157.5 48.0 116.8 0.0 8.8 5.7 2 100 sd1 368.0 1117.9 45.7 106.3 0.0 8.0 5.4 2 100 sd1 330.3 1357.5 41.3 139.1 0.0 9.5 5.6 2 100

DB latency increases

■ Even with highly efficient caches, iostat shows 100% disk busy


count(*)


TOO MANY COUNTS


Rails has the answer!


■ counter_cache column on table

■ Adding records executes INCR

■ Removing records executes DECR

class Product < ActiveRecord::Base belongs_to :store, counter_cache: trueend


But...


■ On very write heavy applications, multiple requests will update the same record

■ DEADLOCK errors


Use background jobs


■ Stop updating counter caches on save

■ Queue a delayed job for the near future

■ Job performs a complete recalculation of the counter cache and is idempotent

■ The higher the count, the further we delay the job (less likely users will notice)


Deduplicate delayed jobs


■ Sidekiq with UniqueJob plugin

■ Updates are serialized via a fixed number of workers

■ Workers can be stopped to alleviate DB load in an emergency

■ Same pattern can be applied elsewhere, like updating Solr indexes


Deduplicate delayed jobs


class UpdateProductCountWorker < WaneloWorker sidekiq_options queue: :product_counts, unique: true wait 10.minutes

def perform!(params) id = params[:id].to_i ActiveRecord::Base.connection.execute %Q{ update stores set product_count = (select count(*) from products where store_id = stores.id) where stores.id = #{id} } endend


STILL TOO MANY COUNTS



Pagination gems run counts



■ Kaminari executes a count(*) to determine total page count




■ Kaminari executes a count(*) to determine total page count SELECT "stores".* FROM "stores" WHERE (state = 'approved') LIMIT 20 OFFSET 0

SELECT COUNT(*) FROM "stores" WHERE (state = 'approved')




■ Kaminari executes a count(*) to determine total page count SELECT "stores".* FROM "stores" WHERE (state = 'approved') LIMIT 20 OFFSET 0

SELECT COUNT(*) FROM "stores" WHERE (state = 'approved')


■ We paginate EVERYTHING

■ We often already know total count from counter cache (or can hard code 10,000)



module Kaminari module ActiveRecordRelationMethods # a workaround for AR 3.0.x that returns 0 for #count when page > 1 # if +limit_value+ is specified, load all the records and count them if ActiveRecord::VERSION::STRING < '3.1' def count(column_name = nil, options = {}) #:nodoc: limit_value ? length : super(column_name, options) end end

def total_count(column_name = nil, options = {}) #:nodoc: @total_count ||= begin c = except(:offset, :limit, :order)

# Remove includes only if they are irrelevant c = c.except(:includes) unless references_eager_loaded_tables?

# .group returns an OrderdHash that responds to #count c = c.count(column_name, options) if c.is_a?(ActiveSupport::OrderedHash) c.count else c.respond_to?(:count) ? c.count(column_name, options) : c end end end endend



Time for some monkey patches!

module ActiveRecord class Relation def custom_counter(count) @total_count ||= count self end endend

module Sunspot module Search class AbstractSearch def custom_counter(count) @total ||= count self end end endend

@products = @store.products. custom_counter(@store.products_count). page(params[:page])


How do we know whatRails is really doing?


ruby-prof and pilfer


■ Profile the entire stack trace of an action or a set of classes

https://github.com/eric/pilfer

https://github.com/ruby-prof/ruby-prof


https://github.com/kigster/ventable





■ Can work as Rack middleware

■ Outputs HTML that will tell you...

if Rails.env.profile? use Rack::RubyProf, :path => '/tmp/profile'end



■ Can work as Rack middleware

■ Outputs HTML that will tell you...

if Rails.env.profile? use Rack::RubyProf, :path => '/tmp/profile'end

■ How long are we spending generating URLs???



module UrlHelpers

def product_path(product) "/p/#{product.id}/#{product.slug}" end

end


The importance offast JSON rendering


HTML vs JSON


■ Since launching native iOS and Android apps, the majority of Wanelo traffic is served via a JSON API

■ The longer we spend rendering JSON, the more CPUs are tied up, the more servers we need



Removing RABL


Things we did not expect


■ RABL serializes, then deserializes JSON to do merges!

■ ActiveSupport defines inefficient :to_json on every object



■ Rendering JSON partials should hash.merge!

■ Fragment caching should marshal hashes, not JSON

■ Caching should allow for MultiGet

■ Should allow for arbitrary composition

■ JSON conversion should use OJ to call :to_json ONCE

https://github.com/wanelo/compositor





require 'active_support/json'require 'active_support/core_ext/object/to_json'

[Object, Array, FalseClass, Float, Hash, Integer, NilClass, String, TrueClass].each do |klass| klass.class_eval do def to_json(options = nil) Oj.dump(self, options) end endend


AsynchronousCommits


Do you read, or do you write?


■ Tools like iostat, vmstat, kstat, collectd pg_stat_user_tables can show you utilization

■ Reads much easier to scale than writes. Read/write splitting, caching.

■ What do you do when writes become the bottleneck?


Writes committed to disk?


■ Some workloads are more lenient for delay/possible data loss

■ Applicable to many technologies

■ Even microsecond delays can reduce load

■ Waiting for Solr to respond can keep DB transactions open longer


i.e. PostgreSQL


i.e. Solr


 <autoCommit>  <maxDocs>1000</maxDocs>

 <maxTime>30000</maxTime></autoCommit>

What happens when datagrows larger than a

single database?



Growth of one key DB table over 3 months


Service OrientedArchitecture


The only real way to scale


■ Tune code, infrastructure for a very particular workload

■ Hide sharding from other codebases

■ Allow small teams to manage small(er) codebases


The only real way to scale


■ Tune code, infrastructure for a very particular workload

■ Hide sharding from other codebases

■ Allow small teams to manage small(er) codebases

■ Everyone talks about why, no-one talks about how in the Rails world


Services are hard (the first time)


■ Synchonous vs asynchronous data persistence

■ Message passing

■ Testing

■ Iterative development


Iteration is the key


■ Isolate data

■ Isolate interface

■ Extract interface with an database adapter

■ Launch service layer

■ Switch interface to user service adapter



class Product < ActiveRecord::Base has_many :savesend



class Product < ActiveRecord::Base has_many :savesend X



class Product < ActiveRecord::Base has_many :savesend Xclass Product < ActiveRecord::Base def saves Save.where(product_id: self.id) endend



class Product < ActiveRecord::Base has_many :savesend Xclass Product < ActiveRecord::Base def saves Save.where(product_id: self.id) endend

Do this everywhere (you have tests, right?)



class Save < ActiveRecord::Base establish_connection "saves_#{Rails.env}"end

■ Set up a read replica

■ Take down site

■ Promote replica to be a master

■ Restart unicorns with new config

■ Bring up site

■ Clean up unnecessary tables on each DB



class Save < ActiveRecord::Base establish_connection "saves_#{Rails.env}"

def self.by_product(product) where(product_id: product.id) endend

class Product < ActiveRecord::Base def saves Save.by_product(self) endend

■ Reduce Ruby interface to minimum possible

■ Easy to deploy this



class Save include SavesClientend

module SavesClient def self.included(other) other.send(:attr_accessor, :id, :product_id) other.extend ClientClassMethods end

module ClientClassMethods def by_product(*args) adapter.new(self).by_product(*args) end

def adapter @adapter ||= SavesClient::DbAdapter end endend



module SavesClient class DbAdapter def self.close_connections # Check in the database connection, # since we're shutting down this thread SavesService::Save.clear_active_connections! end

def by_product(*args) relation :by_product, *args end

def relation(method, *args) SavesClient::AdapterRelation.new(self, SavesService::Save.send(method, *args)) end

def all(scope) # Scope is an AR Relation instance returned from # SavesService::Save.by_product(product) scope.all.map { |m| client_class.new save_attrs_from(m) } end endend



module SavesClient class DbAdapter def self.close_connections # Check in the database connection, # since we're shutting down this thread SavesService::Save.clear_active_connections! end

def by_product(*args) relation :by_product, *args end

def relation(method, *args) SavesClient::AdapterRelation.new(self, SavesService::Save.send(method, *args)) end

def all(scope) # Scope is an AR Relation instance returned from # SavesService::Save.by_product(product) scope.all.map { |m| client_class.new save_attrs_from(m) } end endend

Thread safety is EXTREMELY importantThursday, June 6, 13


module SavesClient class AdapterRelation attr_reader :adapter, :scope

def initialize(adapter, scope) @adapter, @scope = adapter, scope end

def limit(num); end def order(order); end def page(num); end def first; end

def all adapter.all(scope) end endend

■ Calling :by_product instantiates a Relation

■ Calling :all executes the query


What executes the query?


■ The ActiveRecord model moves into the gem

■ The adapter translates the Save methods into AR calls, maps columns into attributes on our class

■ Important to deploy this at this stage, as there many problems to solve, like:

■ Getting your tests green, fixtures consistent

■ Figuring out whether you really covered all access patterns

module SavesService class Save < ActiveRecord::Base endend


Minimize the Ruby access


■ We were able to reduce everything to 7 scopes, i.e. :by_product, :by_user, etc

■ Reduced Relation methods to these:

■ limit

■ page

■ order

■ count

■ all

■ first

■ last

■ pluck

■ find_in_batches


Launch the service layer


■ Now that we have a small public Ruby interface, we can pair that to a Sinatra app

■ Sinatra can serve as a long-term fake for Selenium, even after service is re-written


module SavesService class Web < Sinatra::Base set :environment, ENV['RACK_ENV'] || "development" PAGE_SIZE = 50

ActiveRecord::Base.include_root_in_json = false register Sinatra::ActiveRecordExtension

configure do set :database_file, SavesService.config.db_config set :root, File.expand_path("../../../", __FILE__) disable :raise_errors disable :show_exceptions set :logger, nil end

error do e = env['sinatra.error'] ActiveRecord::Base.logger.error ["#{e.class}: #{e.message}", *e.backtrace].join("\n ") status 500 body '{"errors":":("}' end

before { content_type 'application/json' } endend



get '/products/:pid/saves' do paginate SavesService::Save.by_product(params[:pid])end

private

def paginate(scope) return count(scope) if params[:count].present?

order = params[:order] if params[:order] =~ /\Adesc|asc\Z/i scope = scope.order("created_at #{order || 'desc'}")

limit = params[:limit] ? params[:limit].to_i : PAGE_SIZE page_number = [params[:page].to_i - 1, 0].max * limit scope = scope.offset(page_number).limit(limit)

scope = scope.pluck(params[:pluck]) if params[:pluck]

body Oj.dump(saves: scope)end



#!/usr/bin/env rubyrequire 'optparse'

options = { :environment => 'development', :port => 3000}

OptionParser.new do |opts| opts.banner = "Usage: saves_service [options]"

opts.on("-d", "--dbconfig OPT", "path to database.yml") do |opt| options[:db_config] = File.expand_path(opt, Dir.pwd) end

opts.on("-E", "--environment OPT", "RACK_ENV to use") do |opt| options[:environment] = opt end

opts.on("-p", "--port OPT", "port to use") do |opt| options[:port] = opt endend.parse!

cmd_env = { 'RACK_ENV' => options[:environment], 'DB_CONFIG' => options[:db_config],}.delete_if{|k,v| v.nil? }

rackup_file = File.expand_path('../../config.ru', __FILE__)

exec cmd_env, "unicorn -p #{options[:port]} #{rackup_file}"


DbAdapter vs HTTPAdapter


■ Maps Ruby interface to Net::HTTP::Persistent

■ Deserializes JSON into values on class

■ Scope becomes a mapby_product(1) => "/products/1/saves"

■ Finder methods map paramslimit(10) => "?limit=10"

■ AdapterRelation uses Adapter to fetch records, does not change at all



module SavesClient class AdapterRelation attr_reader :adapter, :scope

def initialize(adapter, scope) @adapter, @scope = adapter, scope end

def limit(num); end def order(order); end def page(num); end def first; end

def all adapter.all(scope) end endend


Deploy as a setting change


Rails.application.config.after_initialize do |app|

if Settings.saves_service.enabled Save.saves_base_url = Settings.saves_service.url else require 'saves_client/db_adapter' Save.adapter = SavesClient::DbAdapter

require 'saves_service/save' saves_db = "saves_#{Rails.env}" SavesService::Save.establish_connection saves_db end

end


Takeaways



■ Choose technologies that are easy to operate and monitor

■ Don't immediately break when they hit resource thresholds

■ Sound replication strategies

■ Assume that data should be tracked, even if you don't yet understand the relevance

■ Small iterative performance improvements can have massive payoff over time


Thanks!


@sax

@ecdysone

@sax

https://github.com/wanelo

https://github.com/wanelo-chef


when rails hits the fan

Technology

proprietary andconfidentialcan

disk reads

disk writes

actv svc

thebad place

db serveris

db serverthursday

etci care