deploying and monitoring rails
Post on 06-May-2015
11.657 Views
Preview:
DESCRIPTION
TRANSCRIPT
Deploying and Monitoring
Ruby on Rails A practical guide
Mathias Meyer and Jonathan Weiss, 25.05.2009
Peritor GmbH
2
Who are we?
Jonathan Weiss
• Consultant for Peritor GmbH in Berlin
• Specialized in Rails, Scaling, Deployment, and Code Review
• Webistrano - Rails deployment tool
• FreeBSD Rubygems and Ruby on Rails maintainer
http://www.peritor.com
http://blog.innerewut.de
3
Who are we?
Mathias Meyer
• Consultant for Peritor GmbH in Berlin
• Specialized in Rails, Performance/Database Tuning, Deployment, and Refactoring
• Maintainer of Macistrano – Webistrano-Client for Mac OS X (and soon the iPhone), run_later and acts_as_solr
http://www.paperplanes.de
4
Agenda
Infrastructure Deployment Monitoring Q & A Practical Session
5
Infrastructure Infrastructure
6
Simple Rails Setup
One Rails instance handles all requests
Rails is single-threaded: There is only one concurrent request
7
Rails Setup
8
Rails Setup
9
Typical Rails Setup
• A load-balancer distributes the incoming requests
• Some load-balancers will deliver static requests themselves
• Several Rails instances handle all requests
• Number of concurrent requests equals number of Rails instances
10
Rails Setup Options
11
Deployment Questions
Mongrel?
Apache?
Ebb?
FastCGI?
mod_rails?
Thin?
Nginx?
Lighttpd?
Pound? mod_proxy_balancer?
HA-Proxy?
Phusion Passenger?
Load-balancer? Proxy?
Rails Application Server?
Swiftiply? Reverse Proxy?
Pen?
12
What we are going to cover today
Rails Application Server
• FastCGI
• Mongrel
• mod_rails / Phussion Passenger
• JRuby + Glassfish & Co.
Proxy/Web Server
• Apache2
• Nginx
• Lighttpd
• HA-Proxy
13
FastCGI
14
FastCGI
• Protocol to communicate with long-running CGI applications
• Usage of either mod_fcgi with Apache 1.3 or mod_fcgi with Lighttpd
• Proxy local and remote FastCGI instances
• Oldest way of deploying Rails
• Deprecated and unstable
• Hard to debug (FastCGI protocol)
15
FastCGI
16
FastCGI
17
18
Mongrel
19
Mongrel
• Developed by Zed Shaw as an alternative to FastCGI
• Complete HTTP-Server that can load arbitrary Ruby-servlets
• Built-in Rails support
20
Mongrel
21
Mongrel
Apache 2.2 – mod_proxy_balancer Lighttpd Nginx HA-Proxy Pound
22
Mongrel Cluster
Utility to manage several Mongrel instances
23
Mongrel Cluster
Control Mongrels
24
Mongrel and Apache 2.2
Define Mongrel Cluster in Apache
25
Simple Mongrel and Apache 2.2
Redirect all traffic to the Mongel cluster
26
A more complex example
• Redirect dynamic requests
• Serve static content
• Support cached pages
• Support maintenance page
• Enable client-side caching of images, stylesheets, and JavaScript
• Compress output if supported
27
Mongrel
• Very robust
• Strict HTTP parser
• Easy to debug (HTTP!)
• Used to be defacto deployment setup with Apache 2.2 and mod_proxy_balancer
• Can be a bit difficult to setup (mongrel_cluster, ports, Apache)
• Not so easy on mass/virtual hosting
28
mod_rails
29
mod_rails a.k.a Phusion Passenger
• Module for Apache 2.2 (and Nginx)
• Allows Apache to control Rails instances
• Apache starts and stops application instances depending on the application load
• Able to run any Rack-compatible Ruby application (Merb, Sinatra & Co.)
• Only manages Rails on one host - no remote instances
• Combine with HTTP-Proxy / balancing solution
30
Install Phusion Passenger
Install Apache module
Load and activate in Apache
31
Install Phusion Passenger
Install Passenger with nginx (installs custom nginx)
Configure nginx
32
Customized Phusion Passenger
Control Rails instance number
33
Control Phusion Passenger
Restart after deployment:
34
Phusion Passenger
One machine
35
Phusion Passenger
One machine Multiple machines
36
Phusion Passenger
• Ready for production
• Makes setup easier – on the single machine level
• Multiple servers still require load balancer
• Suitable for mass-hosting
• upcoming standard way of deploying Rails
37
JRuby
38
• Ruby Runtime on the Java Virtual Machine
• Implemented in Java and Ruby
• Compiles Ruby into Java-bytecode
• Integrates with Java code and libraries
• Java’s promises of native threads and JIT
• Allows for Ruby/Rails applications to be packaged as WAR files
• WAR files deployable on any J2EE-container: Glassfish, JBoss, Tomcat, Jetty, …
39
JRuby on Rails
40
JRuby on Glassfish
One machine
41
JRuby on Glassfish
One machine Multiple machines
42
Setup JRuby on Glassfish
1. Download JRuby and Glassfish
2. From http://blog.headius.com/2008/08/zero-to-production-in-15-minutes.html
43
Warble Configuration
Define min/max Rails runtimes
44
Glassfish for development
Shrunk-down version of Glassfish v3 - packaged as a gem
Can run any Rack-based application
Installation:
• $ jruby -S gem install glassfish
• $ glassfish
• There is no Step 3
45
Rails Setup
46
Proxy Options
47
Proxy Requirements
• Hide cluster backend from the user
• Load-balancer backend instances
• Recognize down hosts
• Fair scheduler
• (Deliver static content)
48
Apache 2.2
• Apache 2.2 introduced mod_proxy_balancer
• mod_proxy_balancer can speak to multiple backends and balance requests
• Apache can acts as a pure proxy or can also serve static files
49
Apache 2.2
• Apache 2.2 introduced mod_proxy_balancer
• mod_proxy_balancer can speak to multiple backends and balance requests
• Apache can acts as a pure proxy or can also serve static files
50
Apache 2.2
• Apache 2.2 introduced mod_proxy_balancer
• mod_proxy_balancer can speak to multiple backends and balance requests
• Apache can acts as a pure proxy or can also serve static files
51
Apache 2.2
Pro
• Stable, robust, and mature
• Many people know how to work with Apache
• Integrates well with other modules (SVN, DAV, Auth, …)
Con
• Apache can be complicated to configure
• The stock Apache is quite resource-hungry compared to pure proxy solutions
52
Nginx – From Russia with love
• Nginx - popular Russian webserver with good proxy support
• Can load-balance multiple backends and deliver static content
• Quite popular with Mongrel as the Rails backend
• Recent Passenger support
53
Nginx Configuration
Simple proxy example
Get complete version here:
http://brainspl.at/nginx.conf.txt
54
Nginx
Pro
• Stable, robust, and fast
• Uses fewer resources (CPU and RAM) than Apache for proxy-mode and static files
• Simpler configuration file
• Can directly talk to memcached - SSI
Con
• More documentation would be nice
• No equivalent for many Apache modules
55
Lighttpd
• Lightweight and fast webserver
• Balancing proxy support
• Good FastCGI support
• Used to be popular – until Mongrel came around
56
Lighttpd Configuration
Simple proxy example
57
Lighttpd
Pro
• Fast and lightweight
• Uses fewer resources (CPU and RAM) than Apache for proxy-mode and static files
• Simpler configuration file
Con
• Unstable for some people
• Slow development cycle
• More documentation would be nice
• Configuration file can be too simple (virtual host aliasing)
• No equivalent for many Apache modules
• (No Passenger support)
58
HA-Proxy
• HAProxy – reliable, high performance TCP/HTTP load balancer
• Proxying and content inspection
• No content serving, just a proxy
• Mature proxy module (fair scheduler)
• ACL support
See also similar Pound and Pen
59
HAProxy
Simple proxy example
60
HAProxy
Pro
• Mature, stable, robust, and fast
• TCP and HTTP balancing
Con
• Few Rails examples
• Usually not needed in a Rails setup
61
Recommended Setups
62
Small Site
Recommendation
Apache 2.2 with mod_rails / Phusion Passenger
63
Medium Site
Recommendation
• Apache/Nginx as the frontend proxy
• Passenger/mod_rails as the backend
• Deliver static files with Apache
64
Large Rails Setup
Recommendation
• Redundant load-balancer
• Redundant proxy / web
• Passenger/mod_rails
65
Heavy Static Files
Recommendation
• Deliver static assets through separate web server farm
• Passenger/mod_rails
66
Java Shop
Recommendation
• Deliver a Rails-WAR file and you are done
• Integrate with existing Java landscape and infrastructure
67
68
Application Server Handler
69
Application Server Handler
70
Application Server Handler
71
Rack
72
Rack Handler
73
Rack
74
Rack Middleware
75
Rack Middleware
• Authentication (HTTP, OpenID)
• Sessions
• Routing
• Caching
• Logging
• Debugging
76
Support
Frameworks
• Rails
• Merb
• Camping
• Sinatra
• Mack
• Maveric
• Halcyon
• Ramaze
• Rum
• Vintage
• Waves
Web Server
• CGI
• FastCGI
• Mongrel
• Phusion Passenger
• JRuby Servlets
• WEBrick
• Ebb
• Thin
• LiteSpeed
• Swiftcore
• Unicorn
77
Remarks
78
Ruby Enterprise Edition
• Copy-On-Write patches to Ruby 1.8
• Saves memory when spawning several Rails instances
• Used by Phusion Passenger if available
79
Thin, Ebb, Evented Mongrel & Co.
• Alternatives to Mongrel
• Claim to be faster, lighter, and what have you
• Rendering “Hello World” is usually not your bottleneck
Stick with stable and robust Mongrel, or better yet, with Passenger
80
Infrastructure Deployment
81
82
Deployment Options
83
Deployment Options
84
Deployment Options
85
Deployment Options
86
What does Capistrano do?
87
Capistrano Deployment Cycle
88
Requirements
89
What doesn’t Capistrano do?
• Plan your initial server setup
• Configure basic services
90
Basic Ingredients
• The cap command
• Variables
• Roles
• Tasks
• Namespaces
91
Basic Ingredients - cap
Your one-stop deployment shop
92
Basic Ingredients - Variables
• Configure basic project information
• Override Capistrano’s default assumptions
• Once set, variables are available globally
• Defined using the set method
93
Basic Ingredients - Roles
• Define types of servers
• Default roles
• :www
• :app
• :db
• All can point to the same server
• But all three must be defined
• At least one database server needs to be primary
94
Basic Ingredients - Roles
Define custom roles as you please
Can be reused when defining tasks
95
Basic Ingredients - Tasks
• Define an atomic set of actions for Capistrano
• Can be called from the command line
• Or other tasks
96
Basic Ingredients - Tasks
To find all the tasks available in your project, use
97
Basic Ingredients - Namespaces
Group tasks together logically
Namespaces and tasks are separated with “:”
98
Get Your Capistrano On
99
Get Your Capistrano On
Capfile, the place to include more recipes
100
Get Your Capistrano On
config/deploy.rb, application specific configuration
101
Capistrano’s Defaults
• Your SCM is Subversion
• Deployment directory is /u/apps/#{application_name}
• User for SCM and SSH is the currently logged-in user
• Commands are run with sudo
102
Get Your Capistrano On
103
Get Your Capistrano On
• Capistrano expects a directory structure
• Can be created with cap deploy:setup
104
The Deployment Lifecycle
105
The Deployment Lifecycle
Check the prerequisites:
106
The Deployment Lifecycle
Set up your application for the first time
107
The Deployment Lifecycle
The initial deployment
1. Checks the revision from the local machine
2. Checks out the code on the remote machines
3. Sets a link called current pointing to the lates release
4. Runs the migrations
5. Fires up application servers
108
The Deployment Lifecycle
109
The Deployment Lifecycle
Subsequent deployments
1. Checks the revision from the local machine
2. Checks out the code on the remote machines
3. Updates current link
4. Restarts application servers
110
The Deployment Lifecycle
111
Common Capistrano Tasks
Deploy and run migrations
Run only the migrations
Restart application servers
Rollback to the previous release
112
Have Your Shell, and Eat It Too
113
Invoking any Command
114
Deployment Strategies
115
Deployment Strategies
Direct checkout (from scratch) on the servers
116
Deployment Strategies
Keep a cached copy of the current SCM head
117
Deployment Strategies
Check out locally and transfer
118
Give it a little Spin
Capistrano expects a script called spin in script/process
For Passenger
119
Customizing Capistrano
120
Write your own Tasks
121
Write your own Tasks
122
Namespace your Tasks
123
Callbacks
Execute a task before another runs
Execute a task after another has finished
Callbacks are run in the order they’re defined
124
Useful Variables
125
Transactions
126
Transactions
127
Transactions
128
Transactions
129
The Rest
• Gem dependencies
• Support for deploying through gateway servers
• Parallel execution on multiple servers
• Server setup with deprec gem
130
Vlad the Deployer
131
Vlad the Deployer
• Simple, Rake-based deployment solution
• Comes with defaults for most popular SCM, app and web servers
132
Vlad the Deployer
Load
Configure
133
Vlad the Deployer
Deployment
134
Vlad the Deployer
• Simple, sometimes too simple
• No callback mechanism, instead the Rake syntax must be used
• Very non-verbose output, using --trace recommended
135
Chef
• System Integration Framework
• Written in Ruby – inspired by Puppet
• Manage multiple nodes
• Configuration
• Packages
• Custom scripts and commands
136
Architecture
137
Recipes
Install and upgrade packages
138
Recipes
Create and update remote files
139
Recipes
Create and update remote files
140
chef-deploy
Enable
Use in recipe
141
Chef and chef-deploy
• In heavy development: sometimes unstable and hard to debug
• Be prepared to get your hands dirty
• Better suited for configuration than deployment
• Community recipes at http://github.com/opscode/cookbooks
142
One Click Deploy
143
Webistrano
144
Webistrano
• Web-UI to Capistrano
• Manages projects and their stages
• Alerting and Accounting
• Scriptable and extendable
• BSD License
http://labs.peritor.com/webistrano
145
Macistrano
• Mac-GUI to Webistrano
• Fire and monitor deployments from your desktop
http://github.com/mattmatt/macistrano
146
Infrastructure Practical Session
147
Practical Capistrano
148
EC2 Instances
• Users
• root / railswaycon
• deploy / railswaycon
• MySQL
• root / ‘’
• Apache Configuration
• /etc/apache2/sites-enabled/default
• Rails
• /srv/www
149
Git Repository
• git://github.com/mattmatt/railscasts.git
• Setup the servers
• Deploy!
150
Tasks
• Make memcached restart with every deploy
• (Usually not a good idea, but why not?)
151
Tasks
• Make Apache recognize maintenance.html and display it if exists
152
Maintenance Page
• Create symbolic link to /etc/apache2/mods-available/rewrite.load in /etc/apache2/mods-enabled
• Update virtual host configuration with rewrite rules
• Run cap deploy:web:disable • Hook it into the deployment process
153
Tasks
• Deploy a Rack-based application
• Install Sinatra locally and on the server
• git://github.com/mattmatt/railswaycon.git
• Create deployment configuration
• Create Apache configuration
154
Infrastructure Monitoring
155
The two questions of monitoring
156
1. Is everything still running?
157
2. What are the trends?
158
Monit
• Process-level monitoring
• Checks PID-files, ports, and permissions
• Reacts by executing a script and/or alerting
• Has a web GUI for administering nodes
159
Monit
MySQL
160
Monit
Apache
161
Monit
Mongrel
162
God
• Ruby-based configuration
• Easily extendible
• Can work with event- and poll-based conditions
• Comes with less recipes than Monit
163
God
Mongrel
164
Munin
• Host-level monitoring
• Master periodically asks nodes for local data
• Checks system resources and records historical data
• Allows to recognize trends and make predictions
• Alerting support
165
Munin
166
Munin
167
Munin
168
Exception Monitoring
• Monitor errors raised by your application
• Usually notifying you by email
• Good old exception_notification Plugin
• Exceptions as a Service
169
Hoptoad
170
Exceptional
171
New Relic
• Rails Performance monitoring
• Live view into your application
172
Other Tools
• Nagios
• Big Brother
• New Relic RPM
• FiveRuns
• JMX
173
Infrastructure Advanced Deployment
174
Search
175
Search
Full text search
Can become very slow on big data sets
176
Full Text Search Engine
Separate Service
• Creates full text index
• Application queries search daemon
• Index update through application or database
Possible Engines
• Ferret
• Sphinx
• Solr
• Lucene
• …
177
Search Slave
Database replication slave
• Has complete dataset
• Migrates slow search queries from master
• Can use different database table engine
178
Database Index
PostgreSQL Tsearch2
• Core since 8.3
• Allows to create full text index on multiple columns or arbitrary SQL expressions
MySQL MyISAM FULLTEXT index
• Only works with MySQL <= 5.0 and MyISAM tables
• Full text index on multiple columns
179
What to use?
Different characteristics
• Real-time updates and stale data
• Lost updates
• Performance
• Document content and format
• Complexity
180
Background Processing
181
Problem
Long running tasks
• Resizing uploaded images
• Mailing
• Computing an expensive operation
• Accessing slow back-ends
When running inside request-response-cycle
• Blocks user
• Blocks Rails instance
• Hard to monitor and debug
182
Solution
Asynchronous processing in the background
Message/Queue Scheduler
183
Background Processing
184
Options
Options for message bus:
• Database
• Amazon SQS
• Drb
• Memcache
• ActiveMQ
• …
Options for background process:
• (Ruby) Daemon
• Cron job with script/runner
• Forked process
• Delayed Job / BJ / (Backgroundrb)
• run_later
• ….
185
Database/Ruby daemon example
186
Cloud Infrastructure
187
Cloud Infrastructure
Servers come and go
• You do not know your servers before deploying
• Restarting is the same as introducing a new machine
You can’t hardcode IPs
database.yml
188
Solution #1
Query and manually adjust
• Servers do not change that often
• New nodes probably need manual intervention
• Use AWS ElasticIPs to ease the pain
Set servers dynamically AWS Elastic IP
189
Solution #2
Use a central directory service
• A central place to manage your running instances
• Instances query the directory and react
190
Solution #2
Use a central directory service
• A central place to manage your running instances
• Instances query the directory and react
191
Central Directory
Different Implementations
• File on S3
• SimpleDB
• A complete service, capable of monitoring and controlling your instances
192
Q & A
193
193
Peritor GmbH
Teutonenstraße 16 14129 Berlin
Telefon: +49 (0)30 69 20 09 84 0 Telefax: +49 (0)30 69 20 09 84 9
Internet: www.peritor.com E-Mail: kontakt@peritor.com
Peritor GmbH - Alle Rechte vorbehalten
top related