fluentd meetup logging infrastructure in paa s
DESCRIPTION
Rakuten, Yohei Sasaki, Waldemar Quevedo, at Fluentd meetup, August 22nd, 2012TRANSCRIPT
1
Logging infrastructure in PaaS with Fluentd
Rakuten. Inc, 22/08/2012
Yohei Sasaki / Waldemar Quevedo Paas Dev&Ops team
2
About us
§ ワリと申します。メキシコ人です。 (Wally, from Mexico)
§ Living in Tokyo since 2010.
§ Member of the PaaS Dev&Ops team at Rakuten, programming in Ruby.
§ Emacs user (org-mode rocks, try .org markup in Github)
§ OpenStreetMap (Mapnik rocks)
§ ささきと申します。日本人です。 (Sasaki, from Japan)
§ Living in Japan since 1982.
§ Tech Lead of the PaaS Dev&Ops team at Rakuten, Architect of Rakuten PaaS.
§ Node.js user / node-fluentd-logger
§ CouchDB, …etc
3
Agenda
§ Cloud Foundry
§ Cloud Foundry + Fluentd § Fluentd Pros & Cons § Misc
4
Cloud Foundry
• Open Application Platform as a Service project
• By VMware
• Written in Ruby
• OpenSource http://github.com/cloudfoundry/vcap • Hosted version http://www.cloudfoundry.com/
5
Why Cloud Foundry?
• Last year it was decided to introduce Cloud Foundry to build a PaaS in Rakuten
http://techtarget.itmedia.co.jp/tt/news/1111/28/news02.html http://www.slideshare.net/rakutentech/cloud-foundry-10423477 • Main points:
– Open Source – Separation of concerns from IaaS – Scalable architecture – Multi language
6
What Cloud Foundry does?
• Easy to deploy applications!!
$ mkdir app.js $ vi app.js $ vmc push yssk22-myapp $ vmc instances yssk22-myapp 3
• Not easy to deploy business ready applications yet. – We are now solving our own issues by modifying Cloud Foundry source
– One critical thing is Logging!
– Now we’ve solved this matter basically, let’s share our case study.
7
Current logging issue in Cloud Foundry
$ vmc push yssk22-myapp!!$ curl http://yssk22-myapp.cloudfoundry.com/!!$ vmc logs yssk22-myapp!![Wed, 22 Aug 2012 08:03:48 GMT] Hello World! !!$ vmc update yssk22-myapp!!$ vmc logs yssk22-myapp!
8
Current logging issue in Cloud Foundry
$ vmc push yssk22-myapp!!$ curl http://yssk22-myapp.cloudfoundry.com/!!$ vmc logs yssk22-myapp!![Wed, 22 Aug 2012 08:03:48 GMT] Hello World! !!$ vmc update yssk22-myapp!!$ vmc logs yssk22-myapp!
Oops, logs are gone.
9
Persisting logs is a must for business applications!
• Monitoring • Analysis • Visualization
Ask a data scientist =>
http://www.slideshare.net/doryokujin/hadoop-and-the-data-scientist
Sorry we lost everything :P
10
Cloud Foundry does not persists the logs from apps…
• Enter Fluentd!
11
Brief overview of Cloud Foundry architecture
Source: http://www.slideshare.net/derekcollison/design-of-cloud-foundry
Droplet
Execution
Agent
12
Approach
DEA
app
app
fluentd
fluentd
DEA
app
app
fluentd
fluentd
Attach a Fluentd collector to one application instance upon dispatch.
Log Storage
13
Collecting the logs from the apps
• Log collector monitors changes in the files – Stdout, stderr logs in general – Catalina, manager, host-manager logs for Java – error.log for PHP apps
• Also makes available:
– VCAP_LOG_PORT env variable to flush logs directly
– VCAP_LOG_STREAM_PORT to inspect the logs in realtime
DEA
app
app
fluentd
fluentd
app fluentd
app fluentd
app fluentd
14
Collecting the logs from the apps
DEA
app fluentd
Fluent::Logger => VCAP_LOG_PORT
Sends logs directly to Fluentd making unnecessary having to write to file. VCAP_LOG_STREAM_PORT
Streams the logs to a client {‘type’: ‘serverLog’, ‘text’: ‘Hello World’, ‘level’: ‘INFO’}
Log Storage
15
Small setup for collecting the logs
Log Storage Server
fluentd Rack app
CloudController
Collected log files
DEA
app
app
fluentd
fluentd
DEA
app
app
fluentd
fluentd
Log Storage Server
Collected log files
/logs/:path
CloudController
/logs/:path
Rack app
16
How is it setup?
Attached to an application instance on startup by DEA
• (1) and (2) is a “stat file” used by the input plugin, which allows it to pick up from the last position it had monitored.
• (3) is the configuration of the plugins dynamically
generated by the DEA • (4) and (5) are the scripts executed to manage the
fluentd process • (6) and (7) are the supervisor and main process
ids from Fluentd • (8) is where any logs produced by the logging
daemon are stored
app/ !
fluentd/!
|-- stderr.stat (1)!
|-- stdout.stat (2)!
|-- fluentd.conf (3)!
|-- startup (4)!
|-- stop (5)!
fluentd_child.pid (6)!
fluentd.pid (7)!
logs/!
|-- fluentd.log (8)!
|-- stdout.log!
|-- stderr.log!
ruby!
run.pid!
startup!
stop!
17
fluentd.conf :: Input configuration
!
<source>!
type forward!
port 12345!
</source>!
!
<source>!
type cf_app_logs!
instance_dir /var/vcap.local/dea/apps/appname-0-3e9676485562c64d81a61de2df05903a!
runtime ruby19!
framework sinatra!
tag logs.storage!
</source>!
!
!
Set up VCAP_LOG_PORT to flush the logs directly
Specify runtime and dir to monitor files
18
fluentd.conf :: Output configuration
<match logs.**>!
type copy!
<store>!
type cf_app_forward!
flush_interval 5s!
buffer_type file!
buffer_path fluentd/fluentd_buffer!
!
cf_app_user [email protected]!
cf_app_name appname!
cf_app_instance_id 3e9676485562c64d81a61de2df05903a!
cf_app_instance_index 0!
Settings for when forwarding to the log storage, we include user information here so that it cannot be overrided by a logging library.
19
fluentd.conf :: Output configuration
<server>!
name logserver!
host 192.168.2.1!
port 4224!
</server>!
<server>!
name backup_logserver!
host 192.168.2.2!
port 4224!
standby!
</server>!
</store>!
<store>!
type cf_app_logs_streaming!
port 43208!
</store>!
</match>!
Location of the log storage server: - We use a active-standby setup at the
moment. - When primary Logserver is down, logs
are forwarded to the one on standby. When both are down, fluentd buffers to files the logs locally until one of them is available.
Set up VCAP_LOG_STREAM port to enable tail –f like support.
20
…And now that we managed to collect the logs,
we also can support the following 2 commands:
21
vmc tail
22
vmc log-storage
23
What we could do
• Persisting the logs in Cloud Foundry is a must! • Support to persist the logs is a key issue in adoption of Cloud
Foundry for developing application services.
• Thanks to Fluentd we could enable users to have log archives in a straightforward way.
• Cloud Foundry and Fluentd themselves are both written in Ruby so integration was relatively easy.
24
What we could not do
• It was difficult to let the user have access to the logs from Nginx (used as a Router in CF)
• Then the app-v1.cf.rakuten.co.jp subdomain becomes available to other user.
• Need to keep track that user X had Y domain at Z time for application named A in real time while storing the logs.
# Sample release process!
vmc push app-v1 --runtime ruby19 => app-v1.cf.rakuten.co.jp!
vmc map app-v1 olympics.rakuten.co.jp => Released!!
vmc push app-v2 --runtime ruby19 => app-v2.cf.rakuten.co.jp!
vmc map app-v2 olympics.rakuten.co.jp => Load balancing between apps!
vmc unmap app-v1 olympics.rakuten.co.jp => Release 2 complete.!
25
Fluentd: Pros and Cons
• Pros – Easy to include in Cloud Foundry compared to other solutions
• Possible to include it like gem 'fluentd' in a Ruby component – Easy to change storage technology depending on scale – Easy to extend – Growing community \o/
• Cons – Fluentd needs better testing tools – Better materials about Fluentd from the community?
• Learning curve is already low • Just we need to share more about how we are using Fluentd • e.g. 'The Little Fluentd Book', 'Advanced Fluentd Recipes’
– Cool.io is not maintained
26
Misc: Playing with the message bus
• Cloud Foundry is built with Eventmachine.
• Fluentd is built with Cool.io.
• Cool.io and Eventmachine do not play along very well.
• The message bus from Cloud Foundry named NATS is also built using Eventmachine.
27
Misc: NATS input plugin
# This works, though a warning is displayed from Cool.io overriding Eventmachine!
require 'nats/client'!
!
module Fluent!
class NatsInput < Input!
Plugin.register_input('nats', self)!
def run!
NATS.start(:uri => @nats_connection_uri) do!
!
# Triggered when an url changing event occurs (remap, change)!
NATS.subscribe('dea.update') do |msg|!
Engine.emit('logs.dea_update', Time.now.to_i, msg)!
end!
!
# Triggered when an application is started or dea is started!
NATS.subscribe('dea.*.start') do |msg|!
Engine.emit('logs.dea_start', Time.now.to_i, msg)!
end!
end!
end!
28
Misc: Adding new forwarding nodes dynamically through NATS?
# DEA
@droplets.each_value do |instance|
begin
instance_collector = Fluent::Logger::FluentLogger.new('logs', instance[:logging_ip], instance[:logging_port])
instance_collector.post('storage', {'type': 'serverConf', 'host': '127.0.0.1', 'port': 23938 })
rescue => e
@logger.error "Error: #{e} -- #{e.backtrace}"
end
end
# Extended Forward plugin
module Fluent
class CloudfoundryAppForwardOutput < ForwardOutput
Plugin.register_output('cf_app_forward', self)
def emit(tag, es, chain)
es.each do |emit_time, record|
# Add a new forwarding server
if record.has_key?('type') and record['type'] == 'serverConf’
add_node(record['host'], record['port’])
next
end
29
FIN
You can enable log collection to your Cloud Foundry by merging this patch: https://github.com/rakutentech/dea/commit/64d271904de1e195cc90ca1958eadf704f530d94 Checkout our github repo for more Cloud Foundry related things: https://github.com/rakutentech/ That’s all, thanks!
ご清聴ありがとうございました!