OpenStackCeilometer

Tong Li (IBM)

Brad Topol (IBM)

Agenda• What is Ceilometer?

• Brief history of Ceilometer

• Meters of Ceilometer

• Workflow

• Ceilometer Architect & Building Blocks

Collecting Process Pipeline

Transformer Publisher

Collector Data store drivers

Alarm Dispatcher

• Ceilometer ReSTful API

• Ceilometer Python Client

• Ceilometer in Devstack

• Inter-Component Security & High Availability

• Who are the users of OpenStack Ceilometer

• Ceilometer Road Map

• Q & A (Demo maybe)

Who are we?• Tong Li

– IBM senior software engineer

– Twitter @email4tong

– Email: litong01@us.ibm.com

• Brad Topol

– IBM distingushed engineer

– Twitter @bradtopol

– Email: btopol@us.ibm.com

What is Ceilometer?

“OpenStack measurement project”

Julien Danjou Ceilometer PTL

A system that keeps track of what is happening in

OpenStack clusters. It was originally created for billing but it has been evolved into an all purpose

measurement repository. It can be extended for purposes such as performance report,

troubleshooting, capacity analytics, smart

scheduling, optimization, ultimately BIG DATA etc.

Brief History of Ceilometer• Before year 2012 billing has been left out of OpenStack core so far

as it was not the primary problem and was non-trivial, yet almost every OpenStack deployment needs a way to track usage information

• Started in May 2012 by developers from Canonical, DreamHost, RedHat, Dell, Intel, AT&T, etc.

• Targeted to do metering, rating, billing

• Grizzly objective: The project aimed to become the infrastructure to collect measurements within OpenStack so that no two agents would need to be written to collect the same data.

• Havana objective: remains the same as Grizzly objective plus become an integrated OpenStack project.

• Current PTL: Julien Danjou

• 91 contributors, 7 core reviewers

Meters of Ceilometer (from Nova)

Meters of Ceilometer (from Neutron)

Additional measurements:

http://docs.openstack.org/developer/ceilometer/measurements.html

Workflow

• Collect interesting info from OpenStack components

(or any other components), create meters (or samples)

• Transform meters into other meters if needed

• Publish meters to any destination (including Ceilometer itself)

• Store received meters into data store

• Read data via Ceilometer REST API

TransformCollect Publish Store Read

Ceilometer Architecture

Collecting

Neutron SwiftGlanceCinder

Notification bus (AMQP) queue name: notification.info

Notification Handlers, Central Agents

APIAPI API API

Nova

API

Publishing pipeline Publishing

Polling

Ceilometercollector & agent

ReceiverReceiver

PublisherPublisher

Pipeline

TransformerMeterTransformer

Transformer

Publisher Receiver

Pipeline: a set of transformers mutating meters into something that publishers know how to send to external systems.

Configure pipelines @pipeline.yaml

New in Grizzly!

Transformer

MeterName=cpu.time

Value=5Timestamp=T

Transformer

MeterName=cpu.time

Value=20

Timestamp=T+1

MeterName=cpu.time

Value=80

Timestamp=T+2

MeterName=cpu.percentage

Value=9Timestamp=T+2

Transform meters into new meters!

New in Grizzly!

Publisher

MeterCeilometer

RPCPublisher

Other Publisher

CeilometerMessage queue

External system

AMQP

signed message

Billing, monitoring,

alarming, statistics,

capacity planning…

New in Grizzly!

Data Store

Message Queueceilometer.collector.metering

SQL Database

MongoDB Database

Meter

Dispatchers

Multiple backend choices

Default backend

HBase

IBM DB2

New inGrizzly

New inHavana

Alarm

Both alarm evaluator and alarm notifier are Ceilometer services

1. Alarm evaluator uses Ceilomter public APIs to pull meters to produce alarms according to alarm definition, then place the alarm message onto the alarm queue

2. Alarm notifier read the alarm queue then inform external systems (log, rest, test)

New inHavana

DispatchersNew in

Havana

Multiple dispatchers support allows Ceilometer to be extended easily with many different outlet for all meters coming into Ceilometer. Dispatchers can be easily developed and deployed to add more features. For example, sending email, text messages.

1. Dispatcher only needs to implement one method.

2. By default, database dispatcher is used even if the configuration file has no trace of any dispatchers.

3. To configure multiple dispatchers:[ceilometer.dispatcher]file = ceilometer.collector.dispatcher.file:FileDispatcherdatabase = ceilometer.collector.dispatcher.database:DatabaseDispatcher

Ceilometer ReSTful APIrequest

Database

Ceilometer APICeilometer API

Consumers

Billing, monitoring,

alarming, statistics, capacity planning…

HTTP

REST

w/ JSON &

XML

Storage abstraction

layer

Raw events:GET /v2/meters/vcpus

Statistics (sum, average, min, max…):GET /v2/meters/vcpus/statistics

Filter:GET /v2/meters/vcpus?q.field=project_id&q.value=foobar&q.op=eq

Group statistics by period:GET /v2/meters/vcpus/statistics?period=3600

New in Grizzly!

Ceilometer ReSTful APIresponse

Response can be in JSON or XML

Raw events:/v2/meters/storage.object.size

Statistics (sum, average, min, max…):/v2/meters/storage.api.request/statistics

Ceilometer Python Client

Ceilometer in Devstack

devstack localrc

• Enable the ceilometer metering services

enable_service ceilometer-acompute,ceilometer-acentral,ceilometer-collector

• Enable the ceilometer alarming services

enable_service ceilometer-alarm-evaluator,ceilometer-alarm-notifier

• Enable the ceilometer api services

enable_service ceilometer-api

Inter-Component Security

Ceilometer High Availability

Who uses Ceilometer?

https://etherpad.openstack.org/p/CeilometerFeeedback

Roadmap

Havana Icehouse

• Integrated Project ✓• Integration with Horizon• Publishing meters to other

systems ✓• Enhance SQL driver ✓• Alarming ✓• Integration with Heat• Deprecating APIv1 ✓• Completing APIv2 ✓• Move publishing part to

Oslo and other projects ✓• Tighter integration with

Nova ✓• Nova-scheduler integration

✓

• Incubated Project ✓• Integration

with Horizon ❌• Agents for other

componentso Swift ✓o Ceph? ❌o Nicira? ❌

• SQLAlchemy storage driver ✓

• Multi-Publisher ✓• API v2 ✓

o User accessible API

✓

o More aggregation ✓

o Multi-dimension ✓

Grizzly

• OpenStack Design Summit in Hong Kong

Questions?

http://launchpad.net/ceilometer

http://docs.openstack.org/developer/ceilometer

http://wiki.openstack.org/ceilometer

Freenode: #openstack-metering

Mailing List: openstack-dev [ceilometer]