nosql databases and monitoring - it monitoring working...
TRANSCRIPT
PES
CERN IT DepartmentCH-1211 Geneve 23
Switzerlandwww.cern.ch/it
CERNITDepartment
NoSQL Databases and MonitoringIT Monitoring Working Group
Jerome Belleman Christos MargiolasCERN – IT-PES
19 September 2011
PES
CERN IT DepartmentCH-1211 Geneve 23
Switzerlandwww.cern.ch/it
CERNITDepartment
2 – NoSQL Databasesand Monitoring
Outline
1 Monitoring Context
2 NoSQL Databases for Monitoring Purposes
3 Examples
4 Outlook
PES
CERN IT DepartmentCH-1211 Geneve 23
Switzerlandwww.cern.ch/it
CERNITDepartment
3 – MonitoringContext
Section 1
Monitoring Context
PES
CERN IT DepartmentCH-1211 Geneve 23
Switzerlandwww.cern.ch/it
CERNITDepartment
4 – MonitoringContext
CERN Batch System Monitoring
Investigate more advanced features of the batch system
Reduce problem identification time
Lego-like, interchangeable set of building blocks
Batch System
Collection
Transport
Processing StorageAnalysis
Live Historical
PES
CERN IT DepartmentCH-1211 Geneve 23
Switzerlandwww.cern.ch/it
CERNITDepartment
5 – MonitoringContext
What to Monitor?
Jobs (number, CPU usage, I/O, . . . )
Users
Hosts
Slots
Queues
. . .
Ñ Correlations & heavy analytics
Live representations
Historical representations
Ñ Browse metrics over time
PES
CERN IT DepartmentCH-1211 Geneve 23
Switzerlandwww.cern.ch/it
CERNITDepartment
6 – MonitoringContext
How Much to Monitor?
ą 200 000 jobs/day
ą 3 700 nodes
ą 30 000 cores
PES
CERN IT DepartmentCH-1211 Geneve 23
Switzerlandwww.cern.ch/it
CERNITDepartment
7 – MonitoringContext
What Do We Need?
2 Scalability
2 Data collection Ñ Many small writes, very often
2 Heavy analytics Ñ Massive reads, quite often
2 Horizontal & vertical aggregations
2 Time views
2 Flexible schemas
Ñ Let’s investigate the use of a NoSQL database
PES
CERN IT DepartmentCH-1211 Geneve 23
Switzerlandwww.cern.ch/it
CERNITDepartment
8 – NoSQL Databasesfor Monitoring
Purposes
Section 2
NoSQL Databases for Monitoring Purposes
PES
CERN IT DepartmentCH-1211 Geneve 23
Switzerlandwww.cern.ch/it
CERNITDepartment
9 – NoSQL Databasesfor Monitoring
Purposes
General Concepts
NoSQL ù Non Relational
A broader set of structured storage
Different consistency levels
Different classes of NoSQL databases
PES
CERN IT DepartmentCH-1211 Geneve 23
Switzerlandwww.cern.ch/it
CERNITDepartment
10 – NoSQLDatabases for
Monitoring Purposes
Different Classes of NoSQL Databases
Column-oriented databases (e.g. Google BigTable,Apache HBase, Apache Cassandra)
Key-value stores (e.g. Amazon Dynamo, ProjectVoldemort, MongoDB, Riak, Google BigTable,Apache Cassandra)
Document stores (e.g. Apache CouchDB, MongoDB)
We’ve been looking at Cassandra.
PES
CERN IT DepartmentCH-1211 Geneve 23
Switzerlandwww.cern.ch/it
CERNITDepartment
11 – NoSQLDatabases for
Monitoring Purposes
Key-Value Stores such as Cassandra
k0 c0,0 c0,1 c0,2 c0,3 c0,4
k1 c1,0 c1,1 c1,2
k2 c2,0 c2,1 c2,2 c2,3 c2,4 c2,5
k3 c3,0 c3,1
k4 c4,0 c4,1 c4,2 c4,3
k5 c5,0 c5,1 c5,2
PES
CERN IT DepartmentCH-1211 Geneve 23
Switzerlandwww.cern.ch/it
CERNITDepartment
12 – NoSQLDatabases for
Monitoring Purposes
How Fast?
(Quite) Simple-minded
High throughput
Horizontal scalability
No need for object-relational mapping
PES
CERN IT DepartmentCH-1211 Geneve 23
Switzerlandwww.cern.ch/it
CERNITDepartment
13 – NoSQLDatabases for
Monitoring Purposes
What Do We Need?
2 Scalability
2 Data collection Ñ Many small writes, very often
2 Heavy analytics Ñ Massive reads, quite often
2 Horizontal & vertical aggregations
2 Time views
2 Flexible schemas
Ñ Let’s investigate the use of a NoSQL database
PES
CERN IT DepartmentCH-1211 Geneve 23
Switzerlandwww.cern.ch/it
CERNITDepartment
13 – NoSQLDatabases for
Monitoring Purposes
What Do We Need?
2� Scalability
2� Data collection Ñ Many small writes, very often
2� Heavy analytics Ñ Massive reads, quite often
2� Horizontal & vertical aggregations
2� Time views
2� Flexible schemas
Ñ Let’s investigate the use of a NoSQL database
PES
CERN IT DepartmentCH-1211 Geneve 23
Switzerlandwww.cern.ch/it
CERNITDepartment
14 – Examples
Section 3
Examples
PES
CERN IT DepartmentCH-1211 Geneve 23
Switzerlandwww.cern.ch/it
CERNITDepartment
15 – Examples
A Time Series Database: OpenTSDB
On top of Apache HBase
Horizontal aggregation Ñ Free time browsing
Vertical aggregation Ñ Drill down details
PES
CERN IT DepartmentCH-1211 Geneve 23
Switzerlandwww.cern.ch/it
CERNITDepartment
16 – Examples
Job Performance Analysis: Cassandra
Batch
Cassandra
Batch
Cassandra
Batch
Cassandra
Batch
Cassandra
ActiveMQQueues
Consumer
ActiveMQQueues
Consumer
PES
CERN IT DepartmentCH-1211 Geneve 23
Switzerlandwww.cern.ch/it
CERNITDepartment
17 – Examples
Fairshare Monitoring: Cassandra
Reading from bhpart output
ą 30 000 groups/users every 5 minutes
malberti
ebye2
collazug
emicelot
SHARE
u_GEANT4
kmassri
andreev
michal
pimia
cerri
kresan
awojtasz
sreia
mcortesi
kulikov
gentile
gear049 gear048
vormc
sbianucc
gear043
gear042gear041
gear040
gear047gear046
gear045kunin
fantechi
jhakola
jcs
smele
mchizhov
msando
kuptsov
na49
palim
legou
jsmolik
unger
ams
atonazz
na48cdr
na48u010
bourquin
korolko
ccasella
torimoto
ams
sgiani
glamanna
wislicki
marusov
rondio
dangelop
na48wien
ecoucemo
abaas
pmatak
luckey
scprod
aelagin
wangqin
blochp
kperl
emonnier
na48plot
na48u005na48u004
na48u007
na48u006
na48u001
na48u003
na48u002
ldjambaz
markytanna48u008
gabri
tvrba
janot
ifisk
denisd
runna49
polenkev
berdugo
karpukhn
dgibin
lferreia
caragxv
malgeri
sgallori
lourenco
ecortina
kwosinsk
mdoucet
karev
emarinov
mthomann
zrelov
acecchet
dumitriu
u_THEORY
pfilips
fedine
dikolev
muijs
malahov
miliescu
norton
na48uk
silviac
fischerg
xma
karyotak
barr
mvaldata
dschmitz
jurajb
gchen
mbigliet
ajago
eperez
mveltri
mion
u_ILC
wotton
rbatley
gear010
gear011 gear012
na48kab
gear014gear015
gear016gear017
gear018
gear019
lkrouglo
ctimmer
schioppo
okounina
colino
goylopez
josa
akorteni
zilizi
massimop
hwalther
bigroots
radoslaw
tpalczew
ruggierg
dagnolo
cciortea
jhegeman
lomidze
clare
patel
olgortch
u_HARPD
aartamon
veenhof
doudarev
l3cos
pchung
kleinkne
marek
bertucci
chiarusi
peyaud
malgeril
safarzad
production
u_NA48CALI
ceccucci
voloshyn
harpshif
zhuanghl
cerminar
na48tech
u_NA61
gpalla
egamberi
dai
jacuh
don
wedlake
winhart
fergus
gianoli
zimanyi
malinin
wjb
hristov
gruenew
capell
robbins
palencia
l3dbsm
saogaki
simoing
pgorb
mvorobie
munday
islepnev
nabgrall
rstock
schach
desalvo
querten
crovelli
na48nut
passelev
newman
sergiant
amsf4
jabitski
bpanzer
takeuti
rvasquez
aboehm
golubeva
duchesne
ondrejch
acgroup
ljones
bagoulia
u_CMST3
alexg
l3prd
dflueras
segoni
stefanpvna48expo
bdulach
pohl
kkousour
gajewski
mclavier
vpoliako
stroebel
u_CMS
na48cali
mcampan
sbologne
yoo
takeut
begona
braccini
tully
mlenti
gear029
gear028
palladuh
na48mbx
formica
gear021
gear020
gear023
gear022
cenci
gear024
gear027
kirkby
cdbase
mbsi
u_DTEAM
sgevorgy
biasini
u_FLUKARP
gear050
mpivanti
lboldizs
rparamat
wreece
lconey
raggi
eddag
mgallas
mannellimmojzis
azzarel
na48u002 na48u003
na48u001
na48u006
na48u007na48u004
na48u005na48u008
na48u009
pjindal
bvallage
zalanna
piotr
bvelghe
kurshets
abizzeti
mzaytsev
fmonraba
crosenbl
shkar
merega49
ferella
abenelli
mdepreng
mcristin
dehmeltk
rplaneta
cecchi
coward
lydia
foka
u_TOTEM
ykohl
imbergam
bifani
asryan
vodib
azemoon
na48mc
u_AMS
mpasqual
piruleta
napolita
spadolsk
u_ITDC
mmozer
dyulenda
kadija
rolandc
rgaglion
zaranek
lenzip
villani
derrej
ascott
nefedov
gfaranta
sgiudici
fsikler
mhansrou
jmartinc
dsaborid
vdezso
u_vl
mellis
choehne
tariq
pmeckiff
riazants
vfriese
mfidecar
winston
vagori
choutko
tspadaro
ciaranfi
drizzi
mfiorini
brybicki
cstrabel
sozzi
cabibbo
aaduszki
wache
ronzhin
nikolicv
gearsgm
molokangmills
poluhina
acervera
gnuessl
mnguyen
larazi
pdupre
gear038
clemenci
murphy49
cblume
jgomezca
pdalpiaz
schwerin
andreasp
amoraux
idymniko
mvelasco
fbucci
eusebio
mmass
anzivino
harpmigr
ponta
kprok
ptgruber
ttrojek
carlino
mmitrov
renebrun
u_OPAL
perretg
holzner
vslepnev
maiera
rbramm
hbeck
benedet
u_vk
wborow
brekhovs
gear034
boiko
hwoehri
l3dbback
vassilim
desantoapacheva
berges
ipharp
u_NTOF
ddifilip
sorel
straessn
msolanog
amakarov
valente
mvandena
pernicka
pdoskaro
plyaskin
anniyev
mateevvp
abajovaq
dilella
vberardi
jsmolikp
ovazquez
na48mon
atlasccc
fmayet
rapuano
fjediny
ndavis49
unalserram
ltrnkova
pavelb
linch
gtof
hitahoch
pnaoumen
bagnaia
kokada
litovl
kscholbe
gonidec
kirillov
abis
u_AMSL3
gowdy
jxu
bonaiuto
na48sg04
na48sg05
moronembwarnaar
na48sg01
na48sg02
na48sg03
michiel
u_PARC
cai
jberdugo
azintche
na48cdr
riemanns
gomezu_HARP
nikonov
emelyano
mpierini
diracww
damianic
cmgtools
sandoval
rgulik
dagos
zucuh
diracoff
u_NA48
u_OPERAu_DELPHI
mbehler
l3cos
samyn
pikna
eiacopin
rubinp
fgreco
lvjv
lsfadmvp
lilee
ipopov
rcarosi
bartalin
sradomsk
pjanot
prusapet
belkin
pentia
gear044
na49soft
guglme
jpanman
yw266
u_L3COS
jmartina
gear039
gugiu
cheze
gear032
gear033gear030
gear031
gear036
jeitler
asemak
gear035
zfodor
trdtest
ddv
janas
na48calimasettil
cmorales
wiebusch
lenab
u_ge
mgouzevi
chimenti
sgaleo
cembra
felcini
u_L3
tomaszc
gostkine
lepew
barna
lazzeroc
goldin
jcatalap romano
booth49
others
nabgra49
u_ALEPH
acaraghe
tagger
costantiblochb
sbazylev
mkopal
calt
l3sgm
koreshevrschroet
apeters
u_ATLAS
mestviri
metzger
pmassaro
levpm
sschifan
u_NA48CDR
majam
jkroon
balev
acassese
mnich
gianluca
smaria
zhons
odyniec
cbern
afanasev
mikulec
veres
u_NA49
campore
rancoita
u_ALICE
phodgson
u_xv
bunyatov
mdelgado
rykalin
andreeva
yallkofe
na48grid
awetzler
mserra
meschi
acottara
adeva
frabetti
steuer
tanticic
swallow
yushchen
ambrosi
khowell
francesc
u_NA38NA50
gdebrecz
mariovor
turban
boothuh
bcabirol
saracino
pue
u_LHCB
makariev
bangeluc
manjitk
tnumao
pepe
kkaadze
schel3
muheim
nesterov
mufast
girrbach
harp_PROD
cavalari
stoynev
laszloa
diracgst
dinh
petrucci
cernyv
rauch
ivanch
popovb49
ferrif
lconey
panosk
u_GEAR
alanaro
u_SLDIV
fmarchet
burin
gennai
fonseca
martinjp
potreb
giordano
amssec
piandani
l3u001
hegner
prodvk
popovb
xbarr
gritsaj
temnikov
sissi
tsenov
kayis
geist
ggrazian
jrfry
bhallgre
blyth
gastaldi
sonia
bobbink
wadhwa
piccini
na48hac
sissi
venditti
ajokhov
melk
cfarnesejengelfr
oussenko
prodna49
rosier
wcreus
fdydak
sconstan
kolesnik
aantonel
psilva
na48u009
rbecker
martoiu
na48nntp
galaktio
vpalladi
gmasetti
rappazzo
perfetto
foc
roberta
gear025
madigo
l3mon
szuba
emanuele
dplasien
evcharp
wieneman
mryb
benitezj
vkruglov
franken
kirsanom
kobayash
aanania
l3u005l3u004
lbetev
u_uh
radu
l3u003
l3u002
jbook
u_C3
u_SFT
na48u010
danpop
bsunanda
jkrasnoh
meurer
mkoval
ikrol
vserdiou
dsoph
choutko
orestano
grappazz
gear013
celeghinsantin
behler
flionett
pastrone
pchrista
ddrijard
phy6phs
petuhov
schopper
slodkow
mutvic
abolshak
u_AMSP
ambrosin
u_NA45
cdelitzs
guaraldo
rozjkov
ylu
mchiba
rovere
sphicas
sstamm
damgov
drijard
roths
seyboth
jorgec yvette
mrizza
sbravar
kowal
goudzovs
lkrcalib
jasinjak
vtrubnik
wanke
aderoeck
nemenov
mdyulend
vetlitsk
cjingzalite
u_COMPASS
wdominik
gear009
gear008gear007
gear006gear005
gear004gear003
gear002gear001
marcello
bolotov
sdutta
portheau
afava
masaaki
buncic
lebedev
petrascu
xmeng
kekel
xyang
dgrandiaelin
wperyt
psoler2
biland
manjit
suenhou
yasar
peppep
gear037
prok
lecoultr
boconnel
plutaj
vite
ddesalva
hayato
mkubesov
ivanshin
rab
berschin
jfield
renfordt
enulescurezinko
vsemenov
brun
fmna49
mulders
aguskov
yfoka
yazkov
rolandi
jemtchou
u_ATLASCAT
u_NOMAD
editgas2 editgas1
shieh
kvi
bourilko
barrin
lucieg
diracdbkatchnov
dmitrifs
u_DIRAC
volnykh
candi
diracxfr
lgatigno
mboguszveszter
srobbins
ufo
choumilo
demina
evaboter
gear026
wahlppetev
ovg
ygouzevi
mnikitin
u_vp
cdbase1
marekg
ciocarln
beaudett
pochon
venelin
wenig
sbg
dinkelak
chmeurer
burgerchumakov
harpprod
cvetan
schacher
gmchen
alopeza ekulish
debeerjwotsch
calvetticogan
biino
tsusa
meunier
anton
grid_NA48
paganouh
okada
fechner
stefansk
komarov
molot
CMS_CERN01_YODA
edoardo
mduda
harpprod
mmoulson
stefanik
pzucchuh
romerov
tschuste
ltortoraburguet
bboimska
mposiada
stefanek
dde
widhalm
mihul
batgerel
nbuncic
fkaster
lamberto
ewk
demelyan
PES
CERN IT DepartmentCH-1211 Geneve 23
Switzerlandwww.cern.ch/it
CERNITDepartment
18 – Outlook
Section 4
Outlook
PES
CERN IT DepartmentCH-1211 Geneve 23
Switzerlandwww.cern.ch/it
CERNITDepartment
19 – Outlook
Outlook
Throughput through simplicity
Relaxed consistency acceptable for monitoring
Horizontal scalability
Flexibility
Many available open-source implementations
New schema design approach
Ñ NoSQL databases seem a good fit for monitoring
PES
CERN IT DepartmentCH-1211 Geneve 23
Switzerlandwww.cern.ch/it
CERNITDepartment
20 – Outlook
Thanks!
Questions?