rodrigo freire - rhel 6 performance & tuning

8/16/2019 Rodrigo Freire - Rhel 6 Performance & Tuning

1/37

RHEL 6 PERFORMANCE & TUNING

Rodrigo FreireSr. Technical Account Manager

22/Mai/2014


2/37

Jeremy Eder / Rodrigo Freire2

Agenda

Performance Tuning Theory

RH!" Performance #m$ro%ement&

'P( Performance Tuning/Po)er Management

Memory/*(MA Performance Tuning

*et)or+ Performance Tuning


3/37

Jeremy Eder / Rodrigo Freire,

Performane T!ning T"eory


4/37


Performane T!ning Food Gro!#$

'P( Memory

#/- *et)or+


5/37

Jeremy Eder / Rodrigo Freire

Performane or Re%iai%i'y(

●

Fa&ter tran&action&or

● fficiency

● Tradeoff&

● Ri&+&

● 'o&t


6/37

Jeremy Eder / Rodrigo Freire"

)a$i O* *e'!#

●

i&ale unnece&&ary &er%ice& and u&e runle%el ,● A%oid di&+ acce&& in the critical $ath

● 'on&ider di&aling file&y&tem ournaling3 a3dir5time

● %er con&ider running &)a$le&& 6%m.&)a$$ine&&7

● 'P( #&olation

● 8e a)are of 8#-S ma+ing Po)er Managementdeci&ion&


7/37Jeremy Eder / Rodrigo Freire9

Effiieny dei$ion

● 8and)idth: Ma;imum$o&&ile through$ut

● Through$ut: 'urrentand)idth u&age

● High efficiency a$$roach

!arger and le&& $ac+et&

● !o) latency a$$roach:Send data immediately

6more and &maller $+t&7


8/37Jeremy Eder / Rodrigo Freire<

More and+id'"(

●

'utting edge co&t&● *on=linear >/$erformance

● 8enefit&


9/37

Jeremy Eder / Rodrigo Freire?

● TST3 MAS(R3 TST3 MAS(R3 TST...● 8e $atient3 accurate and methodical...it@& iterati%e

● no) your hard)areBBBBBBBBB

● Thi& cannot e &tre&&ed enough...● 8e a)are of the latency %& through$ut alancing act

● The enemy of e;treme lo)=latency:atching/coale&cing

● A%oid di&+ )hen you can...

● (&e tool& &uch a& &y&temta$ and $erf

In a n!'$"e%%,,,


10/37


RHEL6 PerformaneIm#ro-emen'$


11/37


Performane Im#ro-emen'$ in RHEL6

Com#onen' Fea'!re

'P(/ernel *(MA C Tic+eted &$inloc+&D 'om$letely Fair SchedulerD

;ten&i%e u&e of Read 'o$y ($date 6R'(7Scale& u$ to "4 E'P(& $er gue&t

Memory !arge memory o$timiation&: Tran&$arent Huge Page& i&ideal for %irtualiation

*et)or+ing %ho&t=net C a +ernel a&ed %irtio )/ etter through$ut and

latency. SR#-E for Gnati%e $erformance3 RFS/PS8loc+ A#-3 MS#3 &catter gather.


12/37


'!ned #rofi%e $!mmary,,,

Tunale default enter$ri&e=&torage

%irtual=ho&t

%irtual=gue&t

%a'eny.#erformane

through$ut=$erformance

+ernel.&chedIminI granularityIn&

4m& 10m& 10m& 10m& 10m&

+ernel.&chedI)a+eu$ IgranularityIn&

4m& 1m& 1m& 1m& 1m&

%m.dirtyIratio 20J RAM 40J 10J 40J 40J

%m.dirtyIac+ground Iratio

10J RAM J

%m.&)a$$ine&& "0 10 ,0

#/- Scheduler6le%ator7

'FK deadline deadline deadline dead%ine deadline

File&y&tem 8arrier& -n -ff -ff -ff

'P( Lo%ernor ondemand $erformance #erformane $erformancei&+ Read=ahead 4;

i$a%e THP 0e$

i$a%e C.*'a'e$ 0e$


13/37

Jeremy Eder / Rodrigo Freire1,

)%o1 e-ie$


14/37


A-ai%a%e Te"no%ogie$

● Solid=State e%ice

● S$inning H


15/37


2ir'!a%i3a'ion T!ning I/O e%e-a'or$ . OLTP

1Lue&t 2 Lue&t& 4 Lue&t&

0

100

10

200

20

,00

Performance #m$act of #/- le%ator& on -!TP or+load

Ho&t running eadline Scheduler

*oo$

'FK

eadline

T r a n & a c t i o n & $ e r M i n u

t e


16/37

Jeremy Eder / Rodrigo Freire1"

)!' if yo! !$e en'ra% *'orage,,,

●

(&e dead%ine ele%ator

● Storage ha%e it& o)no$timiation algorithm

and cache&

● EM image i& on the

&torage


17/37


CPU Performane T!ning


18/37

Jeremy Eder / Rodrigo Freire1<

A +ord ao!' CPU Po+er Mgm',,,C&P.$'a'e$

NOou probably don@t always need )hat you $aid for...

– Recent chi$& from maor %endor& &lo) them&el%e&do)n

• 'alled P=&tate&

– -r lo)er %oltage&/di&ale $ortion& of the core li+etimer&

• 'alled '=&tate&

– And &$in them ac+ u$ on=demand.

• Add&%a'eny

Monitoring:

– (&e $o)erto$3 or turo&tat from +ernel &ource

A +ord ao!' CPU Po+er Mgm',,,C&P.$'a'e$


19/37

Jeremy Eder / Rodrigo Freire1?

CPU T!ning

Eariale freQuencie& – Multi$le core&

– Po)er &a%ing mode& 6c$u&$eed go%ernor&7

• $erformance

•

ondemand• u&er&$ace

;am$le&:

– echo "performance" > \/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

– 8e&t of oth )orld& C cron o& to configure the go%ernor mode u&ingtuned=adm

– tuned-adm profile {default,latency-performance}


20/37


*"ed!%er Po%iie$

● TS S'HI*-RMA! 6a+a S'HI-THR3 the default $olicy7

● FF S'HIF#F- 6realtime $olicy3 fir&t=in=fir&t=out7

– on@t &et your RTPR#- to ??. Thi& )ill &tar%e out +ernel thread&

that need to run &ometime&.

– There i& no )ay to fully i&olate a core for 100J u&er&$ace

$roce&&ing. Recent &tudy in a $re%iou& &lide...

RR S'HIRR3 &ame a& F#F- ut )ith a defined Quantum

– S'HIRR only u&eful )ith 1 ta&+& of &ame $riority.

S'HI8AT'H3 #S- S'HI#S-3 #! S'HI#!

'hange $rogrammatically3 or )ith chrt

# ps -emo pid,pcpu,psr,nice,cmd,rtprio,policy


21/37


0o!r "ard+are mig"' e foo%ing yo!4

● SM#&

●

8#-S $o)er go%ernor&

● 8ro+en 8#-SB


22/37


Memory & NUMA


23/37

Jeremy Eder / Rodrigo Freire2,

Tran$#aren' H!ge Page$

●

Standard $age&: 4 +8● Huge $age: 204< +

● 12 time& largerB

● !e&& !1 T!8

con&um$tion● Memory inten&i%e: #*B

[root@rfreire ~]# (disable THP)

[root@rfreire ~]# time memhog 1g

real 0m0.600s

user 0m0.186s

sys 0m0.412s

[root@rfreire ~]# (enable THP)

[root@rfreire ~]# time memhog 1g

real 0m0.303s

user 0m0.199s

sys 0m0.100s


24/37


Memory T!ning 5 Tran$#aren' H!ge#age$

#ntroduced in RH!".0 – Anonymou& memory only 6&)a$$ale3 can e di&aled7

– 'an coe;i&t )ith traditional huge$age&

– oe& not reQuire a$$lication &u$$ort 6anon memory7.

–

#n RH!".23 added counter&...e;$lained in transhuget!t egrep #trans$thp# /proc/vmstat

nr_anon_transparent_hugepages %0&'

thp_fault_alloc ()0%

thp_fault_fall*ac+ 0

thp_collapse_alloc 0&

thp_collapse_alloc_failed 0

thp_split %&


25/37


NUMA . "a' i$ i'(

●

*on=(niform Memory Acce&&● 'P(=8ound

● 'entral memory controller

●

Acce&& 'o&t


26/37

Jeremy Eder / Rodrigo Freire2"

Multi=&oc+et/Multi=core Architecture u&ed for &caling

–

RH!/" 'om$letely *(MA A)are – Additional3 &ignificant $erformance gain& y

enforcing *(MA locality.

Ho) do you enforce *(MA locality –

numactl -c& -m& /command• 'ommand e;ecute& on 'P(& in &oc+et 1• And memory allocation& are &er%ed out of memory

node 1.

– NUMA automation is an area of significantresearch and investment by both Red Hat and thecommunity.• AutoNUMA, schedNUMA, numad

NUMA


27/37


NUMA


28/37

Jeremy Eder / Rodrigo Freire2<

NUMA To#o%ogy and PCI )!$ Ser%er may ha%e more than 1 P'# u&.

–

-$timal $erformance reduce&/eliminate& inter=node cro&&=tal+. #n&tall*#' in &lot local to node that your a$$lication )ill run on. (&e&y&temta$ numaIfault&.&t$. irQalance )ill learn thi& N&oon.

– #n the elo) ca&e3 the *#' i& in P'# u& 0001. 'P(& 13,339 are localto that P'# &lot.

lspci output:

0001:06:00.0 Ethernet controller: Solarflare Communications SFC900 !Solarstorm"

# cat sysde$icespci0000%:000000%:00%:00.0local&cpulist

0,,',6

# cat sysde$icespci0001%:'00001%:'0%:00.0local&cpulist

1,(,),*

# dmes++rep / node

pci&2us 0000:00: on / node 0 3p4m 05

pci&2us 0001:'0: on / node 1 3p4m 15


29/37

Jeremy Eder / Rodrigo Freire2?

NUMA To#o%ogy and PCI )!$


30/37

Jeremy Eder / Rodrigo Freire,0

Ne'+or1


31/37


Ne'+or1 e'ermini$m

o you really need to u&e T'P

–

#f &o3 e;$eriment )ith T'PI*-!AO &oc+et o$tion 6*agle7 – From i+i$edia:

if there is ne data to send

if the indo si7e 8 SS and a$aila2le data is 8 SS

send complete SS se+ment no

else

if there is unconfirmed data still in the pipe

enueue data in the 2uffer until an ac;noled+e is recei$ed


32/37


)!ffer )%oa'

8uffer& are e%ery)here... ))).ufferloat.net

hat i& uffer loat

– !atency cau&ed y e;ce&&i%e uffering

– Side=effect if ignoring latency in the race for

greater through$ut.● htt$://))).ufferloat.net/$roect&/loat/)i+i/#ntrod

uction

– Find out aout your uffer&.•

(&e @&& =e@ or @net&tat =nt@• *#' ring uffer& and ne) 8yte Kueue !imit&

● htt$://linu;$lumer&conf.org/2011/oc)/&e&&i

on&/191● htt$&://l)n.net/Article&/44,?0/

http://www.bufferbloat.net/http://www.bufferbloat.net/projects/bloat/wiki/Introductionhttp://www.bufferbloat.net/projects/bloat/wiki/Introductionhttp://linuxplumbersconf.org/2011/ocw/sessions/171http://linuxplumbersconf.org/2011/ocw/sessions/171http://linuxplumbersconf.org/2011/ocw/sessions/171http://linuxplumbersconf.org/2011/ocw/sessions/171http://www.bufferbloat.net/projects/bloat/wiki/Introductionhttp://www.bufferbloat.net/projects/bloat/wiki/Introductionhttp://www.bufferbloat.net/


33/37

Jeremy Eder / Rodrigo Freire,,

M!%'i7!e!e Ne'+or1ing 8a1a R**9

●

#n%ented to allo)!inu; net)or+ing to&cale along )ithhard)are

●

2 &oc+et/< core&e;tremely common3o$timie for thi&u&e=ca&e

●

Ha&h of &rc/d&t#P:P-RT determine&recei%ing 'P(


34/37


)!' '"e O* $"o!%d "and%e a%% of '"i$,,,

Performance Lrou$ couldn@t agree moreB

– -ut of the o; $erformance e;$erience highe&t $riority.

• efault& )or+ for the maority of u&e=ca&e&

• Auto=tuning )here it doe&n@t

• Hand tuning a& a la&t re&ort

– irQalance eing taught aout P'# u& locality6localIc$uli&t7

• ernel already +no)&3 RH!"., )ill &et it for you.

– numad can automatically alance *(MA nodeutiliation to a%oid *(MA fault&.


35/37

Jeremy Eder / Rodrigo Freire,

*ome !$ef!% re$o!re$● Perftun guide

htt$&://acce&&.redhat.com/&ite/documentation/en=(S/RedIHatInter$ri&eI!inu;/"/html=&ingle/PerformanceITuningILuide

/inde;.html

● Sy&temta$:

htt$&://acce&&.redhat.com/&ite/&olution&/441

htt$&://acce&&.redhat.com/&ite/documentation/en=(S/RedIHatInter$ri&eI!inu;/"/html/Sy&temTa$I8eginner&ILuide/inde

;.html

● 8enchmar+ing tool&:

htt$&://acce&&.redhat.com/&ite/&olution&/19,


36/37

Jeremy Eder / Rodrigo Freire,"


37/37

THAN: 0OU4

Rodrigo Freire

rfreireUredhat.com

htt$://$eo$le.redhat.com/rfreire/cce=$erftun=&.$df
mailto:[email protected]:[email protected]

rodrigo freire - rhel 6 performance & tuning

Documents