redhat summit perf analysis and tuning part 1 2013

8/13/2019 Redhat Summit Perf Analysis and Tuning Part 1 2013

1/69


2/69

Performance Analysis and

Tuning Part 1Larry WoodmanSenior Consulting Engineering RHEL/VM

Bill GrayPrincipal Performance Engineer Red Hat

!une "# $%"#


3/69

Agenda: Performance Analysis Tuning Part II

Part I Red Hat Enterprise Linux tuned profiles! top

"enc#mar$ results

%cala"ilty &'% %c#eduler tuna"les ( &groups

Hugepages Transparent Hugepages! )*+(1,+ -on .niform *emory Access /-.*A0 and -.*A

Part II

-et2or$ Performance and Latency3performance

is$ and 'ilesystem I4 3 T#roug#put3performance

%ystem Performance(Tools perf! tuna! systemtap

5 6 A


4/69

Red Hat Enterprise Linux: %cale .p 6 4ut

%cale 4ut" node "%%%s nodes

%cale .p

" CP&

&p to'%() CP&s

*raditional scale+out capa,ilities -a.e ,een complemented o.ert-e past fi.e years it- scale+up capa,ilities

Brings open source .alue and fle0i,ility to 01)2)' Ser.er mar3et

Support for scala,le arc-itecturesMulti+core and -ypert-reading

4ernel 5&M6 and SMP en-ancements


5/69

Red Hat Enterprise Linux 7+enc#mar$ platform of c#oice

Survey of Benchmark Results 6/2013System Performance Evaluation Committee www.spec.org SPECcpu2006 SPECvirt_sc2010, sc2013 SPEC!!2013

"ransaction Processing Council #"PC$ www.tpc.org"PC%& 3 of top 6 categories

"PC%C "op virtuali'ation w( )*+S"C -S) worloa/s % www.stacresearc.comSP Sales an/ istri!ution %www.sap.com(campaigns(!encmar
http://www.stacresearch.com/http://www.stacresearch.com/


6/69

Red Hat Enterprise Linux 7+enc#mar$ platform of c#oice

cpu)887 9irtsc)818 ;Enterprise)818 9irtsc)81< ;"")818=

78=

?8=

188=

1)8=

)7

#17

%7 %7 %7

8(7

"%%7

%7 %7 %7

?8=

?7=

78=

@8=>@=

%PE& +enc#mar$ Pu"lications )811 3 )818= "%7 '%7 '%7

.m=dirty2,ac3ground2ratio

"%7 R6M ;7

.m=sappiness )% "% #%


10/69

Red Hat Enterprise Linux 7%c#eduler Tuna"les


11/69

/proc/sys/3ernel/sc-ed2

Red Hat Enterprise Linu0 ) *uned+adm ill increase Fuantum onpar it- Red Hat Enterprise Linu0 ;

ec-o "%%%%%%% /proc/sys/3ernel/sc-ed2min2granularity2ns

Minimal preemption granularity for CP& ,ound tas3s= See

sc-ed2latency2ns for details= *-e default .alue is '%%%%%%9ns:=

ec-o ";%%%%%% /proc/sys/3ernel/sc-ed2a3eup2granularity2ns

*-e a3e+up preemption granularity=


12/69

Sc-eduler tries to 3eep all CP&s ,usy ,y mo.ing tas3s form

o.erloaded CP&s to idle CP&s etect using Iperf statJ loo3 for e0cessi.e ImigrationsJ

/proc/sys/3ernel/sc-ed2migration2cost

6mount of time after t-e last e0ecution t-at a tas3 is considered

to ,e Icac-e -otJ in migration decisions= 6 I-otJ tas3 is less li3elyto ,e migrated so increasing t-is .aria,le reduces tas3migrations= *-e default .alue is ;%%%%% 9ns:=


13/69

%c#ed*igration &ost

exit18 exit188 exit1888 for$18 for$188 for$18888B88

@8B88

188B88

1@8B88

)88B88

)@8B88

8B88=

)8B88=

>8B88=

78B88=

?8B88=

188B88=

1)8B88=

1>8B88=

RHEL7B< Effect of sc#edmigration cost on for$(exit

ms

percent impro9ement

usec(call

Percent


14/69

for$/0 "e#a9ior

sc#edc#ildrunsfirst

&ontrols 2#et#er parent or c#ild runs first

efault is 8: parent continues "efore c#ildren runB

efault is different t#an RHEL@

Red Hat Enterprise Linux 7


15/69

Standard HugePages $MB

Reser.e/free .ia

/proc/sys/.m/nr2-ugepages

/sys/de.ices/node//-ugepages//nr-ugepages

&sed .ia -ugetl,fs

GB Hugepages "GB Reser.ed at ,oot time/no freeing

&sed .ia -ugetl,fs

*ransparent HugePages $MB

@n ,y default .ia ,oot args or /sys &sed for anonymous memory

Red Hat Enterprise Linux 7Hugepages( * Tuning

P-ysical Memory

Virtual 6ddress

Space

*LB

"$1 data"$1 instruction


16/69

)*+ standard Hugepages

# echo 2000 > /proc/sys/vm/nr_hugepages# cat /proc/meminfoMemTotal: 16331124 kB

MemFree: 11788608 kB

HugePages_Total: 2000HugePages_Free: 2000HugePages_Rsvd: 0HugePages_Surp: 0Hugepagesize: 2048 kB

# ./hugeshm 1000

# cat /proc/meminfoMemTotal: 16331124 kBMemFree: 11788608 kB

HugePages_Total: 2000

HugePages_Free: 1000HugePages_Rsvd: 1000HugePages_Surp: 0Hugepagesize: 2048 kB


17/69

K cat /proc/meminfo more

1,+ Hugepages

*oot arguments % /efault_ugepages'415, ugepages'415, ugepages4

HugePages_Total: 8HugePages_Free: 8HugePages_Rsvd: 0HugePages_Surp: 0

Kmount +t -ugetl,fs none /mnt

.(mmapwrite (mnt(un 33 writing 2097152 pages of random junk to file /mnt/junkwrote 8589934592 bytes to file /mnt/junk

HugePages_Total: 8HugePages_Free: 0HugePages_Rsvd: 0HugePages_Surp: 0

K cat /proc/meminfo more


18/69

Transparent Hugepages

ec-o ne.er /sys/3ernel/mm/transparent2-ugepagesne.er

[root@dhcp-100-1-!0 code"# t$me %/memory 1! 0real 0m12.434suser 0m0.936ssys 0m11.416s

# cat /proc/meminfoMemTotal: 16331124 kBAnonHugePages: 0 kB

Boot argument transparent2-ugepagesalays 9ena,led ,y default:

K ec-o alays /sys/3ernel/mm/red-at2transparent2-ugepage/ena,led

# t$me %/memory 1!&B real 0m7.024sreal 0m7.024s user 0m0.073s

sys 0m6.847s# cat /proc/meminfoMemTotal: 16331124 kBAnonHugePages: 15590528 kB

S'(()*' 12%+/,%0 1%,,. !6


19/69

Performance RHEL 7 ( %andy +ridge%pec;"" Fa9a 2( 1,+ #uge pages

Sandy Bridge -as "GB -ugepages

Support in RHEL;=1 and )=$

RHEL)= *ransparent Huge pages

7se 2+ 86_69 page vs 9 page

: ;&Eava etc

utomatically use uge pages

-or all anonymous memory aemon to gater free /ynamically

RHEL)=$ 9disa,le *HP:RHEL)=$ $M HugePage

RHEL)=$ "GB HugePage

%=%7

$=%7

'=%7

)=%7

1=%7

"%=%7

"$=%7

"'=%7

8B8=

GB1=

1)B7=

RHEL)=' SPECN,, / $M/"G -ugepages


20/69

*emory ones

&p to )' GB9P6E:

Hig-mem Oone

1() MB or #()1MB

5ormal Oone

")MB M6 Oone%

End of R6M

5ormal Oone

'GB

M6#$ Oone

")MB M6 Oone%

#$+,it )'+,it


21/69

Separate page+lists for anonymous and pagecac-e Pre.ents mi0ing of anonymous and file+,ac3ed pages

on acti.e and inacti.e LR& lists

Eliminates long pauses -en all CP&s enter direct

reclaim during memory e0-austion Pre.ents sapping -en copying .ery large files

Pre.ents sapping of data,ase cac-e during ,ac3up=

%plit LR. pagelists


22/69

Per -ode(one split LR. Paging ynamics

anonLR.

fileLR.

I-A&TIE

'REE

.ser Allocations

Reacti9ate

Page aging

s2apout

flus#

Reclaiming

.ser deletions

anonLR.

fileLR.

A&TIE


23/69

C#at is -.*A

5on &niform Memory 6ccess

6 result of ma3ing ,igger systems more scala,le ,ydistri,uting system memory near indi.idual CP&s====

6ll multi+soc3et 01)2)' ser.er systems are 5&M6

Most ser.ers -a.e " 5&M6 node / soc3et

Recent 6M systems -a.e $ 5&M6 nodes / soc3et

4eep interlea.e memory in B


24/69

Typical %ystem +uilding +loc$

Memory Controller and node R6M

P< lin3s


25/69

T2o -.*A node system

5ode % R6M

P< lin3s


26/69

'our -.*A node system!fully3connected topology

5ode % R6M

P< lin3s


27/69

'our -.*A node system!ring topology

5ode % R6M

P< lin3s


28/69

Per -.*A3-ode Resources

Memory Qones9M6 > 5ormal Qones:

CP&s


29/69

-.*A -odes and ones

End of R6M

5ormal Oone

5ormal Oone

'GB

M6#$ Oone

")MB M6 Oone%

)'+,it

5ode %

5ode "

Jone reclaim mode


30/69

Controls 5&M6 specific memory allocation policy W-en set and node memory is e0-austed

Reclaim memory from local node rat-er t-an allocatingfrom ne0t node

Sloer allocation -ig-er 5&M6 -it ratio W-en clear and node memory is e0-austed

6llocate from all nodes ,efore reclaiming memory

Aaster allocation -ig-er 5&M6 miss ratio

efault is set at ,oot time ,ased on 5&M6 factor

Jonereclaimmode


31/69

Learn a"out &P.s 9ia lscpu

# lscpuArchitecture: x86_64CPU op-mode(s): 32-bit, 64-bitBte !rder: "itt#e $%di&%CPU(s): 4'!%-#i%e CPU(s) #ist: '-3hre&d(s) per core: *

Core(s) per soc+et: *'CPU soc+et(s): 4UA %ode(s): 4. . . ."*d c&che: 32/"*i c&che: 32/"2 c&che: 206/

"3 c&che: 3'12'/UA %ode' CPU(s): ',4,8,*2,*6,2',24,28,32,36UA %ode* CPU(s): 2,6,*',*4,*8,22,26,3',34,38UA %ode2 CPU(s): *,0,,*3,*1,2*,20,2,33,31UA %ode3 CPU(s): 3,1,**,*0,*,23,21,3*,30,3

i li &P. i l t


32/69

isualiJe &P.s 9ia lstopo

/from #2loc pac$age0

# lstopo


33/69

Learn -.*A layout 9ia numactl

# numactl --hardware

&&i#&b#e: 4 %odes ('-3)%ode ' cpus: ' 4 8 *2 *6 2' 24 28 32 36%ode ' sie: 604*0 B%ode ' ree: 63482 B%ode * cpus: 2 6 *' *4 *8 22 26 3' 34 38%ode * sie: 60036 B%ode * ree: 6368 B

%ode 2 cpus: * 0 *3 *1 2* 20 2 33 31%ode 2 sie: 60036 B%ode 2 ree: 6381 B%ode 3 cpus: 3 1 ** *0 * 23 21 3* 30 3%ode 3 sie: 60036 B%ode 3 ree: 631* B%ode dist&%ces:

%ode ' * 2 3': *' 2* 2* 2**: 2* *' 2* 2*2: 2* 2* *' 2*3: 2* 2* 2* *'


34/69

%ample remote access latencies

' soc3et / ' node "=;0

' soc3et / 1 node $=80

1 soc3et / 1 node $=10

#$ node system ;=;0

9#%/#$ inter+node latencies '0: "% 9 #$/"%$' #="7: "# 9 #$/"%$' #="7: '% 9 )'/"%$' )=$7: '1 9''1/"%$' '#=17: ;; 9''1/"%$' '#=17:

Red Hat Enterprise Linux 7B>


35/69

*etal efault *etal -uma&TL D* efault D* -uma&TL8

)88888

>88888

788888

?88888

1888888

1)88888

1>88888

RHEL7B> %PE&;"" 4FD numactl

inst 2#7 2#?8

@88888

1888888

1@88888

)888888

)@88888

%

"%

$%

#%

'%

;%

)%

8%

1%

Eig#t D* ,uests 2it# Fa9a Load

numad it- RHEL)=' -ost and )=# guests

default

numadnumatune

=,ain

Care#ouses

+4P%

*ultiguest 4racle 4LTP Cor$load


60/69

)8. >8. ?8.8

)88888

>88888

788888

?88888

1888888

1)88888

1>88888

%

;

"%

";

$%

$;

#%

#;

4racle 4LTP in TP* > D* ,uests


61/69

S-ipping in Red Hat Enterprise Linu0 )=' Potential future impro.ements e.ice and


62/69

Red Hat Enterprise Linu0 ) Performance Aeatures

I*&5EJ tool ? adNusts system parameters to matc-en.ironments + t-roug-put/latency=

*ransparent Huge Pages ? auto select large pages foranonymous memory static -ugepages for s-ared mem

5on+uniform Memory 6ccess 95&M6:

numastat en-ancements

numactl for manual control

numad daemon for auto placement *&56 ? integration / Red Hat Enterprise Linu0 )='

9===Come ,ac3 for part $===:

cgroups Arc#itecture


63/69

&group default mount points


64/69

K cat /etc/cgconfig=conf

mount ]cpuset /cgroup/cpusetZcpu /cgroup/cpuZcpuacct /cgroup/cpuacctZ

memory /cgroup/memoryZde.ices /cgroup/de.icesZfreeQer /cgroup/freeQerZnet2cls /cgroup/net2clsZ,l3io /cgroup/,l3ioZ

^

K ls +l /cgroup

dr0r+0r+0 $ root root % !un $" "### ,l3iodr0r+0r+0 # root root % !un $" "### cpudr0r+0r+0 # root root % !un $" "### cpuacctdr0r+0r+0 # root root % !un $" "### cpuset

dr0r+0r+0 # root root % !un $" "### de.icesdr0r+0r+0 # root root % !un $" "### freeQerdr0r+0r+0 # root root % !un $" "### memorydr0r+0r+0 $ root root % !un $" "### net2cls

&group #o23to


65/69

"GB/$CP& su,set of a ")GB/1CP& system

Knumactl ++-ardare

Kmount +t cgroup 000 /cgroups

Km3dir +p /cgroups/test

Kcd /cgroups/test

Kec-o " cpuset=mems

Kec-o $+# cpuset=cpus

Kec-o "G memory=limit2in2,ytesKec-o YY tas3s

[root@dhcp-100-19-50 ~]# forkmany 20MB 100procs &

cgroups


66/69

[root@dhcp-100-19-50 ~]# top -d 5

top - 12:24:13 up 1:36, 4 users, load average: 22.70, 5.32, 1.79

Tasks: 315 total, 93 running, 222 sleeping, 0 stopped, 0 zombie

Cpu0 : 0.0%us, 0.2%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st


Cpu2 :100.0%us, 0.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st




Cpu6 : 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st


Mem: 16469476k total, 1993064k used, 14476412k free, 33740k buffers

Swap: 2031608k total, 185404k used, 1846204k free, 459644k cached

erify correct "indings# 8


67/69

ec#o 8 M cpusetBmems ec#o 83< M cpusetBcpus numastat node% node"numa2-it 17>?) '#1881numa2miss $#';( $"#';$%local2node ")'1)'1 '$#")$ot-er2node $#;1# $";%"#)

K /common/loodman/code/memory 'faulting too3 "=)")%)$stouc-ing too3 %=#)'(#8s

K numastat node% node"numa2-it )88>)


68/69

ec#o 1 M cpusetBmems ec#o 83< M cpusetBcpusK numastat node% node"numa2-it ")$##"1 '#'"%)numa2miss $#';( 18?)>@?local2node ")$#"(' '"1'(%ot-er2node $#;1# "%(1%8'

K /common/loodman/code/memory 'faulting too$ 1BG77)stouc#ing too$ 8B>@>


69/69

6A %;/$1/"#Principled *ec-nologies Red Hat

redhat summit perf analysis and tuning part 1 2013

Documents