optimizing and improving opensimulator performance

8/9/2019 Optimizing and Improving Opensimulator Performance

http://slidepdf.com/reader/full/optimizing-and-improving-opensimulator-performance 1/32

Optimizing and Improving

OpenSimulator PerformanceJustin Clark-Casey

OpenSimulator core developer



In this presentation

● The System Architecture

– Components and interactions

– Potential ottlenecks

– !ules of thum for good performance

● "easuring and Testing

– #hat the heck is going on$

●

The Conference – %&& connections for the keynote



#hy listen to me$

● OpenSimulator core developer and consultant for '( years)

● *ave carried out grid performance analysis for commercialand education clients)

● *ave made many performance improvements+ especiallyfor conference this year and last)

– ,ghosts ranch to soon e integrated)

● .ut even I/m not an e0pert on everything

–

Performance is a very comple0 topic in this architecture) – "any different net1ork 2 server arrangements)



The System Architecture

● Talking aout a *ypergrid-enaled grid installation)

– Though much also applies to standalone)

– "uch also applies to a non-*ypergrid installation)

● Assume you are already familiar 1ith fundamentalsystem concepts

– Standalone+ grid+ *ypergrid)

– Assets+ inventory)

– *TTP+ TCP+ 34P)



5rid Components

Simulator6ie1er

5rid Services

Pulic

Private

*TTP

34P

*TTP Simulator

*TTP

*TTP

*TTP



*ypergrid Components

6ie1er

*ome Services 7oreign Services

*ome Simulator

7oreign Simulator

UDP

UDP



.ottlenecks

● In general four aspects

– 8et1ork

– Processing 9CP3:5P3 on vie1er side;

–

Storage – Soft1are

● for three components

– Simulator

– 5rid Services

– 6ie1er:Client



8et1ork .ottlenecks

● 6ie1er < Simulator+ Services

– 34P= need lo1 latency

– *TTP= need high and1idth

–

!ule of thum= try at least for >&& kit per connection – pingtest)net

● Simulator < Simulator same grid

– *TTP= ?o1 latency

● Install < Install via *ypergrid

– *TTP= ?o1 latency+ good and1idth



Processing .ottlenecks

● 6ie1er

– "ore prims+ more avatars+ higher load)

● 5P3:CP3 limiting factor

●

Simulator – "ore prims+ more avatars+ more scripts+ higher load)

● CP3 limiting factor in the first instance

– !ecommended minimum @ cores per simulator

– !ule of thum= core per active region

● One region vs multiple regions per simulator)

● Soft1are inefficiency 9more on that later;)



Storage .ottlenecks

● Services

– Asset storage dominates everything)

● All assets immutale so deletion e0tremely tricky)

– 4eduplicating asset service in future release● Or use S!AS today BS!AS)

– 7aster storage is etter ut proaly not critical)

●

Simulator – Allo1 as many assets to e cached as possile)



Soft1are .ottlenecks

● 6ie1er

– I kno1 little aout this)

● Services

– "aye inefficient CD #eserver though deated)

● Simulator

– 7ree threads for allE approach inefficient)

● Improvements for conference)

– "ono vs #indo1s )8FT)

–

Offload 5etTe0ture:5et"esh to direct service calls B4I!FCT SF!6ICF – *andling ad foreign installations in the *ypergrid)



"easuring and Testing

● Fvery install can e different)

● 6ery large numer of moving parts

– "ore operating system than app:1eserver)

● Identifying and reproducing an issue

– Is critical to actually fi0 it)

– Often much G>&H of the 1ork)



"easuring

● 5enerations of stat systems)))

● 4ominant is no1 ,sho1 stats all)

– Console commands on simulator and services

– !ecord ra1 numers and moving averages

– 7acility to periodically record to file)



"easuring

Simulator 9root;D sho1 stats all

clientstack)keynote)Client?ogouts4ueTo8o!eceives = &

clientstack)keynote)Ino0PacketsCount = &+ &:s+ &:s

clientstack)keynote)IncomingPacketAsync!euests#aiting = &

entitytransfer)keynote)Inter!egionTeleportAttempts = &

entitytransfer)keynote)Inter!egionTeleportCancels = &entitytransfer)keynote)Inter!egionTeleport7ailures = &

httpserver)>&&&)*TTP!euestsServed = & reuests+ & reuests:s+ & reuests:s

scene)keynote)!ootAgents = & avatars

scene)keynote)Script?ines = & lines:sec

server)memory)Process"emory = %)>K ".server)net1ork)*TTP!euests"ade = @@ reuests+ & reuests:s+ & reuests:s

server)processor)CP3Percent = & percent

"ore information at BStats)



CP3 Sample 5raph



Agent Time 5raph



STP #ork Items #aiting



5raphing

● In my o1n tools repository) See B5!AP*I85reference)

– 8e1 this year)

●

#orks on ,stats record start log files) – 7ormat may change in the future)

● Primitive graphs

– 8o outlier filtering+ time series alignment+ yet+ etc)

– .ut can still e useful in diagnosing issues after the fact)



Testing

● 5eneral net1ork

– iperf L Tool for performing TCP:34P throughput andlatency tests Biperf

● Services – 5et te0ture test code at BTFMT3!F ?OA4 TFST)

– Other1ise not much yet)

–

"ay e ale to use general test tools such as SiegeBSIF5F)



Testing

● Simulator

– pCampot BPCA"P.OT

– Program to manage connections of many ots from

a single console)● 4ifferent ehaviours 9teleport+ 1alk around+ etc);

– This is a maNor tool I use in conference performancetesting)

– #ill continue to make ehaviour more like realvie1ers)



The Conference

● 5etting %&& connections in the same space is ahuge challenge)

– ?oad gro1s e0ponentially not linearly)

– 6ery rarely een done up to no1)



Considerations

● Can other parts of the grid take it 9e)g) services;$

● 4o 1e have the and1idth$

– 5etting people to login early and hence cache assets helps)

●

4o 1e have the CP3$ – @% core system for @ regions

● eah+ 1e/re pushing it)

● 6arregion vs) % neighouring regions

– 7ault tolerance

– Code maturity



The Process

● !egular Tuesday load tests supplemented ypCampot)

– 7irst locally then Amazon FC@)

●

Stats recording and analysis)● !eproduce prolems seen in live tests 1ith

pCampot)

● Changes to OpenSimulator code or config changes)

● !epeat until e0hausted)



Prolems that Fmerged

● 8eeded to fi0 issues in:improve pCampot)

● 8eeded to create graphing and analysis code)

● So 1hat do you need$ .esides a miracle)

– Stats+ lots of stats)

●

!educe 34P traffic 1here possile)● OpenSimulator trying to do too many things at once)

– Super thread happy)

– ?ots of conte0t s1itching)

– *olds up other 1ork)

● Physics

– 8ot a maNor issue in the keynote scenario)



Configuration t1eaks

● BInterest"anagement in OpenSim)ini

– !oot!otation3pdateTolerance Q &)>>

● !educe 34P updates on avatar rotation)

– Child!eprioritization4istance Q KKKKK● !educe thread 1ork for neighour sim oservation)

– ChildTerse3pdatePeriod Q %

● !educe 34P traffic from child connections)



Configuration t1eaks

● BClientStack)?inden34P

– enaleRadaptiveRthrottles Q false

● Oserved ad ehaviour under very high load 1ithadaptive)

– sceneRthrottleRma0Rps Q >&&&&&&&

● %&& mits:s

– clientRthrottleRma0Rps Q @>&&&

●

mit:s



Soft1are changes

● Pure performance

– Process M"? as stream not document)

● Perceived performance

– Skip ??Client6ie1 ueue for o1n mvmt packets)● ?ess thread happy

– Seuential processing)

● Appearance:Attachments

● Some incoming:outgoing 34P handling)

● Comple0 L need to avoid one task locking another)



7uture 4irections

● Can proaly make efficiency gains y further controllingconcurrency)

– Simulator-1ide No scheduler$

● 4istriuted Scene 5raph)

– Simple component to maintain connections even if otherprocesses crash)

● Allo1 risk of running more regions per simulator)

– .ut improving single system efficiency easier)

●

6arregions – *igh concurrency challenges over a larger area)



ThankyouE

Justin Clark-Casey

OS6# Consulting

http=::Nustincc)orghttp=::t1itter)com:Nustincc

NustinccNustincc)org

9please see ne0t slides for references;



!eferences

B4irect Service

http=::opensimulator)org:1iki:4irectRServiceR!euests

B5raphing

https=::githu)com:Nustincc:opensimulator-tools:tree:master:analysis:opensimulator-stats-analyzer

Biperf

http=::soft1are)es)net:iperf:

BPCampot

http=::opensimulator)org:1iki:PCamp.ot

mailto:[email protected]

mailto:[email protected]

http://twitter.com/justincc



!eferences

BPerformance

http=::opensimulator)org:1iki:Performance

BSiege

http=::111)Noedog)org:siege-home:

BS!AS

https=::githu)com:coyled:sras

BStats

http=::opensimulator)org:1iki:Sho1Rstats

BTe0ture ?oad Test

https=::githu)com:Nustincc:opensimulator-tools:tree:master:test:te0ture-load-test

https://github.com/justincc/opensimulator-tools/tree/master/analysis/opensimulator-stats-analyzer

http://software.es.net/iperf/

http://software.es.net/iperf/

https://github.com/justincc/opensimulator-tools/tree/master/analysis/opensimulator-stats-analyzer



2 A

https://github.com/coyled/sras

http://www.joedog.org/siege-home/

http://opensimulator.org/wiki/Performance

optimizing and improving opensimulator performance

Documents