sonic-7: tuning and scalability for sonic enterprise messaging analyzing, testing and tuning esb/jms...
TRANSCRIPT
SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Analyzing, testing and tuning ESB/JMS performance
David HentchelPrincipal Solution Engineer
© 2007 Progress Software Corporation2 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Agenda
Methodology: Review the recommended approach to project
and proceduresAnalysis: Understand how to characterize performance
requirements and platform capacityTesting: Learn how to simulate performance scenarios using
the Sonic Test HarnessTuning: Know the top ten tuning techniques for the Sonic
Enterprise Messaging backbone
Analyzing, testing and tuning ESB/JMS performance
© 2007 Progress Software Corporation3 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
D I S C L A I M E R
Setting Performance Expectations
System performance is highly dependent on machine, application and product version. Performance levels described here may or may not be achievable in a specific deployment.
Tuning techniques often interact with specific features, operating environment characteristics and load factors. You should test every option carefully and make your own decision regarding the relative advantages of each approach.
D I S C L A I M E R
© 2007 Progress Software Corporation4 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Agenda
Methodology:
Review the recommended approach to project and procedures
Analysis: Understand how to characterize performance requirements and
platform capacityTesting: Learn how to simulate performance scenarios using the Sonic
Test HarnessTuning: Know the top ten tuning techniques for the Sonic Enterprise
Messaging backbone
Analyzing, testing and tuning ESB/JMS performance
© 2007 Progress Software Corporation5 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Performance Concepts and Methodology
Terms and definitions Performance engineering concepts Managing a performance analysis project
• Skills needed for the project
• Performance Tools
• Project timeline
© 2007 Progress Software Corporation6 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Performance Engineering Terms
“System Under Test”
“Test Harness”
“Load” = “Sessions” * “Delivery Rate”
“Latency” = ReceiveTime – SendTime“Test Components”
“ExternalComponents”
“Platform” “System Metric”
R
V
“Variable” =client param,app param,system param
V
V
Expert Tip: Limit scope to those test components thatare critical to performance and under your control
© 2007 Progress Software Corporation7 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Concepts: Partitioning Resource Usage
% C
PU
Partitionable resources can be broken down as the sum of the contributions of each test component on the system
• Total resource usage is limited by system capacity• Becomes the bottleneck as it nears 100%• Goal is linear scalability as additional resource is added• Vertical versus Horizontal scalability
• Total latency is the sum across all resource latencies, i.e.:Latency = CPU_time + Disk_time + Socket_time + wait_sleep
X
Free
Z
Y
( Overhead X Message rate ) = Load∑(Writes/sec)(msg/sec)(writes/msg)svcs
Bottom-Up Rule: Test and tune each component before youtest and tune the aggregate.
© 2007 Progress Software Corporation8 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Concepts: Computer Platform Resources
Favorite tools: task mgr, perfmon, top, ping –s, traceroute
Disk I/O(read/write)
Network I/O(send/receive)
CPU time
Memory(in use, swap)
# Threads
• Use level of detail appropriate to the question being asked• Machine resource (such as %CPU) artifacts:
• side effects from uncontrolled applications• timing of refresh interval• correlation with test intervals
• Scaling across different platforms and resource types
© 2007 Progress Software Corporation9 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
The Performance Engineering Project
The Project is Goal Driven
Analyze
Test
Tune
For each iteration1. Test performance vs goal2. Identify likeliest area for gain
Startup tasks1. Define project completion goals2. Staff benchmarking skills3. Acquire test harness
Expert Tip: Schedule daily meetings to share results andreprioritize test plan.
© 2007 Progress Software Corporation10 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Performance Project Skills
Requirements Expert• SLA/QoS levels – minimal & optimal• Predicted load – current & future• Distribution topology
Integration Expert• Allowable design options• Cost to deploy• Cost to maintain
Testing Expert• Load generation tool• Bottleneck diagnosis• Tuning and optimization
SOLUTION
I.E.T.E.
R.E.
Cos
t/Ben
efit
Load/Distribution
Design Options
© 2007 Progress Software Corporation11 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Tools for a Messaging Benchmark
Configurator – creates conditions to bring system under test into known state
Harness – the platforms and components whose performance response is being measured
Analyzer – tools and procedures to make meaningful conclusions based on result data.
Platform
Component
System Under Test
TestHarness
TestConfigurator
TestAnalyzer
© 2007 Progress Software Corporation12 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Performance Project Timeline
1 2 3 4 5 6 7 8
DevelopmentProject
Week
Process Dev
Service Dev Sizing
System Test
Deployment Plan
Launch
PerformanceProjectPerf Prototype
Design Application Tuning
© 2007 Progress Software Corporation13 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Agenda
Methodology: Review the recommended approach to project and
procedures
Analysis:
Understand how to characterize performance requirements and platform capacity
Testing: Learn how to simulate performance scenarios using the Sonic
Test HarnessTuning: Know the top ten tuning techniques for the Sonic Enterprise
Messaging backbone
Analyzing, testing and tuning ESB/JMS performance
© 2007 Progress Software Corporation14 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Performance Analysis
Performance scenarios: requirements and goals• Some generic performance scenarios
System characterization: platforms and architecture
Test cases: specification for benchmark
Utilization(% total)Capacity
(units/sec)
Efficiency(units/msg)
X Load(msg/sec)
=
© 2007 Progress Software Corporation15 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Performance Scenario Specification
First, triage performance-sensitive processes• substantial messaging load and latency requirements
• impact of resource-intensive services
Document only process messaging & services • leave out Application specific logic – this is a prototype
Set specific messaging performance goals• Message rate and size
• Minimum and average latency required
Try to quantify actual value and risk• Why this use case matters
Rule of Thumb: Focus on broker loads over 10 msg/sec or 1 MByte/sec,and service loads over 10,000 per hour.
© 2007 Progress Software Corporation16 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Generic Scenario: Decoupled process
Asynchronous, loosely coupled, distributed services.Assumptions: Services allow concurrent, parallel distribution Messaging is lightweight, pub sub End-to-end process completes in real time May be part of a Batch To Real Time patternFactors to analyze: Speed and scalability of invoked services Distributed topology Quality of Service Aggregate Latency over time across batched invocations
Expert Tip: The easiest way to manage decoupled SOA processes isthe ESB Itinerary.
© 2007 Progress Software Corporation17 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Generic Scenario: Real time data feed
High speed distribution of real time eventsAssumptions: Read-only data pushed dynamically to users Messages are small Service mediation is simple and fast Latency is very important, but QoS needs are modestFactors to analyze: Quality of Service, esp. worst case for outage and
message loss Message rate and fanout (pub sub) Scalability of consumers
© 2007 Progress Software Corporation18 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Generic Scenario: Simple request reply
Typical web service call that waits for response
Assumptions: client is blocked, pending response small request message, response is larger latency is critical
Factors to analyze: Latency of each component service Load balancing of key services Recovery strategy if loop is interrupted Client network, protocol and security specs
© 2007 Progress Software Corporation19 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Example: Performance scenario specification
Overall project scope:• Project goals and process• Deployment environment• System architecture
For each Scenario:• Description of business process• Operational constraints (QoS, recovery, availability)• Performance goals, including business benefits
validate CreditCard Event
make a paymentEvent
Midas BillingSystem
Make Payment with Credit Card
make_payment (Progress Adapter)
CC Validate
Payementech(Mocked Object)
Make_payement API
Client Apps
is Billing System Up? Queue
yes
no
Credit Card SyntaxValidation
1
2
3
4
5
7-n
6
8
9
10
XML validation
7
© 2007 Progress Software Corporation20 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Characterizing platforms and architecture
Scope current and future hardware options available to the project
Identify geographical locations, firewalls and predefined service hosting restrictions
Define predefined Endpoints and Services Define data models and identify required
conversions and transformations.
© 2007 Progress Software Corporation21 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
DMZField DMZ
Platform configuration specification
CPUnumbertypespeed
Networkbandwidthlatencyspeed
Disktypespeed
Firewallcryptoslatency
Memorysizespeed
Rule of Thumb:LAN 15 – 150 MBytes / secondDisk: 2 – 10 MBytes / secondXSLT: 200 – 300 KBytes / second
© 2007 Progress Software Corporation22 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Network
Memory
Disk
CPU
Platform Profile: Real-time messaging
% capacity
System resource limitations
90%
70%
20%
5%
Rule of Thumb: Real-time, 1 KB messages, broker performance is about 1,000 to 10,000 msg/sec for each 1 gHz cpu power.
© 2007 Progress Software Corporation23 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Network
Memory
Disk
CPU
Platform Profile: Queued requests
% capacity
System resource limitations
50%
20%
40%
85%
Rule of Thumb: Queued msgs, 1 KB messages, broker performance is about 100 to 1,000 msg/sec for each 1 gHz cpu power.
© 2007 Progress Software Corporation24 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Architecture Spec: Service distribution
Identify services performance characteristics Identify high-bandwidth message channels Decide which components can be modified Annotate with message load estimates
Back Office PartnerField Front Office
CRM
Fin
an
ce
SF
ACRM
SC
M
ESBESB
Tra
ckin
g
Ser
vice
ESB
Po
S
ERP
Adapter
SCM
Adapter
Partner ESB
Integration Broker
© 2007 Progress Software Corporation25 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Architecture Spec: Data Integration
Approximate the complexity of data schemas Identify performance critical transformation events Estimate message size Identify acceptable potential services
Data Schemas 1…n Data Schemas 2…m
?
Expert Tip: Transform tools vary in efficiency:XSLT – slowest (but most standard)Semantic modeler – generally faster (e.g. Sonic DXSI)Custom text service – fastest, but not as flexible
© 2007 Progress Software Corporation26 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
DEMO: Test lab setup
Test hardware• guidelines for lab computers• setting up the lab network
Test architecture• location of test components• installation of brokers• configuration of service containers
Test design assets• sample service definitions & wsdls• sample test documents
© 2007 Progress Software Corporation27 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Specifying Test Cases
Factors to include:• Load, sizes, complexity of messaging• Range of scalability to try (e.g. min/max msg size)• Basic ESB Process model• Basic distribution architecture
Details to avoid:• Application code (unless readily available)• Detailed transformation maps
Define relevant variables:• Fixed factors• Test Variables• Dependent measures
© 2007 Progress Software Corporation28 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Typical test variables
JMS Client variables:• Connection / session usage
• Quality of Service
• Interaction mode:
• Message size and shape
ESB container variables• Service provisioning and parameters
• Endpoint / Connection parameters
• Process implementation and design
• Routing branch or key field for lookup
Expert Tip: External JMS client variables are easily managedwith the Test Harness.
© 2007 Progress Software Corporation29 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Example: Test Case Specification
For each identified Test Case there is a section specifying the following: Overview of test
• How this use case relates to the scenario• Key decision points being examined
Functional definition• How to simulate the performance impact• Description of ESB processes and services• Samples messages• Design alternatives that will be compared
Test definition• Variables manipulated • Variables measured
Completion criteria• Throughput and latency goals• Issues and options that may merit investigation
© 2007 Progress Software Corporation30 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Agenda
Methodology: Review the recommended approach to project and
proceduresAnalysis: Understand how to characterize performance requirements and
platform capacity
Testing:
Learn how to simulate performance scenarios using the Sonic Test Harness
Tuning: Know the top ten tuning techniques for the Sonic Enterprise
Messaging backbone
Analyzing, testing and tuning ESB/JMS performance
© 2007 Progress Software Corporation31 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Testing Performance
Staging test services in the test bed Staging brokers and containers Configuring the Sonic Test Harness Running performance tests and gathering
data Evaluating results for each test case
© 2007 Progress Software Corporation32 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Staging Test ServicesDeploying existing services
Appropriate to use actual implementation of a service IF• robust implementation exists
• minimal effort to set up in test environment
• no side effects with other test components
Production ready services merit special treatment:• perform unit load tests to get baseline
• document possible tuning / scaling options
© 2007 Progress Software Corporation33 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Staging Test ServicesPrototyping proposed new services
Prototype should include:• Correct routing logic for Use Case process
• Approximately correct resource usage
• Generic data
Prototype does not need:• Detailed business logic
• Exception handling code
• Invocation of non-critical library calls
It’s a prototype. Just keep it simple
© 2007 Progress Software Corporation34 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Staging Test ServicesSimulating non-essential services
Use ‘stub’ service as placeholder for service step that are not performance-sensitive
Can return generic data Ensures ESB process for target use case will run
correctly Useful stub services:
• Transform service• GetFile service• PassThrough service• Enrichment service• Prototype service (version 7.5.2 or later)
© 2007 Progress Software Corporation35 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Demo: Provisioning test services
StatusInfo
WSI: Address Svc
StatusRequest
StatusQuery
XForm: Build query
DBSvr: Query
TestHarness
WebPortal
Adapter: M/F Callout
AddrInfo
QueryResult
EnrichedResult
WSI: Address Svc
StatusQuery
StatusQuery
PassThru:
DBSvr: Query
PassThru: Sleep
StatusQuery
QueryResult
QueryResult
Business Use Case Performance Test Case
© 2007 Progress Software Corporation36 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Provisioning test brokers
Test broker must be similar to production• Correct Sonic release and JVM• Expected deployment heap size and settings
CAA configuration• Optimize network for replication channels• Locate on separate host to avoid bottlenecks• If failover testing is part of plan:
– define fault tolerant (JNDI) connections
DRA configuration• Set up subset of clusters and WAN simulations• Measure local broker configs first, then expand
Expert Tip: For each client connection scenario define a namedJNDI Connection Factory and document its characteristics.
© 2007 Progress Software Corporation37 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Provisioning ESB containers
Use ESB Container to manage service groups• name accordingly to service group role
• plan to reshuffle services during tuning phase
• provision jar files out of sonicfs
Use MF Container to control distribution• name according to domain/host location
• configure Java heap appropriately– for IBM jdk make -Xms = -Xmx– for caching services (e.g. Transform, XML Server), add
extra memory for locally-cached data
Rule of Thumb: Default heap size is fine for most ESB containers;if memory is limited, reduce size, but no smaller than:8MB + ∑(service jar size) + (#Listeners) * (200KB)
© 2007 Progress Software Corporation38 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Demo: Setting up containers for test
Workbench view of containers• Coding and debugging the prototype
Runtime view of containers• managing the distributed environment
• reinitializing back to a known state
© 2007 Progress Software Corporation39 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Simulating endpoint producers and consumers
Endpoint protocols and performance• Test Driver options for various protocols
Simulating process/thread configuration Implementing endpoint interaction modes Configuring client Quality of Service (QoS) Generating message content Demo of client/endpoint simulation
System Under Test
TestHarness
© 2007 Progress Software Corporation40 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Endpoint protocols and performance
JMS• fastest client protocol• strongest range of QoS and Failover
HTTP• moderate performance and QoS• rigid connection model (requires client or router reconnect logic)
Web Services• slowest performance• QoS and recovery depend on WS-* extensions
File-based• flat file pickup / dropoff, FTP• limited to disk speeds (i.e. 1 to 5 MB / sec)• appropriate for batch processing scenarios
JCA• appropriate for EJB server scenarios• limited to EJB transaction speeds (i.e. 100 to 1000 msg / sec)
Rule of Thumb: HTTP acceptors typically achieve ½ to ¼ the performance of comparable JMS/TCP acceptors.
© 2007 Progress Software Corporation41 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Client session configuration
Broker performance depends on scalability of connections and sessions• JMS best-practice is one thread per session
• JMS sessions can efficiently share a connection
• Use session pool for clients and app servers
For test simulation:• determine allowable range of client threads
• test connection/session numbers up to max # threads
• distribute client processes / drivers across multiple machines, if needed, to avoid client-side bottleneck
Rule of Thumb: Create one connection for every 10 to 20 sessions
© 2007 Progress Software Corporation42 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Configuring client Quality of Service (QoS)
HTTP and Web Services clients• best possible QoS is at least once• even with WS-Reliability
JMS Client:• CAA w NonPersistentReplicated → exactly once• Many shared sub’s versus one durable sub• NonPersistent Request/Reply → at least once• Discardable send to avoid queue backup• Flow to disk to prevent blocked senders
ESB service:• Exactly once uses JMS transaction• At least once uses client ack• Best effort uses dups_ok ack
Broker:• Sync (default) versus Async disk i/o
Expert Tip: Best compromise is at least once QoS based on CAA, Persistent, Async disk and DUPS_OK.
© 2007 Progress Software Corporation43 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Generating message content
Simulate message size / distribution for accurate results Message content may trigger ESB routing rules Some services depend on message content
• key values must match existing data / rules• duplicate key value could cause error• services that cache content require accurate key distribution
Simulating content in the client / driver• file-based message pool• message template generation• Java / object message generator• Message properties
StatusRequest
priority=2org=1
Cust Info
Sonic Test Harnesssupports all these
gen random int
gen sample xml
Addr svc lookup
transform rule
© 2007 Progress Software Corporation44 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Demo: Simulating clients with Test Harness
JNDI connection configuration Producer / Consumer parameters Message generation
System Under Test
TestHarness
Connection
Broker
JNDI
MsgPool
Reply
Request
MessageGeneration
© 2007 Progress Software Corporation45 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Running performance test iterations
Logistics of test orchestration• managing multiple Test Harness clients
• configuring test intervals
• test warm-up and cool-down
Data collection correlation Ensuring repeatability of results Demo of Test Harness iterations
© 2007 Progress Software Corporation46 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Logistics of test orchestration
Managing multiple Test Harness clients• Simplest option is multiple command windows
– use telnet sessions for remote hosts– initiate test and warmup– hit <enter> key in each window
• Advanced environments can use distributed driver:– Grinder, SLAMD, JMeter, LoadRunner, …
Configuring test intervals• long enough to detect trend effects• short enough to allow fast iteration across tests
Test warm-up and cool-down• helps eliminate first-time test artifacts• ensures steady-state performance numbers
Rule of Thumb: Start with 10-20 intervals of 30 secs; if steady state is soonreached, reduce to 10 intervals of 10 seconds, plus warmup.
© 2007 Progress Software Corporation47 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Data collection correlation
Test Harness output:• Throughput (msg/sec)
• Latency (msecs per round trip)
• Message size (bytes)
System measures:• CPU usage (%usr, %sys)
• Disk usage (writes/sec)
Broker metrics:• Messaging rate (bytes/second)
• Peak queue size (bytes)
© 2007 Progress Software Corporation48 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Ensuring repeatability of results
Experimental method requirement• critical in measuring impact of change
• validate by rerunning identical test
Most common artifacts impacting repeatability• messages left in queue
• duplicate files dropped in file system
• growing database size / duplicate keys
• disconnected Durable subscribers
• cached Service artifacts (ESB default)
Expert Tip: Run initial tests twice in succession; if results differ by morethan 3% or so, investigate why.
© 2007 Progress Software Corporation49 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Demo of Test Harness iterations
Baseline test Change test harness properties Rerun test Show spreadsheet across tests
© 2007 Progress Software Corporation50 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Evaluating performanceMeasurement against goal
Short of goal• Perform bottleneck analysis / attempt tuning
• Review option of scaling up resources
• Review design change options
• Give up and re-think goal
Meet or exceed goal• Continue scaling up and tuning ‘til it breaks
• Give up and declare success
© 2007 Progress Software Corporation51 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Evaluating performanceBottleneck analysis
Review of resource consumption• Determine cpu-bound, disk-bound, net-bound
• Identify components using the resource
• Possibility of offloading to other hosts
Examine trends in scalability tests• Possibility of improving throughput by adding more
client sessions, ESB listeners, clustered brokers, etc.
• Option of rebalancing (# threads, java heap, priority
Go through Top Ten tuning tips and others… Last resort: recode or redesign to save cycles
© 2007 Progress Software Corporation52 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Evaluating performanceCompare across test runs
Carefully planned test runs yield fertile comparisons:• estimate cost/benefit of a feature or option
• estimate incremental overhead of a tunable parameter
• narrow the field of concerns and alternatives
Advice in collating and analyzing test runs• collect test summary results in spreadsheet
• save raw data and logs in a separate place
• save test config, so you can replicate later
• schedule ad hoc review after each test sequence
Expert Tip: Verify key conclusions by replicating key test runsthat led to that conclusion
© 2007 Progress Software Corporation53 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Update Scenario
Demo: Example test result matrix
Query Scenario
ORX 1 KB 112 331
ORX 10 KB 146 460
ORX 100 KB 152 1197
XSVR 1 KB 121 68
XSVR 10 KB 632 139
XSVR 100 KB 2113 688
Msg S
ize
DB Svc
Thruput *
Latency **
Test 1
Test 2
Test 3
Test 4
Test 5
Test 6
* kbytes / second** milliseconds
© 2007 Progress Software Corporation54 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Agenda
Methodology: Review the recommended approach to project and
proceduresAnalysis: Understand how to characterize performance requirements and
platform capacityTesting: Learn how to simulate performance scenarios using the Sonic
Test Harness
Tuning:
Know the top ten tuning techniques for the Sonic Enterprise Messaging backbone
Analyzing, testing and tuning ESB/JMS performance
© 2007 Progress Software Corporation55 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Performance Tuning with Sonic ESB®
Diagnostics• Review of ESB architecture
• Factors influencing message throughput
• Factors influencing message latency
• Factors influencing scalability
Top Ten Tuning Tips Other tuning issues
• Broker parameters
• Java tuning
• ESB tuning
• Specialized ESB services
© 2007 Progress Software Corporation56 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Review ESB Architecture
SOA view
SERVICES
COMMUNICATION BACKBONE
NETWORK
SERVICECONTAINER
ESBCONTROL
System view
MessageLog DBMS Cache /
Swap
Th
read
s
VM
Th
read
s
VM
Th
read
s
VM
Th
read
s
VM
I/O Controller
Mem
ory
Network Interface
Router/Switch Router/Switch
© 2007 Progress Software Corporation57 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
ESB System Usage: CPU
Sources of CPU cost:
• Application code
• XML Parsing
• CPU cost of i/o
• Network sockets
• Log/Data disks• Web Service protocol
• Web Service security• Security
• Authorization• Message or channel encryption
© 2007 Progress Software Corporation58 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
ESB System Usage: Disk
Sources of Disk overhead:
• Database services
• Large File / Batch services
• Message recovery log
• Might not be used if other guarantee mechanisms work
• Message data store
• Disconnected consumer
• Queue save threshold overflow
• Flow to disk overflow
• Message storage overhead depends on disk sync behavior
• Explicit file synchronization ensures data retained on crash
• Tuning options at disk, filesystem, JVM and Broker levels
Expert Tip: To estimate best-case message log performance, usethe DiskPerf utility bundled with Test Harness.
© 2007 Progress Software Corporation59 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
ESB System Usage: Network
Sources of Network overhead:
• Client application messages and replies
• Service to service steps within the ESB (except intra-container)
• ESB Web Service callouts
• CAA broker replication messages
• Metadata (JMX, cluster, DRA) messages (normally < 1%)
Computing network bandwidth:
• Network card: 100 mbit ~= 12 MB/sec, 1 gbit ~= 120 MB/sec
• Network switches are individually rated
Computing network load:
• message rate X message size
• include response messages and intermediate steps
• add ack packets (very small) for each message send
Expert Tip: To estimate best-case network performance, usethe DiskPerf utility bundled with Test Harness.
© 2007 Progress Software Corporation60 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Tip #1: Increase sender and listener threadsto make service & client scale
Increase # Listeners for key entry Endpoints Add more Service/Process instances Warning: Intra-Container ignores endpoint
• split scalable service into separate container• turn intra-container messaging off• note: sub-Process is always intra-container
Container1
ServiceX
…
Container2
ServiceX
… …
Expert Tip: Stop increasing Listeners when CPU usagenears maximum acceptable.
© 2007 Progress Software Corporation61 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Tip #2: Implement optimal QoSto balance speed versus precision
ESB
QoS
MQ
SettingMessage Loss Events
Duplicate
Msg Events
N/A DiscardableBuffer overflow,
Any crashNever
Best EffortNonPersistent,
DupsOK ack
Broker crash,
Client crashNever
At Least OncePersistent,
DupsOK ackNever Never
Exactly Once Transacted Never Never
Expert Tip: With CAA configured, Best Effort service is equivalentto At Least Once, with substantially lower overhead.
(Based on CAA brokers and fault-tolerant connections)
© 2007 Progress Software Corporation62 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Tip #3: Re-use JMS objectsto reduce setup costs
Objects with client and broker footprint:• Connection• Session• Sender/Receiver/Publisher/Subscriber• Temporary destination
Tuning strategies:• Reuse JMS objects in client code• Share each Connection across sessions• Share Sessions across Producers and Consumers
– but not across JVM Threads• For low-load topics/queues
– Use Anonymous Producer – Use wildcard or multi-topic subscription
Rule of Thumb: Up to 500 queues per Broker and10,000 topics per broker.
© 2007 Progress Software Corporation63 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Tip #4: Use intra-container service callsto avoid broker hops
BrokerDispatch Outbox
Marshall Msg
Send Msg
…
…Dispatch Outbox Unmarshall Msg
Marshall Msg Call onMessage
Send Msg …
Receive Msg
Unmarshall Msg
Call onMessage
Instantiate Proc
…
Inter-Container Messaging
Intra-Container Messaging
v7.5: better!faster!
© 2007 Progress Software Corporation64 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Tip #5: Use NonPersistentReplicated modeto reduce disk overhead
Normal broker mechanisms require disk sync• contributes to latency across the board
• interferes with batching of packets
• limited bandwidth
Disabling disk sync eliminates this overhead• Send mode NonPersistentReplicated
• Optional broker params to disable entirely
• WARNING: Log-based recovery will lose recent messages
• BUT: CAA failover will not
Rule of Thumb: Network overhead for Replication channel isabout ½ the Persistent Msg load of the broker.
© 2007 Progress Software Corporation65 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Tip #6: Use XCBR instead of CBRto eliminate Javascript overhead
CBR rules implemented via Javascript• dynamic change with complex rules
• very high overhead for runtime engine
XCBR rules extract data fields for comparison• only simple comparisons supported
• no script engine overhead
• use message property data key for best effect
Rule of Thumb: Invocation of the Rhino javascript engine requiresabout 100 milliseconds cpu time on a typical platform.
© 2007 Progress Software Corporation66 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Tip #7: Use message batchingto accelerate message streams
Message transfer overhead is generally fixed Hidden ack messages amenable to tuning:
• AsyncNonPersistent mode decouples ack latency• Transaction Commit allows 1 ack per N messages• DupsOK ack mode allows ‘lazy’ ack from consumer• Pre-Fetch Count allows batched transmit to consumer
ESB Design option: send one multi-part message instead of N individual messages
XML transforms and other services handle multi-record data efficiently
Producer Broker Consumer
Msg
Msg
…Ack
Msg
Msg
Ack
…
© 2007 Progress Software Corporation67 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Tip #8: Minimize XML/SOAP operationsto avoid parsing overhead
Bypass SOAP and Web Services processing• Use HTTP Direct Basic instead of SOAP or WS• Risk of invalid XML if source is unreliable
Combine multiple XML parsing steps into one• Save target XPath results as Message props• Also relevant for BPEL correlation ID’s
InputMessage
XMLTransform
CustomJAXB
InputMessage
propY=9propX=1
InputMessage
JAXB ObjMsg part
XCBR
JAXBService
© 2007 Progress Software Corporation68 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Tip #9: Use high-speed encryptionto reduce security overhead
Default SSL encryption uses old RCA stack• At least 2X slower than more modern options
Change to any JSSE compliant stack:• modify client DSSL_PROVIDER_CLASS to:
progress.message.net.ssl.jsse.jsseSSLImpl
• change broker SSL provider from RSA to JSSE
Use more efficient cipher suites:• RSA_With_Null_MD5 is the smallest and fastest
Reduce broker memory overhead by deleting any unused ciphers
© 2007 Progress Software Corporation69 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Tip #10: Use stream API'sto improve large message performance
SonicMQ Recoverable File Channels• Uses JMS layer to manage large file xfer• Queue-based initiation of transfer• High-speed JMS pipeline for blocks of data• Recovery continues at point interrupted
Sonic ESB open-source Large Message Service• Provides dynamic provisioning• Interacts with ESB processes
SonicStream API (version 7.5 or later)• Topic-based, pipeline into Java Stream api• No recovery
© 2007 Progress Software Corporation70 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Broker Tuning Parameters
Core Resources:• JVM heap size• JVM thread, stack limits• DRA, HTTP and Transaction threads• TCP settings
Message storage:• Queue size and save threshold• Pub/sub buffers• Message log and store
Message management• Encryption• Flow control and flow-to-disk• Dead message queue management
Connections• Mapping to NIC’s• Timeouts• Load balancing
Rule of Thumb: For non-trivial queues, multiply defaultsettings by 10 to 100.
© 2007 Progress Software Corporation71 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Java Tuning Options
‘Fastest’ JVM depends a little on the application and a lot on the platform
VM heap needs to be large enough to process load, but small enough to avoid system swapping
Garbage Collection:• default settings good for optimal throughput• use advanced (jdk4 or later) GC options to
optimize worst-case latency
Rule of Thumb: On windows platforms, the Sun 1.5.0 JVM is10% to 50% slower than the default IBM 1.4.2 JVM.
© 2007 Progress Software Corporation72 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
ESB Tuning Options
Load balancing and scalability of services:• number of distributed service instances
• number of service listener threads
Container Java VM heap size Intra-Container messaging Endpoint and connection parameters
• same principles as JMS client
Expert Tip: Start with small Java heap and only increase -Xms size if it improves performance.
© 2007 Progress Software Corporation73 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Discussion of Service tuning
Transformations XML Server BPEL Server Database Service DXSI Service
© 2007 Progress Software Corporation74 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Other fun things you can tune
Database: indexing, query optimization SOA patterns: federated query, temporal
aggregation, split/join, caching XML: DOM, SAX, XStream Disk: Device balancing, RAID, mount params Network: Nagle algorithm, timeouts
Expert Tip: If you save service resources in sonicfs, the ESBcontainer will dynamically load and cache them.
© 2007 Progress Software Corporation76 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Questions?
© 2007 Progress Software Corporation77 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging
Thank you foryour time
© 2007 Progress Software Corporation78 SONIC-7: Tuning and Scalability for Sonic Enterprise Messaging