IBM Software Group
®
WebSphere® Support Technical Exchange
Service Integration Bus Best Practices in WebSphere Application Server
06th September, 2016
Jhansi Kolla
IBM® Support Technical Exchange
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 2
Agenda
We will discuss the most common Service Integration
Bus (SIB) best practices in WebSphere Application
Server (WAS).
How to tune the SIB component to get the maximum
outcome. (SIB performance tuning)
How to improve the performance by selecting proper SIB
policies according to your architecture and environment.
Other WAS runtime components that effect SIB
performance and how to tune them.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 3
Intoduction
SIB is the Default Messaging Provider in WAS.
Many WAS stacked products use SIB as their base
component. So this WSTE also applies to all those
stacked products.
We will also cover a few things which are not available in
the info center.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 4
SIB Tuning Levels
There are different ways we could configure SIB, in order
to improve SIB performance and best outcome of it:
Configuration
Runtime
JMS™ Producer side
JMS™ Consumer side
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 5
SIB Policy Options: HA + Scalability If you are using messaging cluster, you can avoid a single point of failure by
creating and running more than one Message Engine (ME) (at least 2). You can
always manually add messaging engines (in case of just HA policy is setup).
It‟s always good to have Scalability + High Availability if your architecture allows.
Best practice: To improve the throughput and scalability of the messaging functionality:
Add MEs to handle very large numbers of connections and rely on remote put to forward
messages to their destination.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 6
Preferred Servers Only Option In the case of default HA policy if you choose to run your ME on preferred
servers only, then make sure that you have at least one spare JVM™
available to host ME (in case if ME hosting server has any issues and ME
needs to failover).
Add more
preferred
servers
Best Practice: Configure custom One of N policy with different preferred servers (if
you configure to run the ME on preferred servers option) for each ME and with
failback option.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 7
JVM Heap Settings for Performance Each message residing inside a Messaging Engine(ME) needs around 400 bytes of Meta data (header) of heap memory, and in the case in
Pub-Sub scenario if a durable subscriber is not active then Messaging Engine has to hold a message reference apart from actual Message.
Each and every message references consumes around 100 bytes of Meta data (header) of heap memory.
So while designing the JVM memory requirements, message load (i.e the number of messages present in Messaging Engine) has to be
considered including in failure cases. In general it is better to set the minimum and maximum heap sizes to the same value for maximum
performance. This setting avoids costly JVM Garbage Collector (GC) compaction operations.
Message Retrieval Time: Messages have to be read from the message Store to run search/selector algorithms on them. The best way to
reduce message retrieval time is to increase JVM heap sizes so that more messages can be cached in memory.
Best Practice: Configuring sufficient JVM heap size to avoid the hangs or OOM during
message processing and improve performance. We suggest minimum of 2 GB or above in
customers production ME JVMs. In general setting same Max and Min heap size avoid costly
JVM garbage collector (GC) compaction operations.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 8
Balancing Message Workload• If the producing application is connected directly to an ME in the cluster, all its
messages will go to the local SIB partition destination (by default). To spread the
messages across the ME‟s in the messaging cluster choose below configuration.
• This also increases the throughput of the message producers.
(Note: in this case message order can no longer be guaranteed due to the workload balancing of messages across queue partitions).
Best Practice: To increase the messages throughput and Work Load Management (WLM)
across the messaging cluster JVMs, configure “don‟t prefer a local queue point over the other
queue points” and “Messages may be sent to different queue points” in the queue
configuration panel.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 9
Balancing message workload
Best practice: To increase the throughput and WLM choose above options on the
Queue object that points to the JMS Destination.
Producer Example
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 10
remotePut (Producer side) The producing application, connected to ME 1, produces messages to a queue
'Raleigh' localized on ME 2
ME 1 creates a remote queue point (RQP) to forward the messages. Once message
reached RQP then ME1 sends that message to MQ2. This eats system resources.
Best practice: If your architecture allows, always connect the application locally to
avoid the remotePut for better performance.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 11
remoteGet (consumer side)When consuming from a Queue Point that is not located on the same ME that the application is
connected to, a process similar to store and forward is occurring, this is generally referred to as
remote get. The connected ME (ME2) requests a message from the ME with the Queue Point B (ME1)
The ME1 selects a message, persistently locks it for the ME2 and sends a copy of the message to ME2 (store and forward)
The consuming application receives the message, processes it and commits the receive.
The ME2 persists the act of committing the message and asynchronously informs ME1 of the commit.
Asynchronously, ME1 deletes the message and ME2 removes the commit state.
Best Practice: remoteGet eats lot of system resources. For better performance ALWAYS
avoid remoteGet. Make sure your consumer applications are connected to the local ME
which host the SIB destination.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 12
Pros and cons of remotePut & remoteget
Best Practice: Configure your system to avoid remote get (always)
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 13
Avoid remoteGet (Continued)• By default, applications are able to connect to any available ME and prefer an ME in the same server, if one is
available. Connection Factory and Activation Specification definitions provide three Target properties which allows
you to target specific MEs or groups of MEs when establishing connections.
Target
• The name of an ME or a group of MEs to connect
Target Type
• Bus member: Targets MEs in a specific server or cluster
• Custom messaging engine group: Allows you to define a custom group of engines to satisfy the connection
• Messaging engine: Targets a specific ME by name
Target Significance
Required: No connection will be established unless an ME satisfying the target criteria is available
Preferred: If none of the target MEs are available it connect to any ME in that BUS.
Best Practice: Only let messages on a single queue point be visible in order to avoid remote gets and improve
scalability. The above setting avoids the remoteGet.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 14
Always minimize the messages in the Message Store
• Minimize the number of messages currently stored in the message store and in the
cache.
• This is both messages on queue points (and subscriptions) and messages stored for
transmission to other MEs (remote queue points) in this or other buses (SIBLink and
MQLink).
• In a healthy environment as soon as a message arrives on the SIB destination, a
consumer should pick it up. Messages on the SIB destinations should be zero or near
to zero.
Best practice: Do not use queues for long term storage of messages. Keep the queues
depth minimum all the times. SIB is not a database.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 15
Data Store Locking Issues
• If ME is abnormally shutdown or the connection between ME hosting server
and data store hosting machine is broken, the lock on the data store is not
removed automatically. When the ME starts next time, the existing stale lock
on the data store won‟t allow the new ME instance to take ownership and the
ME won‟t start.
• To resolve this enable the KeepAlive at the DB server to lower value like 4 or
5 minutes (default is 2 hours)
Best practice: Configure the KeepAlive at DB server to a lower value like 4 or 5
minutes to release the stale lock on the SIBOWNER table. Keep in mind of
connectionTimeOut and transactionTimeOut when you set this keepAlive.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 16
Enable Restrict Long Running Locks• In WAS version 8.5 a new locking mechanism is implemented that solves this problem without the
need for setting the KeepAlive timeout. Just enable the “Restrict long running locks” feature in the
messaging engines Message Store configuration and restart the messaging engine.
• A new algorithm that ensures ME no longer holds long running locks on the SIBOWNER table.
• Active ME now holds only short locks on the SIBOWNER table while revalidating its ownership at
regular intervals
• The right to own the SIB tables is always performed during ME start-up
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 17
Detect JVM hang and failover• The messaging engine holds ownership of the database as long as it is active and processing. The
moment the messaging engine goes down or loses its connection to the database, the ownership is
lost to another instance as decided by the HA manager. The standby instance has no trouble
coming up as there are no database locks held by the failed messaging engine.
• Whenever active ME server does not crash, but hangs then this MELastUpdate don‟t update the
time after heartbeats are lapsed it removes server1 ME lock on data store, and server2 ME is
started with out any issues.
Best practice: Enable “Restrict long running locks” from WAS 8.5 onwards.
Note: if you migrated from
WAS 7.* or 8.* to 8.5 you
need to run the
“sibDBUpgrade” command
in order to modify the SIB
tables to use this new
function in WAS 8.5.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 18
File Store Locking Issues
• NFS3 has some file locking issues. Don‟t use NFS3 for a file store.
• In case of clusters make sure file store is accessible from all ME servers in that cluster.
• Run the IBM File System Locking Protocol Test for WebSphere Application Server:
http://www.ibm.com/support/docview.wss?uid=swg24010222
• Network File System (NFS) Recommendations for WebSphere Application Server:
http://www.ibm.com/support/docview.wss?uid=swg21645066
Best practice: Use NFS4 or approved file system (as per above test) for the file store.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 19
Data store performance settings
The workload that ME imposes on DB is slightly different than normal DB workloads.
ME performs mainly SQL INSERT / DELETE. This has to be considered while DB
tuning.
Each ME can request a large number of concurrent conns to the DB. By design ME
used many threads to perform DB updates concurrently.
Best practice: Making available sufficient number of DB connections to ME is the key
to have better performance during the peak loads.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 20
Thread pool settings Default: Each ME can request a large number of concurrent connections to the DB. By design ME
uses many threads to perform DB updates concurrently.
SIBFapThreadPool is the service integration bus format API protocol (SIBFap) FAP outbound
channel thread pool, and it is utilized by all JMS application sending or consuming messages
to/from the server. The optimum value for its maximum size should be around 50.
Best practice: By default these SIB threadPools are enough, but if you have
many Message Driven Beans (MDB)‟s then you need to revise these SIB
threadPools settings in addition to the default threadPools.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 21
Consume All Messages Before ME Stop
If you are planning to stop the ME, consume all messages from SIB destinations before
stopping. It helps to start the ME quickly when you restart the ME.
If the ME has thousands of messages on queues and the ME is stopped (precisely or
abnormal shutdown), ME restart takes longer because it must reconstitute (recreate) all
message on the queues. To avoid this try to consume all the messages from the SIB
destination as much as possible.
Best Practice: Consume all messages before ME stop. For maintenance
purposes stop the ME first and then JVM, this makes proper house keeping of
messages and ME shutdown. Never shutdown ME abnormally.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 22
Message Reliability Vs Performance• Plan your configuration carefully to achieve maximum reliability while minimizing complexity
• Default message reliability (Quality of Service) is Assured Persistent. This guarantees that messages will never be lost but
it also has a high impact on performance.
• The fastest messaging can be achieved with Quality of Service is set to Best Effort Nonpersistent. These
messages are not stored in the message store and so messaging performance is much faster... but it is also much easier to
lose messages. This setting is best for highly transient data that is valid for a short period of time and will soon change.
• There are several other settings in between Assured Persistent and Best effort Nonpersistent. A table in the WAS Info
Center shows each and explains the conditions under which each type of message would be discarded by a messaging
engine.
• http://www.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/cjj9000_.html
Best effort non-persistent
Express non-persistent
Reliable non-persistent
Reliable persistent
Assured persistent.
Best Practice: Leave messages at the default Assured Persistent level. It is slower but it's the safest option and in most
cases performance using this QoS isn't an issue.
• In case if you have to change the default reliability level, plan your configuration carefully to achieve maximum reliability
while minimizing the complexity. Note: As the reliability level decreases, the faster the messages can be processed.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 23
ME not starting:• There are many reasons for ME is not starting.
• Check whether ME is not deactivated. If so enable the ME through the core group settings.
• Check for any locking issues of the message store.
• In case of clusters check all ME servers logs if ME started in the other servers.
• Resolve any file store or data store issues.
• Make sure HMGR is enabled in case of ME clusters.
• If you configured to run ME on preferred servers only and check whether any preferred server is running
with out any issues.
• Split brain or any communication issues between the ME server in clustered environment.
Other useful links:SIB- Messaging Engine startup problems
SIB Messaging Engine Data Store Connectivity Problems and Solutions
SIB Messaging Engine startup problems and their solutions
SIB message store problems
SIB Message flow problems
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 24
Scaling producers• In case of HA SIB policy add MEs to handle very large numbers of
connections and rely on remote put to forward messages to their
destination
• Adding more MEs when using messaging clusters will automatically create
additional queue partitions. Each partition has the capacity of a normal
queue point. The queue capacity scales linearly with the cluster. (note: at
the same time don‟t use the queues for long term storage or don‟t use like
database)
Best Practice: Add MEs and configure preferred servers to handle very large numbers of
connections and throughput.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 25
Don‟t use the Message listener ports. These are very old
fashioned and low efficiency consumers.
Use Activation Specifications which are more productive
and more flexible.
Best Practice: Use activation specification at consumer
side for better performance.
Consumer/MDB Side Tuning
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 26
Why are messages piled up on SIB destinations??
Producer is producing the message faster than consumer consuming
the messages.
Consumer is hung or having its internal issues. (Java™ cores will
show any hangs during that time)
Consumer is not running.
Check if any of above applies to your messages accumulation.
Best Practice:
1) In the Activation Specifications configuration panel: Increase the
„Maximum concurrent MDB invocations per endpoint‟ if you want to
increase maximum throughGet.
2) Activate MDB‟s in all servers
3) In queue configuration panel: configure „Messages on all queue
points are visible‟.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 27
Consumer/MDB Side Tuning Message consumers in WebSphere are almost always Message Driven Beans (MDB). MDBs use Activation
Specifications for message consuming. Important performance related options include:
Always activate MDBs in all servers.
Maximum concurrent MDB invocations per endpoint (10 is the default). Increasing this value can improve
throughput, especially if MDBs are slow to complete.
Even if the MDB is deployed in the cluster, if you don‟t configure “Always activate MDBs in all servers” only the local MDB
where the ME is running is able to consume messages. If you enable this, all MDB‟s on all servers are able to
consume messages from the remote MEs. This greatly increases message consumption rates.
Best practice: Tune your MDB thread pool to ensure it can run the maximum number of MDBs without waiting. Configure
“Always activate MDB‟s in all servers” and set Maximum concurrent MDB invocations as per your architecture allows.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 28
Configuring dedicated threadpools for MDB‟s
ThreadPool I WSVR0629I: The request buffer for thread pool "SIBJMSRAThreadPool"
has reached its capacity.
SIBJMSRAThreadPool is the thread pool used by consumer applications (MDBs) to
connect to messaging engines.
In general default threadPool settings are enough in normal conditions. Lets say, you 8
MDB’s and each MDB has 10 default “Maximum concurrent MDB invocations per
endpoint. You may need to increase the SIBJMSRAThreadPool setting to 80. If you do
not modify this, MDBs many need to wait for the threadPool to make threads available.
In this situation, MDB’s message processing may get hung.
Best practice: Make sure you have configured enough SIBJMSRAThreadPool threads
available for the MDB’s. Tune your SIBJMSRAThreadPool to ensure it can run the
maximum number of MDB’s (configured) with out waiting. If you need to increase this
setting you may need to review the over all server wise threadPools too.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 29
Scaling consumers
•By default a JMS consumer is only able to consume messages stored on the queue
partition that it is connected to. Even if there are messages on the other queue points of
the same cluster queue, by default consumer is not able to see them. If MDB is not able to
see the messages on the remote queue points, this can be fixed by enabling the
“Message visibility” option
Best practice: To increase the messages consumption rate, set “Messages on all
queue points are visible”. This makes, let consumers consume the messages
from all queue partitions.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 30
Best practices for using JMS connections• Once the application is successfully created the connection by getConnection()
method, your application must ensure that connections are closed whenever they
have completed their work. If a connection exception is thrown, WAS does not
automatically call connection.close() on behalf of your application connection.
• Application should call the connection.close() method in the finally {} block.
• Use Shared connections vs unshared connections to suit your applications design.
Default is Shared connections. Sessions should never be used across multiple
threads.
• Shared connections are transaction scope. Until transaction is closed, connection
don‟t go back to the Free Pool even though your application called the
connection.close() method.
Best Practices:
• Call connection.close() in finally {} block so that connection is properly closed.
• For long running transaction use unshread connections. It close the connection once
application call connection.close().
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 31
Best practices for Work Load Management• If WLM is important, set the server weights accordingly. The SIB TRM (Topology
Routing Manager) is the component that does the proper routing of the SIB messages.
SIB TRM works based on the WLM.
Best practice: In case of SIB clusters, if you need work load spread across all servers
set the servers weight equally. In addition to this in the queue configuration choose
“Do not prefer a local queue point over other queue points” and “Messages may be
sent to different queue points”.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 32
Best practices for HA in SIB
• If HA is not enabled in the messaging cluster servers, ME failover won‟t happen.
• In high availability policy messaging clusters, it generate only one ME in the entire
messaging cluster. Always add one or two messaging engines extra to avoid single point
of failure in the environment. This also helps in performance improvements and
increased throughput.
Best practice: Make sure that HA is enabled in all servers in the SIB clustered servers. Make
sure at least 2 or more ME‟s are configured even in the HA policy messaging clusters.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 33
sib.msgstore.cachedDataBufferSizeIn SIB, message is first stored in memory. This cache has a limited sized and the default
setting could possibly be insufficient for intensive loads. The purpose of the cached data
buffer is to optimize the performance of the messaging engine by caching in memory the data
that the messaging engine might otherwise have to read from the data store.
cachedDataBufferSize contain data for which the quality of service (QoS) attribute is better
than best effort nonpersistent. These messages are spilled from the heap to the datastore
when the in memory cache is exhausted. This cache can be controlled by the following 2
variables:
sib.msgstore.cachedDataBufferSize (default 320000 ~320KB) - the size in bytes
of the cache
sib.msgstore.cachedDataItemMaximumSize (100000) - the maximum size an
object allowed into the cache
When a message is spilled from memory (cache filled up) it is persisted to the database. He
Best practice: To improve the performance, you may need to increase these values, at the
same time be aware that these cache sizes are taken from heap. So you may also need to
increase the heap too depends up on your requirements.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 34
sib.msgstore.DiscardableDataBufferSize
discardableDataBufferSize refers to the data for which the quality of service
attribute is best effort nonpersistent.
This data comprises both data that is involved in active transactions, and any other best effort
nonpersistent data that the messaging engine has neither discarded nor consumed. The
messaging engine holds this data entirely within this memory buffer and never writes the data
to the data store. When the messaging engine adds data to the discardable data buffer, for
example when the messaging engine receives a best effort nonpersistent message from a
client, the messaging engine might discard data already in the buffer to make space. The
messaging engine can discard only data that is not involved in active transactions. This
behavior enables the messaging engine to discard best effort nonpersistent messages.. The
messaging engine holds this data entirely within this memory buffer and never writes the data
to the data store.
Default value is: 320000 (approx. 320K)
Best practice: Increasing the size of the discardable data buffer allows
more best effort nonpersistent data to be handled before the messaging
engine begins to discard messages.
cachedDataBufferSize defines the depth in bytes of messages with nonpersistent qualities of services.
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 35
SIB custom properties
• Most commonly used SIB custom properties:
https://www.ibm.com/developerworks/community/blogs/aimsupport/entry/commonly_used_ser
vice_integration_bus_custom_properties_in_websphere_application_server
• Complete SIB custom properties:
https://www.ibm.com/support/knowledgecenter/SSEQTP_8.0.0/com.ibm.websphere.nd.doc/inf
o/ae/ae/rjj_custom_props.html
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 36
CWSIT0103E - No messaging Engine found troubleshooting
• We do see many PMRs quite often with this issue (Even though ME is running).
• I created a blog how to debug this issue with step by step walk through instructions.
• Blog:
https://www.ibm.com/developerworks/community/blogs/aimsupport/entry/common_reasons_fo
r_CWSIT0103E_no_messaging_engine_was_found_exceptions_in_service_Integration_bus
Troubleshooting CWSIT0103E – No ME found(even though ME is running)
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 37
Connect with us!
1. Get notified on upcoming webcastsSend an e-mail to [email protected] with subject line “wste
subscribe” to get a list of mailing lists and to subscribe
2. Tell us what you want to learnSend us suggestions for future topics or improvements about our
webcasts to [email protected]
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 38
Questions and Answers
IBM Support Technical Exchange
WebSphere® Support Technical Exchange 39
Additional WebSphere Product Resources Learn about upcoming WebSphere Support Technical Exchange webcasts,
and access previously recorded presentations at:http://www.ibm.com/software/websphere/support/supp_tech.html
Discover the latest trends in WebSphere Technology and implementation, participate in technically-focused briefings, webcasts and podcasts at:http://www.ibm.com/developerworks/websphere/community/
Join the Global WebSphere Community: http://www.websphereusergroup.org
Access key product show-me demos and tutorials by visiting IBM Education Assistant:http://www.ibm.com/software/info/education/assistant
View a webcast replay with step-by-step instructions for using the Service Request (SR) tool for submitting problems electronically:http://www.ibm.com/software/websphere/support/d2w.html
Sign up to receive weekly technical My Notifications emails:http://www.ibm.com/software/support/einfo.html