tibco businessevents® extreme...about this book this guide describes performance tuning of tibco...

TIBCO BusinessEvents® Extreme

Performance Tuning Guide

Software Release 1.0.2Published March 06, 2013

Important InformationSOME TIBCO SOFTWARE EMBEDS OR BUNDLES OTHER TIBCO SOFTWARE. USE OF SUCH EMBEDDED OR BUNDLED TIBCOSOFTWARE IS SOLELY TO ENABLE THE FUNCTIONALITY (OR PROVIDE LIMITED ADD-ON FUNCTIONALITY) OF THE LICENSEDTIBCO SOFTWARE. THE EMBEDDED OR BUNDLED SOFTWARE IS NOT LICENSED TO BE USED OR ACCESSED BY ANY OTHERTIBCO SOFTWARE OR FOR ANY OTHER PURPOSE.

USE OF TIBCO SOFTWARE AND THIS DOCUMENT IS SUBJECT TO THE TERMS AND CONDITIONS OF A LICENSE AGREEMENTFOUND IN EITHER A SEPARATELY EXECUTED SOFTWARE LICENSE AGREEMENT, OR, IF THERE IS NO SUCH SEPARATEAGREEMENT, THE CLICKWRAP END USER LICENSE AGREEMENT WHICH IS DISPLAYED DURING DOWNLOAD OR INSTALL-ATION OF THE SOFTWARE (AND WHICH IS DUPLICATED IN LICENSE.PDF) OR IF THERE IS NO SUCH SOFTWARE LICENSEAGREEMENT OR CLICKWRAP END USER LICENSE AGREEMENT, THE LICENSE(S) LOCATED IN THE “LICENSE” FILE(S) OFTHE SOFTWARE. USE OF THIS DOCUMENT IS SUBJECT TO THOSE TERMS AND CONDITIONS, AND YOUR USE HEREOF SHALLCONSTITUTE ACCEPTANCE OF AND AN AGREEMENT TO BE BOUND BY THE SAME.

This document contains confidential information that is subject to U.S. and international copyright laws and treaties. No part of this documentmay be reproduced in any form without the written authorization of TIBCO Software Inc.

TIB, TIBCO, TIBCO Adapter, Predictive Business, Information Bus, The Power of Now, Two-Second Advantage, TIBCO ActiveMatrixBusinessWorks, are either registered trademarks or trademarks of TIBCO Software Inc. in the United States and/or other countries.

EJB, Java EE, J2EE, and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in theU.S. and other countries.

All other product and company names and marks mentioned in this document are the property of their respective owners and are mentionedfor identification purposes only.

THIS SOFTWARE MAY BE AVAILABLE ON MULTIPLE OPERATING SYSTEMS. HOWEVER, NOT ALL OPERATING SYSTEMPLATFORMS FOR A SPECIFIC SOFTWARE VERSION ARE RELEASED AT THE SAME TIME. SEE THE README FILE FOR THEAVAILABILITY OF THIS SOFTWARE VERSION ON A SPECIFIC OPERATING SYSTEM PLATFORM.

THIS DOCUMENT IS PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING,BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT.

THIS DOCUMENT COULD INCLUDE TECHNICAL INACCURACIES OR TYPOGRAPHICAL ERRORS. CHANGES ARE PERIODICALLYADDED TO THE INFORMATION HEREIN; THESE CHANGES WILL BE INCORPORATED IN NEW EDITIONS OF THIS DOCUMENT.TIBCO SOFTWARE INC. MAY MAKE IMPROVEMENTS AND/OR CHANGES IN THE PRODUCT(S) AND/OR THE PROGRAM(S)DESCRIBED IN THIS DOCUMENT AT ANY TIME.

THE CONTENTS OF THIS DOCUMENT MAY BE MODIFIED AND/OR QUALIFIED, DIRECTLY OR INDIRECTLY, BY OTHERDOCUMENTATION WHICH ACCOMPANIES THIS SOFTWARE, INCLUDING BUT NOT LIMITED TO ANY RELEASE NOTES AND"READ ME" FILES.

Copyright © 2010, 2012 TIBCO Software Inc. ALL RIGHTS RESERVED, TIBCO Software Inc. Confidential Information

ContentsAbout this book ................................................................................................................ vii

Conventions ............................................................................................................. viiTIBCO BusinessEvents® Extreme community ........................................................... viii

1. Introduction ................................................................................................................... 1What is TIBCO BusinessEvents® Extreme ? ................................................................ 1Approach .................................................................................................................. 1Features .................................................................................................................... 2Tuning ...................................................................................................................... 2Monitoring ................................................................................................................ 3

2. Approach ....................................................................................................................... 5Concepts ................................................................................................................... 5Guidelines ................................................................................................................. 6

3. Using TIBCO BusinessEvents® Extreme features ........................................................... 11Managed objects ...................................................................................................... 11Data grid backed objects ........................................................................................... 12Transactions ............................................................................................................ 12Keys and indexes ..................................................................................................... 14High-availability ....................................................................................................... 14Distribution ............................................................................................................. 15

4. Tuning ......................................................................................................................... 17Deployment ............................................................................................................. 17JVM ........................................................................................................................ 17Shared memory ....................................................................................................... 18Swapping ................................................................................................................. 19Multi-node ............................................................................................................... 19Analyzing Deadlocks ................................................................................................ 20

5. Performance monitoring ................................................................................................ 35JVM ........................................................................................................................ 35Graphical monitoring tools ........................................................................................ 37The statistics tab ...................................................................................................... 43Application monitoring .............................................................................................. 44Node monitoring ...................................................................................................... 51System monitoring ................................................................................................... 62

Index .............................................................................................................................. 69

iii

List of Figures5.1. Visual VM .................................................................................................................. 365.2. JConsole .................................................................................................................... 365.3. Starting system monitors ............................................................................................ 375.4. Select system monitors to start .................................................................................... 375.5. Node transaction rate .................................................................................................. 385.6. Domain-wide transaction rate ....................................................................................... 395.7. Node transaction execution time .................................................................................. 405.8. Domain-wide transaction execution time ....................................................................... 415.9. CPU monitor .............................................................................................................. 425.10. Shared memory monitor ........................................................................................... 435.11. Shared memory monitor ........................................................................................... 445.12. Transaction class statistics ......................................................................................... 455.13. Enable transaction class statistics collection ................................................................ 455.14. Disable and display transaction class statistics ............................................................. 465.15. Transaction locking statistics ..................................................................................... 475.16. Enable transaction locking statistics collection ............................................................. 485.17. Disable and display transaction locking statistics .......................................................... 485.18. Managed object report .............................................................................................. 495.19. Query statistics ........................................................................................................ 505.20. Shared memory usage .............................................................................................. 525.21. Shared memory mutex enable ................................................................................... 525.22. Shared memory mutex display ................................................................................... 535.23. Process local mutex enable ........................................................................................ 545.24. Process local mutex display ....................................................................................... 545.25. Shared Memory IPC ................................................................................................. 555.26. High Availability Partitions Report .............................................................................. 555.27. Shared Memory Allocator Summary ........................................................................... 565.28. Shared Memory Allocator Buckets Report .................................................................. 575.29. System Threads Report ............................................................................................. 585.30. Files Report ............................................................................................................. 595.31. Node Processes Report ............................................................................................. 595.32. Distribution report .................................................................................................... 605.33. Enable distribution statistics ...................................................................................... 605.34. Display distribution statistics ..................................................................................... 615.35. Disable distribution statistics ..................................................................................... 615.36. Clearing distribution statistics .................................................................................... 625.37. JNI Cache statistics ................................................................................................... 625.38. Kernel information .................................................................................................... 635.39. System information ................................................................................................... 635.40. Virtual Memory Report Enable .................................................................................. 645.41. Virtual Memory Report ............................................................................................. 645.42. Network Report Enable ............................................................................................. 655.43. Network Utilization Report ........................................................................................ 655.44. Disk Report Enable ................................................................................................... 665.45. Disk Report .............................................................................................................. 665.46. System Activity Report Enable ................................................................................... 675.47. System Activity Report .............................................................................................. 675.48. Node Processes Report ............................................................................................. 68

v

About this bookThis guide describes performance tuning of TIBCO BusinessEvents® Extreme applications. Thisguide provides the information needed to design, monitor, and improve the performance of TIBCOBusinessEvents® Extreme applications.

This book is intended for the following types of readers:

• TIBCO BusinessEvents® Extreme application developers.

• System architects.

This guide is organized into these general areas:

• Description of the approach and technical background required to design and understand TIBCOBusinessEvents® Extreme application performance. This information is in Chapter 2.

• Overview of TIBCO BusinessEvents® Extreme features and how to use them to build a high-performance application. The trade-offs on using TIBCO BusinessEvents® Extreme features aredescribed. This information is in Chapter 3.

• Architectural and system configuration parameters that impact TIBCO BusinessEvents® Extremeapplication performance. Scaling options are discussed. This information is in Chapter 4.

• Monitoring tools. System and application monitoring tools are discussed and the interpretationof the results and how to use it for performance tuning is described. This information is inChapter 5.

This book is part of a set of TIBCO BusinessEvents® Extreme documentation, which also includes:

TIBCO BusinessEvents® Extreme Installation — This guide describes how to install the TIBCOBusinessEvents® Extreme software.

TIBCO BusinessEvents® Extreme Quick Start — This guide describes how to quickly getstarted using Java IDEs to develop TIBCO BusinessEvents® Extreme applications.

TIBCO BusinessEvents® Extreme Architect's Guide — This guide provides a technical overviewof TIBCO BusinessEvents® Extreme .

TIBCO BusinessEvents® Extreme Administration — This guide describes how to install,configure, and monitor an TIBCO BusinessEvents® Extreme deployment.

TIBCO BusinessEvents® Extreme Java Developer's Guide — This guide describes how toprogram a TIBCO BusinessEvents® Extreme application.

TIBCO BusinessEvents® Extreme System Sizing Guide — This guide describes how to sizethe systems used to deploy TIBCO BusinessEvents® Extreme applications

TIBCO BusinessEvents® Extreme Javadoc — The reference documentation for all TIBCOBusinessEvents® Extreme APIs.

ConventionsThe following conventions are used in this book:

vii

Bold — Used to refer to particular items on a user interface such as the Event Monitor button.

Constant Width — Used for anything that you would type literally such as keywords, data types,parameter names, etc.

Constant Width Italic — Used as a place holder for values that you should replace with anactual value.

TIBCO BusinessEvents® Extreme communityThe TIBCO BusinessEvents® Extreme online community is located at https://devzone.flu-ency.kabira.com. The online community provides direct access to other TIBCO BusinessEvents®Extreme users and the TIBCO BusinessEvents® Extreme development team. Please join us onlinefor current discussions on TIBCO BusinessEvents® Extreme development and the latest informationon bug fixes and new releases.

viii

About this book

1Introduction

What is TIBCO BusinessEvents® Extreme ?TIBCO BusinessEvents® Extreme is an in-memory transactional application server that providesscalable high-performance transaction processing with durable object management and replication.TIBCO BusinessEvents® Extreme allows organizations to develop highly available, distributed,transactional applications using the standard Java POJO programming model.

TIBCO BusinessEvents® Extreme provides these capabilities:

• Transactions - high performance, distributed "All-or-None" ACID work.

• In-Memory Durable Object Store - ultra low-latency transactional persistence.

• Transactional High Availability - transparent memory-to-memory replication with instant fail-overand fail-back.

• Distributed Computing - location transparent objects and method invocation allowing transparenthorizontal scaling.

• Data Grid - secondary object storage, adhoc queries and notification of transactional data changes.

• Integrated Hotspot JVM - tightly integrated Java execution environment allowing transparent lowlatency feature execution.

ApproachPerformance must be designed into an application. It is not realistic to expect performance to betested in at the end of the development cycle. A common sense approach is provided to designTIBCO BusinessEvents® Extreme applications that perform. The key metrics and trade-offs thatimpact application performance are:

1

• Scaling versus path length

• Horizontal versus vertical scaling

• Contention

• Latency versus through-put

FeaturesTIBCO BusinessEvents® Extreme features make high-performance applications possible. However,these features must be used where appropriate. Each of these TIBCO BusinessEvents® Extremefeatures can have an impact on application performance. They must be used when appropriate.

• Managed versus non-Managed objects.

• Transactional versus non-transactional code.

• Keys and indexes.

• Transactions versus Java monitors for concurrency control.

• High-availability (partitioned) Managed objects versus non-partitioned Managed objects.

• High-availability versus distribution.

• Data grid backed Managed objects.

TuningTuning a TIBCO BusinessEvents® Extreme application involves both application architecture de-cisions and appropriate configuration of system parameters. The types of questions that must beanswered are:

• How should the JVM heap sizes be configured?

• How to ensure that the garbage collector does not impact the latency of running applications?

• What size of shared memory should be configured?

• Should System V or file mapped shared memory be used?

• How much disk space is needed?

• Should the application execute on single or multiple nodes?

• How should the application be deployed within a node?

• Should application data be partitioned across multiple nodes?

• When should distribution be used?

2

Chapter 1. Introduction

MonitoringTIBCO BusinessEvents® Extreme provides a rich set of application and system monitoring tools.These tools provide a way to monitor TIBCO BusinessEvents® Extreme applications running undera load to determine how to improve the performance. Standard JVM monitoring tools complementthe TIBCO BusinessEvents® Extreme tools.

3

Monitoring

2Approach

This chapter describes the design approach for TIBCO BusinessEvents® Extreme applications. Italso defines the key concepts that must be understood to perform performance analysis and tuning.

Concepts• Path length

The amount of time that it takes to process a unit of application work (e.g. processing a requestand sending its response), excluding any time spent blocked (e.g. disk I/O, or waiting for a re-sponse from an intermediate system).

• Horizontal scaling

Adding more computing nodes (i.e. machines) to a system.

• Vertical scaling

Adding more resources (e.g. CPUs or memory) to a single computing node in a system.

• Contention

Competition for computing resources. When resources are not available the application waits andoften uses up other system resources competing for the requested resource.

• Latency

The time between when a request is issued and a response is received. Latency can consist of avariety of components (network, disk, application, etc...).

• Through-put

A measure of the overall amount of work that a system is capable of over a given period of time.

5

GuidelinesIdentifying performance requirements• Clear and complete requirements.

Start with a clearly stated set of requirements. Without this, performance work cannot be judgedas either necessary or complete.

• What are the units of work to be measured?

Request messages? Request and response messages? Some larger business aggregation of requestsand responses? Are there logging requirements?

• Which protocol stacks will be used?

How many connections? What are the expected request rates per connection? What do the mes-sages look like?

• What are the Request/Respone latency requirements?

What is the maximum allowable latency? What is the required average latency? What percentageof Request/Response pairs must meet the average latency? Occasionally there are no latencyrequirements, only throughput requirements.

• What is the sustained throughput requirement?

How many units of work per second must be completed, while still meeting the average latencyrequirements?

• What is the burst throughput requirement?

How many units of work per second must be completed, while still meeting the maximum latencyrequirements?

• Are third party simulators required for the testing?

What role do the simulators play in the performance testing? What are their performance charac-teristics? Are they stable, correct, predictable, scalable and linear? Are they capable of generatinga load that meets the performance requirements for the application?

Measuring performanceWorking on performance without first collecting meaningful and repeatable performance data iswasted effort. No one can predict the exact performance of an application, nor can one predict wherethe top bottlenecks will be. These things must be measured. And they must be re-measured as theapplication or environment changes.

• Develop an automated test.

Performance testing involves the repeated configuration and coordination of many individualparts. Doing these steps manually guarantees a lack of repeatability.

• Measure meaningful configurations.

6

Chapter 2. Approach

Do not test performance in the VMware® image. Test in an environment that is the same orsimilar to the production environment.

Use production mode binaries.

Test with assertions disabled.

Eliminate deadlocks from the application. The performance of a path which contains a deadlockis unboundedly worse than the same path without a deadlock. Performance tests which encounterdeadlocks in the steady state are invalid, and indicate application problems.

Do not run performance tests with any extraneous code, logging or tracing enabled. Developersoften add extra code and tracing to an application that may be of use in the development process.This code will not be used in the production system, and can significantly perturb performancemeasurements.

Use representative numbers of client connections. There is much to be learned from the perform-ance characteristics of a single client talking to the application via a single connection. Performancetesting should start with this case. However, TIBCO BusinessEvents® Extreme is optimized forparallelism. Almost all well designed applications will support multiple connections. Performancetesting configuration should mirror the actual application configuration. If the target configurationcalls for 100 connections at 10 messages per second, per connection, test it that way. This is notthe same as one connection at 1000 messages per second.

• Measure the steady state of an application, not the load stabilization time nor the applicationstartup time.

Complex systems generally have complex initialization sequences, and while there are oftenperformance requirements for this startup time, they are generally not the primary performancerequirements. Repeatable performance runs are done against an already started and stable applic-ation, with a warm-up period that allows for the test load to stabilize.

• Run on otherwise idle hardware.

Steady states cannot be meaningfully measured if there is concurrent uncontrolled machine usageon the target system(s).

• Start measuring performance early in a project.

Do not wait until the end of a project to create the performance tests. Performance measurementcan, and should, be done throughout the life cycle of the project. Once a test is in place, it shouldbe mostly a matter of configuration to integrate it with the application. Begin working with thereal application as soon as possible.

• Performance runs versus data collection runs.

Make a distinction between test runs which are measuring best performance, and those that arecollecting data for analyzing performance. Best performance runs should have little or no extradata collection enabled.

• Don't measure saturated systems.

When a system has reached the maximum rate of work that it can process, it is said to be saturated.Production systems are intentionally not run in this way, nor should performance testing be runin this manner. At the saturation point (or approaching it) systems can exhibit various forms ofundesirable behavior; excessive CPU utilization per request, increased memory utilization, non-

7

Guidelines

linearly deteriorating response times and throughput. As the system nears the saturation point,the performance will generally decrease due to these effects. Saturation can occur at differentlevels, including protocol stacks, application logic, and the system itself. Performance testingshould be organized to identify the various saturation points of the system.

For example, CPU utilization should not be driven to 100%. Typically tests should designed todrive the CPU utilization to a maximum of between 80 and 90%.

• Sweeping the load.

Nothing useful can be gained by running a performance test with a single configuration that sat-urates the application. Proper performance testing calls for starting with a simple configurationand a modest workload that doesn't tax the application, and then increasing the load in subsequentruns to identify the linearity of the application and the saturation point. The exercise is then re-peated with other configurations.

Analyzing performanceIn TIBCO BusinessEvents® Extreme applications, we concern ourselves with three main types ofperformance:

• Single-path performance: the CPU cost and wall clock time for a single request.

• Multi-threaded or scaling: running the single path concurrently on multiple threads.

• Multi-node or horizontal scaling: running the single path concurrently on multiple threads onmultiple machines.

We generally want to look first at multi-threaded performance. The TIBCO BusinessEvents® Extremeruntime environment is optimized for running on multi-processor, multi-threaded platforms. Regard-less of the single path speed, additional performance is most easily obtained by running the singlepath concurrently on multiple threads.

At this point you should have a set of data that describes the application functioning normally andin saturation. You will already have done some analysis that lead to your choice of configurationsto measure.

Now look at your data asking scalability questions: Pick unsaturated data points with the samenumber of requests per second, and differing numbers of clients. How does the CPU utilizationchange as the number of clients are increased? If your data shows near-perfect linearity and scalingyour application may not need tuning. In this case, additional performance can be gained by addingmore or faster CPUs. Usually the data shows a lack of scaling or linearity, an inability to utilize allof the CPUs, or overall performance is not acceptable on the target hardware. The next task is tounderstand why. At this point, performance work resembles scientific research:

1. A set of experiments are run and data is collected.

2. The data is analyzed.

3. Hypotheses are made to explain the data.

4. A change is made to the system under test and the process is repeated.

At this point we transition to statistics collection runs to help us identify scaling bottlenecks. Typicallyscaling bottlenecks are a result of contention. Contention can be for a variety of resources; processing

8

Chapter 2. Approach

cycles, network I/O, disk I/O, transaction locks, Java monitors, system memory, etc. Excessivegarbage collection can also be a cause of performance problems. When trying to identify the causeof scaling problems there are no absolute rules which will tell us ahead of time which statistics reportswill produce the most interesting data.

In looking at these data, one should first look for anything that indicates gross problems with therun, such as application failure, deadlocks or swapping. If seen, the results should be disregarded,and the problem corrected before continuing.

At this point, you should have an automated, repeatable test and data which demonstrate performanceand/or scaling issues with your target application. You can now begin to use the collected data tooptimize your application.

After you have removed the bottlenecks and the application scales well across multiple processorsit may still not meet the performance requirements. The single execution path performance shouldbe examined at this time with a Java profiling tool.

Horizontal scaling may also be examined at this point as a way to increase overall system throughput.Add a single node at a time (or pairs for High-availability active and replica nodes) to the test con-figuration and re-run the measurement process.

9

Guidelines

3Using TIBCO BusinessEvents® Extremefeatures

This chapter describes the key TIBCO BusinessEvents® Extreme features and how to use themto ensure optimal application performance.

Managed objectsFeatures:

• Transactional.

• Persisted in shared memory.

• Shared across multiple JVMs.

Cost:

Compared to a POJO, a Managed object will consume additional processing associated withproviding transactional coherency, additional temporary shared memory resources associated withproviding rollback capability, and shared memory associated with persisting the object.

Usage:

• As a replacement for placing object state in a database.

• To transactionally synchronize multi-threaded access to shared application data.

• To provide in-memory objects which can be navigated to with keys.

• When application state needs to be persisted across multiple invocations of the JVM.

• When application state needs to be shared between multiple JVMs.

11

Avoid:

• For temporary objects.

• For data which does not need to be persisted.

Data grid backed objectsFeatures:

• Transactional.

• Cached in shared memory.

• Persisted in distributed data grid.

• Shared across multiple JVMs.

Cost:

Compared to a Managed object, a data grid backed object will consume additional processing forobject creates and destroys to update the data grid. Additional processing is also required if theobject is not already cached in shared memory when accessed by an application.

Usage:

• As a replacement for placing object state in a database.

• As a secondary storage for managed objects.

• Storing large numbers of Managed objects that do not fit comfortably in shared memory.

Avoid:

• For Managed objects that have a short duration lifecycle.

• For Managed objects that require extremely low latency access.

TransactionsFeatures:

• Provide multi-reader, single writer object locking.

• May lock both Managed objects and transactional POJOs.

• Automatic deadlock detection.

• Automatic rollback of modifications to transactional resources when a deadlock or error is en-countered.

Cost:

12

Chapter 3. Using TIBCO BusinessEvents® Extremefeatures

Additional processing for each object that is locked. Additional processing and temporary heapspace for each transactional field which is modified. Additional processing for deadlock detection.Temporarily blocked threads when there is transaction contention.

Usage:

• Used to access Managed objects.

• May be used to transactionally isolate and protect modifications to POJOs.

• Used when multiple reader/single writer access to a resource is desired. Provides scalability formultiple readers executing simultaneously in multiple threads, while still providing data consistencythrough exclusive write locking.

• Small transactions scale better than large transactions.

Avoid:

• Using transactions to manage non-transactional resources (e.g. Network).

• Using transactions when transactional semantics for a resource are not required (e.g a counterthat needs to atomically increment but never rollback).

• Deadlocks. Although the TIBCO BusinessEvents® Extreme runtime automatically rolls backand replays deadlocked transactions, this is very expensive compared to avoiding the deadlockentirely. If deadlocks are seen in your testing, the involved code should be re-organized or re-written to eliminate the possibility of deadlock.

• Promotion locks. When two threads concurrently execute the same code path containing a pro-motion lock, a deadlock will be generated. Promotion locks can be eliminated by changing thecode to take a write lock instead of a read lock at the first access in the transaction to the Managedobject to be modified.

• Transaction lock contention. When a transaction is blocked waiting to acquire a lock, it remainsblocked at least until the transaction holding the lock commits or aborts. It may remain blockedlonger if there are multiple threads competing for the same transaction locks.

• Long running transactions. Transactional resources in multi-threaded applications are generallyshared between threads. Locking a resource in a long running transaction can block other threadsfor the duration of the transaction. Short running transactions scale better than long runningtransactions.

• Large transactions (those that contain many locked resources). Large transactions tend to bemore prone to generating contention and deadlocks. When there is contention between largetransactions, even if there are no deadlocks, the deadlock detection becomes more expensive.

Summary:

Transactions are a powerful tool for maintaining application data consistency and scaling. But thisfeature comes at a cost. Avoid using transactions where they are not necessary.

Java monitorsFeatures:

• Monitors (the Java synchronize keyword) provide a simple mutual exclusion mechanism.

13

Transactions

• Lighter weight than transactions.

• Easy to cause undetected deadlocks.

• Multiple threads sharing read access to a resource become single-threaded when accessing theresource.

Usage:

• Use monitor when synchronization is required for non-transactional resources.

Avoid:

• Using monitors on transactional resources (they are already protected by transaction locking).

Keys and indexesFeatures:

• Keys are only allowed on Managed objects.

• Allows the application to quickly and efficiently navigate to a unique Managed object or group ofManaged objects.

• Supports unique, non-unique, ordered and unordered keys and queries.

Cost:

Each key requires additional processing resources at object creation time, additional shared memoryresources.

Usage:

• Use keys as you would use an index in a database.

• Use unique keys instead of extent iteration for finding a single object.

• Use non-unique keys instead of extent iteration for finding a group of ordered or unordered objects.

Avoid:

• Using keys on objects that don't require navigation to find them.

• Defining unnecessary key fields.

High-availabilityFeatures:

• Transparent, transactional, high performance replication of object data across nodes.

• Transparent routing of data to a partition or node.

• High performance, automated support for migration of object ownership from a failed active nodeto a replica node.

14


Cost:

Additional CPU cycles and memory resources for managing the internal transaction resources whenmodifying a Managed object. Additional network I/O for propagating the modifications to the replicanodes.

Reads of highly available objects have the same performance as reads of Managed objects. No extracycles are consumed and no network I/O is generated.

Usage:

• Use highly available objects to provide non-stop access to application data in the case of nodefailure.

• Use partitions with multiple replica nodes to provide a transparent, transactional push mechanismof object data to a group of nodes.

• Use highly available objects methods to execute behavior on the currently active node for a par-tition.

• Use highly available objects to transparently scale an application load horizontally across multiplenodes.

Avoid:

• Modifying highly available objects unnecessarily. Modifications cause network I/O and processingcycles on the replica nodes. If there is data being modified that is not necessary for the applicationto see after a fail-over, do not keep this data in a highly available object. Use either Managed objectsor POJOs.

Note, that in comparision to Managed objects and POJOs, a highly available object incurs extraprocessing costs even when there are no replica nodes defined for its partition.

• Making highly available objects larger than necessary. Each time a modification occurs, the entireobject is copied to the replica nodes.

• Replicating object data to more nodes than is required. Each additional replica node requiresadditional network I/O and processing.

• For simple load balancing consider using a hardware based solution instead of the locationtransparent routing provided by highly available objects.

DistributionFeatures:

• Direct application access to the creation of remote objects and their data.

• Direct application access to remote method invocation.

Avoid:

• Distributed deadlocks. Distributed deadlock detection uses a timeout to detect a deadlock. Thisimplies that a distributed transaction will wait the entire value of the timeout value before adeadlock is reported. During this period of time the transaction is stalled.

15

Distribution

• For simple load balancing consider using a hardware based solution instead of the locationtransparent routing provided by distributed objects.

16


4Tuning

This chapter describes how to tune TIBCO BusinessEvents® Extreme applications. Applicationand system parameters are described.

DeploymentThe TIBCO BusinessEvents® Extreme runtime supports multiple processes communicating throughshared memory, or a memory mapped file. When a JVM is started using the deployment tool, allruntime resources required by the JVM are available in the same process space. There are caseswhere multiple JVMs on a single node may be appropriate for an application (see the section called“Multiple JVMs” on page 18), but there is a performance impact for dispatching between JVMs.

JVMHeap sizeBy default, TIBCO BusinessEvents® Extreme doesn't modify the JVM heap (-Xms<size> and-Xmx<size>) or stack (-Xss<size>) memory options. If during testing, the JVM is found to runshort of, or out of memory, these options can be modified either setting them as arguments to thedeployment tool.

Both JConsole and VisualVM can be used for looking at heap memory utilization.

Garbage collectionBy default, TIBCO BusinessEvents® Extreme doesn't modify any of the JVM garbage collectionparameters.

17

Multiple JVMsTypically, an TIBCO BusinessEvents® Extreme deployment will consist of a single JVM per node.However, there may be cases where multiple JVMs per node are required (e.g. Exceeding a per-process limit on the number of file descriptors).

TIBCO BusinessEvents® Extreme supports multiple JVMs deployed within a single node. TheseJVMs may all access the same Managed objects.

Multiple JVMs are deployed by using the detach option to the deployment tool.

Shared memory• Size

Shared memory needs to be large enough to contain all of the application's Managed objects, theruntime state, and any in-flight transactions. See the TIBCO BusinessEvents® Extreme SizingGuide for information on how to determine the correct size.

When using data grid backed Managed objects, shared memory only needs to be large enoughto store the sub-set of cached Managed objects.

• mmap

By default TIBCO BusinessEvents® Extreme uses a normal file in the file system. The mmap(2)system call is used to map it into the address space of the TIBCO BusinessEvents® Extremeprocesses.

In a development environment, this is very convenient. Many developers may share a machine,and the operating system will only allocate memory as it is actually utilized in the shared memoryfiles. Cleanup of stranded deployments (where the processes are gone but the shared memoryfile remains) may be as simple as removing file system directories.

A performance disadvantage when using mmap'd files for shared memory is that the operatingsystem will spend cycles writing the memory image of the file to disk. As the size of the sharedmemory file and the amount of shared memory accessed by the application increases, the oper-ating system will spend more and time writing the contents to disk.

TIBCO BusinessEvents® Extreme is not supported running from a networked filesystem (e.g. NFS).

• System V Shared memory

TIBCO BusinessEvents® Extreme also supports using System V Shared Memory for its sharedmemory.

An advantage of using System V Shared Memory is that the operating system does not spendany cycles attempting to write the memory to disk.

Another advantage is that the memory is allocated all at once by the operating system and cannotbe swapped. In some cases this also allows the operating system to allocate the physical memorycontiguously and use the CPU's TLB (translation lookaside buffer) more efficiently.

18

Chapter 4. Tuning

To reclaim System V Shared Memory the TIBCO BusinessEvents® Extreme node mustbe stopped and removed using the administrator remove node command. Theshared memory is not released by removing the node deployment directory.

CachingManaged objects that are backed by the data grid support caching of a subset of the object data inshared memory. The cache size should be set so that it is large enough to allow a working set ofobjects in shared memory. This will avoid having to constantly refresh object data from the datagrid, which will negatively impact performance. TIBCO BusinessEvents® Extreme uses a LRU(least recently used) algorithm to evict objects from shared memory, so objects that are accessedmost often will remain cached in shared memory.

SwappingThe machine where a TIBCO BusinessEvents® Extreme node runs should always have enoughavailable physical memory so that no swapping occurs on the system. TIBCO BusinessEvents®Extreme gains much of its performance by caching as much as possible in memory. If this memorybecomes swapped, or simple paged out, the cost to access it increases by many orders of magnitude.

On Linux one can see if swapping has occurred using the following command:

$ /usr/bin/free total used free shared buffers cachedMem: 3354568 3102912 251656 0 140068 1343832-/+ buffers/cache: 1619012 1735556Swap: 6385796 0 6385796

On Solaris, the following command can be used:

$ /etc/swap -lswapfile dev swaplo blocks free/dev/dsk/c1t0d0s1 118,9 16 16780208 16780208

Multi-nodeAn TIBCO BusinessEvents® Extreme application can be, and often is, run on a single node. WithHigh-availability and Distribution features, TIBCO BusinessEvents® Extreme can run distributedapplications across multiple nodes. From an operational point of view, there are very few benefitsfrom running multiple nodes on a single machine. This document recommends and assumes thateach node will be run on its own machine.

When an application reaches its throughput limit on a single node, additional performance can begained by adding multiple nodes. This is called horizontal scaling. For an application that is notdesigned to be distributed, this often poses a problem. Sometimes this can be addressed by addinga routing device outside of the nodes. But sometimes this cannot be addressed without rewritingthe application.

A distributed TIBCO BusinessEvents® Extreme application can be spread across an arbitrarynumber of nodes at the High-availability data partition boundary. If the active node for a set of par-titions has reached throughput saturation, one or more of the partitions may be migrated to othernodes.

19

Swapping

Analyzing DeadlocksWhen TIBCO BusinessEvents® Extreme detects a deadlock a detailed trace is sent to the log filesshowing which resource deadlocked, which transactions were involved in the deadlock, which re-sources they had locked, and which resource they were blocked waiting for. Additionally a stacktrace is logged showing where in the application the deadlock occurred.

Single Node DeadlocksLock order deadlock. The program below will generate a single transaction lock orderingdeadlock between two threads, running in a single JVM, in a single node.

// $Revision: 1.1.2.1 $package com.kabira.snippets.tuning;

import com.kabira.platform.Transaction;import com.kabira.platform.annotation.Managed;

/** * Deadlock Example from the ActiveSpaces Transactions Tuning Guide. * <p> * <h2> Target Nodes</h2> * <ul> * <li> <b>domainnode</b> = A * </ul> */public class Deadlock{

static MyManagedObject object1; static MyManagedObject object2;

public static void main(String[] args) throws InterruptedException { // // Create a pair of Managed objects. // new Transaction("Create Objects") {

@Override public void run() { object1 = new MyManagedObject(); object2 = new MyManagedObject(); } }.execute();

// // Create a pair of transaction classes to lock them. // Giving the object parameters in reverse order will // cause two different locking orders, resulting in a deadlock. // Deadlocker deadlocker1 = new Deadlocker(object1, object2); Deadlocker deadlocker2 = new Deadlocker(object2, object1);

// // Run them in separate threads until a deadlock is seen. // while ((deadlocker1.getNumberDeadlocks() == 0) && (deadlocker2.getNumberDeadlocks() == 0)) {

20

Chapter 4. Tuning

MyThread thread1 = new MyThread(deadlocker1); MyThread thread2 = new MyThread(deadlocker2);

thread1.start(); thread2.start();

thread1.join(); thread2.join(); } }

@Managed static class MyManagedObject {

int value; }

static class MyThread extends Thread {

private Deadlocker m_deadlocker;

MyThread(Deadlocker deadlocker) { m_deadlocker = deadlocker; }

@Override public void run() { m_deadlocker.execute(); } }

static class Deadlocker extends Transaction {

private final MyManagedObject m_object1; private final MyManagedObject m_object2;

Deadlocker(MyManagedObject object1, MyManagedObject object2) { m_object1 = object1; m_object2 = object2; }

@Override public void run() {

// // This will take a transaction read lock on the first object. // int value = m_object1.value;

// // Wait a while to maximize the possibility of contention. // blockForAMoment();

// // This will take a transaction write lock on the second object. // m_object2.value = 42;

//

21

Analyzing Deadlocks

// Wait a while to maximize the possibility of contention. // blockForAMoment(); }

private void blockForAMoment() { try { Thread.sleep(500); } catch (InterruptedException ex) { } } }}

The log file for the A node will contain a trace similar to the following:

2010-05-19 10:58:36.190955|OBJ|WARN |19821|osproxy.cpp(857)|deadlock detected in transaction id 156:2

Transaction 156:2 deadlocked attempting to write lock com.kabira.snippets.tuning.Deadlock$MyManagedObject:1 (890258:6145144:112456:1 offset 64534360) locks read { 155:2 } which is held by transaction id 155:2

Callstack for transaction 156:2: deadlock on com.kabira.snippets.tuning.Deadlock$MyManagedObject:1

Objects currently locked in transaction id 156:2 com.kabira.snippets.tuning.Deadlock$MyManagedObject:2 (890258:6145144:112456:2 offset 64099448) read lock

Transaction 155:2 is blocked waiting for a write lock on com.kabira.snippets.tuning.Deadlock$MyManagedObject:2 (890258:6145144:112456:2 offset 64099448) locks read { 156:2 } which is held by transaction 156:2

Objects currently locked in transaction 155:2 com.kabira.snippets.tuning.Deadlock$MyManagedObject:1 (890258:6145144:112456:1 offset 64534360) read lock

2010-05-19 10:58:36.192736|JAV|WARN |19821|native_tran.cpp(373)|Deadlock detected in Java thread: com.kabira.ktvm.transaction.DeadlockError: at com.kabira.platform.ManagedObject.setInteger(Native Method) at com.kabira.snippets.tuning.Deadlock$Deadlocker.run(Deadlock.java:95) at com.kabira.platform.Transaction.execute(Transaction.java:310) at com.kabira.snippets.tuning.Deadlock$MyThread.run(Deadlock.java:65)

Looking at this section by section we see:

2010-05-19 10:58:36.190955|OBJ|WARN |19821|osproxy.cpp(857)|deadlock detected in transaction id 156:2

This line is the beginning of the deadlock trace, and shows which transaction id (156:2) detectedthe deadlock. The transaction which detects the deadlock rolls back and retries.

Transaction 156:2 deadlocked attempting to write lock com.kabira.snippets.tuning.Deadlock$MyManagedObject:1 (890258:6145144:112456:1 offset 64534360)

22

Chapter 4. Tuning

locks read { 155:2 } which is held by transaction id 155:2

Callstack for transaction 156:2: deadlock on com.kabira.snippets.tuning.Deadlock$MyManagedObject:1

Objects currently locked in transaction id 156:2 com.kabira.snippets.tuning.Deadlock$MyManagedObject:2 (890258:6145144:112456:2 offset 64099448) read lock

This section shows the transaction which detected the deadlock. What type of lock it was trying totake, which resource it was trying to lock, what other transaction is holding a lock which causedcontributed to this deadlock, all transaction locks that it was already holding at the time.

Transaction 155:2 is blocked waiting for a write lock on com.kabira.snippets.tuning.Deadlock$MyManagedObject:2 (890258:6145144:112456:2 offset 64099448) locks read { 156:2 } which is held by transaction 156:2

Objects currently locked in transaction 155:2 com.kabira.snippets.tuning.Deadlock$MyManagedObject:1 (890258:6145144:112456:1 offset 64534360) read lock

This section shows the other involved transaction. What type of lock it is blocked waiting to take,which resource it is trying to lock, and what other transaction locks it is currently holding.

2010-05-19 10:58:36.192736|JAV|WARN |19821|native_tran.cpp(373)|Deadlock detected in Java thread: com.kabira.ktvm.transaction.DeadlockError: at com.kabira.platform.ManagedObject.setInteger(Native Method) at com.kabira.snippets.tuning.Deadlock$Deadlocker.run(Deadlock.java:95) at com.kabira.platform.Transaction.execute(Transaction.java:310) at com.kabira.snippets.tuning.Deadlock$MyThread.run(Deadlock.java:65)

This section shows the call stack of the transaction which detected the deadlock.

Promotion deadlock. Lock promotion is when a transaction currently holding a read lock onan object attempts to acquire a write lock on the same object (i.e. Promoting the read lock to a writelock). If blocking for this write lock would result in deadlock, it is called a promotion deadlock.

The program below will generate a single promotion deadlock between two threads, running in asingle JVM, in a single node.


import com.kabira.platform.Transaction;import com.kabira.platform.annotation.Managed;

/** * Promotion deadlock Example from the ActiveSpaces Transactions Tuning Guide. * <p> * <h2> Target Nodes</h2> * <ul> * <li> <b>domainnode</b> = A * </ul> */public class PromotionDeadlock{

static MyManagedObject targetObject;

public static void main(String[] args) throws InterruptedException

23

Analyzing Deadlocks

{ // // Create a Managed objects. // new Transaction("Create Objects") { @Override public void run() { targetObject = new MyManagedObject(); } }.execute();

// // Create a pair of transaction classes that will both // promote lock the Managed object, resulting in a // promotion deadlock. // Deadlocker deadlocker1 = new Deadlocker(targetObject); Deadlocker deadlocker2 = new Deadlocker(targetObject);

// // Run them in separate threads until a deadlock is seen. // while ((deadlocker1.getNumberDeadlocks() == 0) && (deadlocker2.getNumberDeadlocks() == 0)) { MyThread thread1 = new MyThread(deadlocker1); MyThread thread2 = new MyThread(deadlocker2);

thread1.start(); thread2.start();

thread1.join(); thread2.join(); } }

@Managed static class MyManagedObject {

int value; }

static class MyThread extends Thread {

private Deadlocker m_deadlocker;

MyThread(Deadlocker deadlocker) { m_deadlocker = deadlocker; }

@Override public void run() { m_deadlocker.execute(); } }

static class Deadlocker extends Transaction {

private final MyManagedObject m_targetObject;

24

Chapter 4. Tuning

Deadlocker(MyManagedObject targetObject) { m_targetObject = targetObject; }

@Override public void run() {

// // This will take a transaction read lock on the object. // int value = m_targetObject.value;

// // Wait a while to maximize the possibility of contention. // blockForAMoment();

// // This will take a transaction write lock on the object // (promoting the read lock). // m_targetObject.value = 42;

// // Wait a while to maximize the possibility of contention. // blockForAMoment(); }

private void blockForAMoment() { try { Thread.sleep(500); } catch (InterruptedException ex) { } } }}

The trace messages are similar to those show in the previous section for a lock order deadlock,with the difference being that promotion deadlock will be mentioned:

06-01 08:45:57|runtime::Events::TraceWarning|5391|103:177|Warning|promotion deadlock detected in transaction id 130:2

Transaction 130:2 deadlocked attempting to promote (write lock) com.kabira.snippets.tuning.PromotionDeadlock$MyManagedObject:1 (890258:9384688:2001:1 offset 64698248) locks read { 133:2, 130:2 }, promote waiter { 133:2 } which is held by transaction id 133:2

Callstack for transaction 130:2: promotion deadlock on com.kabira.snippets.tuning.PromotionDeadlock$MyManagedObject:1

Objects currently locked in transaction id 130:2 com.kabira.snippets.tuning.PromotionDeadlock$MyManagedObject:1 (890258:9384688:2001:1 offset 64698248) read lock

Transaction 133:2 is blocked waiting for a promote lock on com.kabira.snippets.tuning.PromotionDeadlock$MyManagedObject:1 (890258:9384688:2001:1

25

Analyzing Deadlocks

offset 64698248) locks read { 133:2, 130:2 }, promote waiter { 133:2 } which is held by transaction 130:2

Objects currently locked in transaction 133:2 com.kabira.snippets.tuning.PromotionDeadlock$MyManagedObject:1 (890258:9384688:2001:1 offset 64698248) read lock

06-01 08:45:57|runtime::Events::TraceWarning|5391|103:179|Warning|Deadlock detected in Java thread: com.kabira.ktvm.transaction.DeadlockError: at com.kabira.platform.ManagedObject.setInteger(Native Method) at com.kabira.snippets.tuning.PromotionDeadlock$Deadlocker.run(PromotionDeadlock.java:92)

at com.kabira.platform.Transaction.execute(Transaction.java:309) at com.kabira.snippets.tuning.PromotionDeadlock$MyThread.run(PromotionDeadlock.java:63)

Complex deadlock. The previous examples showed simple deadlocks, occurring between twotransactions. More complex deadlocks are possible involving more than two transactions. For ex-ample, transaction 1 deadlocks trying to acquire a lock on an object held by transaction 2 who isblocked waiting on a object held by transaction 3.

To aid in analyzing complex deadlocks the following will be found in the trace messages:

For each contended object, a display of the locks is included, including any promotion waiters.

If the runtime detects that a deadlock happens due to a read lock being blocked, it includes thetransaction blocked waiting for the promotion.

Distributed deadlocksSingle node deadlocks are bad for performance because they are a source of contention, leading tolower throughput, higher latency and higher CPU cost. But the deadlocks are detected immediately,because each node has a built in transaction lock manager.

Distributed deadlocks are extremely bad for performance because they use a timeout mechanismfor deadlock detection. The default setting for this timeout is 60 seconds in a production build.

The program below will generate a distributed transaction lock ordering deadlock between twotransactions running across multiple nodes.


import com.kabira.platform.Transaction;import com.kabira.platform.annotation.Managed;import com.kabira.platform.highavailability.PartitionManager;import com.kabira.platform.highavailability.PartitionManager.EnableAction;import com.kabira.platform.highavailability.PartitionMapper;import com.kabira.platform.highavailability.ReplicaNode;import com.kabira.platform.swbuiltin.EngineServices;import static com.kabira.platform.highavailability.ReplicaNode.ReplicationType.*;

/** * Distributed deadlock example from the ActiveSpaces Transactions Tuning Guide * <p> * <h2> Target Nodes</h2> * <ul> * <li> <b>domainname</b> = Development * </ul> * Note this this sample blocks on B and C nodes, * and needs to be explicitly stopped. */

26

Chapter 4. Tuning

public class DistributedDeadlock{ private static TestObject object1; private static TestObject object2;

public static void main(String[] args) throws InterruptedException { // // Block all but the A node. // new NodeChecker().blockAllButA();

// // Define the partitions to be used by this snippet // new PartitionCreator().createPartitions();

// // Create a pair of objects, one active on node B, // and the other active on node C. // new Transaction("Create Objects") { @Override public void run() { object1 = new TestObject(); object2 = new TestObject();

// // For each distributed object, assign it a // reference to the other. // object1.otherObject = object2; object2.otherObject = object1; } }.execute();

// // Create a pair of objects, one active on node B, // and the other active on node C. // new Transaction("Spawn Deadlockers") { @Override public void run() { // // Ask them each to spawn a Deadlocker thread. // This should execute on node B for one of them // and node C for the other. // object1.spawnDeadlocker(); object2.spawnDeadlocker(); } }.execute();

// // Now block main in the A node to keep the JVM from exiting. // new NodeChecker().block(); }

private static class PartitionCreator { void createPartitions() {

27

Analyzing Deadlocks

new Transaction("Partition Definition") { @Override protected void run() throws Rollback { // // Set up the node lists - notice that the odd node list // has node B as the active node, while the even // node list has node C as the active node. // ReplicaNode [] evenReplicaList = new ReplicaNode [] { new ReplicaNode("C", SYNCHRONOUS), new ReplicaNode("A", SYNCHRONOUS) }; ReplicaNode [] oddReplicaList = new ReplicaNode [] { new ReplicaNode("B", SYNCHRONOUS), new ReplicaNode("A", SYNCHRONOUS) };

// // Define two partitions // PartitionManager.definePartition("Even", null, "B", evenReplicaList);

PartitionManager.definePartition("Odd", null, "C", oddReplicaList);

// // Enable the partitions // PartitionManager.enablePartitions( EnableAction.JOIN_CLUSTER_PURGE);

// // Install the partition mapper // PartitionManager.setMapper( TestObject.class, new AssignPartitions()); } }.execute(); } }

// // Partition mapper that maps objects to either Even or Odd // private static class AssignPartitions extends PartitionMapper {

@Override public String getPartition(Object obj) { this.m_count++; String partition = "Even";

if ((this.m_count % 2) == 1) { partition = "Odd"; }

return partition; } private Integer m_count = 0; }

@Managed

28

Chapter 4. Tuning

private static class TestObject { TestObject otherObject; String data;

public void lockObjects() { // Transaction.writeLockObject(this); this.doWork();

// // Delay longer on the B node to try to force the deadlock // to occur on the C. Otherwise, both sides could see // deadlocks at the same time, making the log files less clear // for this snippet. // if (EngineServices.getNodeName().equals("B")) { block(10000); } else { block(500); }

// Transaction.writeLockObject(otherObject); otherObject.doWork();

block(500); }

public void spawnDeadlocker() { new DeadlockThread(this).start(); }

private void block(int milliseconds) { try { Thread.sleep(milliseconds); } catch (InterruptedException ex) { } }

private void doWork() { data = "work"; } }

private static class DeadlockThread extends Thread {

private Transaction m_deadlockTransaction;

DeadlockThread(TestObject object) { m_deadlockTransaction = new DeadlockTransaction(object); }

@Override public void run() { while (true) { if (m_deadlockTransaction.execute()

29

Analyzing Deadlocks

== Transaction.Result.ROLLBACK) { return; } } } }

private static class DeadlockTransaction extends Transaction {

private final TestObject m_object;

DeadlockTransaction(TestObject object) { m_object = object; }

@Override public void run() throws Rollback { if (getNumberDeadlocks() != 0) { System.out.println("A deadlock has been seen, " + "you may now stop the distributed application"); throw new Transaction.Rollback(); } m_object.lockObjects(); } }

private static class NodeChecker extends Transaction { String nodeName;

// // If we are not the A node, block here forever // void blockAllButA() { execute(); while (!nodeName.equals("A")) { block(); } }

@Override protected void run() { nodeName = EngineServices.getNodeName(); }

public void block() { while (true) { try { Thread.sleep(500); } catch (InterruptedException ex) { } } } }}

30

Chapter 4. Tuning

The deadlock information may now be spread across multiple log files. In this case, both the B andC node log files. The B log file will look similar to:

04-21 11:32:08|runtime::Events::TraceWarning|4603|115:1468|Warning|Global transaction 67:230:2:2485325816672 deadlocked on node BLocal objects locked: com.kabira.snippets.tuning.DistributedDeadlock$TestObject:2 (890258:8376920:2485328244931:2 offset 65668456) write lock

04-21 11:32:08|runtime::Events::TraceWarning|4603|115:1472|Warning|Global transaction id: 66:230:1:2485325880423Transaction 129:3 is blocked waiting for a nolock on com.kabira.snippets.tuning.DistributedDeadlock$TestObject:2 (890258:8376920:2485328244931:2 offset 65668456) locks write { 'com.kabira.snippets.tuning.DistributedDeadlock$DeadlockTransaction'[230:2] } Objects currently locked in transaction 129:3

04-21 11:32:08|runtime::Events::TraceWarning|12372|115:1476|Warning|Deadlock detected in Java thread: com.kabira.ktvm.transaction.DeadlockError: at com.kabira.platform.ManagedObject.sendTwoWay(Native Method) at com.kabira.snippets.tuning.DistributedDeadlock$TestObject.doWork(DistributedDeadlock.java)

at com.kabira.snippets.tuning.DistributedDeadlock$TestObject.$lockObjectsImpl(DistributedDeadlock.java:178)

at com.kabira.snippets.tuning.DistributedDeadlock$TestObject.lockObjects(DistributedDeadlock.java)

at com.kabira.snippets.tuning.DistributedDeadlock$DeadlockTransaction.run(DistributedDeadlock.java:247)

at com.kabira.platform.Transaction.execute(Transaction.java:303) at com.kabira.snippets.tuning.DistributedDeadlock$DeadlockThread.run(DistributedDeadlock.java:219)

The C node log file will look similar to the following:

04-21 11:32:08|runtime::Events::TraceWarning|5222|116:1454|Warning|distributed deadlock detected in transaction id 131:3

Transaction 131:3 deadlocked attempting to lock wait com.kabira.snippets.tuning.DistributedDeadlock$TestObject:1 (890258:8376920:2485328244931:1 offset 57043208) locks write { 'com.kabira.snippets.tuning.DistributedDeadlock$DeadlockTransaction'[230:1] } Callstack for transaction 131:3: distributed deadlock on com.kabira.snippets.tuning.DistributedDeadlock$TestObject:1

Objects currently locked in transaction id 131:3

Object is write locked in transaction 'com.kabira.snippets.tuning.DistributedDeadlock$DeadlockTransaction'[230:1] Callstack for transaction 230:1: dispatch calling distribution dispatcher on com.kabira.snippets.tuning.DistributedDeadlock$TestObject:2 distribution calling com.kabira.snippets.tuning.DistributedDeadlock$TestObject.$doWorkImpl()V on com.kabira.snippets.tuning.DistributedDeadlock$TestObject:2

Objects currently locked in transaction 'com.kabira.snippets.tuning.DistributedDeadlock$DeadlockTransaction'[230:1]

31

Analyzing Deadlocks

com.kabira.snippets.tuning.DistributedDeadlock$TestObject:1 (890258:8376920:2485328244931:1 offset 57043208) write lock

04-21 11:32:08|runtime::Events::TraceWarning|5222|116:1456|Warning|Deadlocked waiting for object reference '890258:8376920:2485328244931:1'

Looking section by section through the B log file we see

04-21 11:32:08|runtime::Events::TraceWarning|4603|115:1468|Warning|Global transaction 67:230:2:2485325816672 deadlocked on node BLocal objects locked: com.kabira.snippets.tuning.DistributedDeadlock$TestObject:2 (890258:8376920:2485328244931:2 offset 65668456) write lock

04-21 11:32:08|runtime::Events::TraceWarning|4603|115:1472|Warning|Global transaction id: 66:230:1:2485325880423Transaction 129:3 is blocked waiting for a nolock on com.kabira.snippets.tuning.DistributedDeadlock$TestObject:2 (890258:8376920:2485328244931:2 offset 65668456) locks write { 'com.kabira.snippets.tuning.DistributedDeadlock$DeadlockTransaction'[230:2] } Objects currently locked in transaction 129:3

This is a list of the global transactions involved in the deadlock and what objects they had lockedat the time of the timeout.

04-21 11:32:08|runtime::Events::TraceWarning|12372|115:1476|Warning|Deadlock detected in Java thread: com.kabira.ktvm.transaction.DeadlockError: at com.kabira.platform.ManagedObject.sendTwoWay(Native Method) at com.kabira.snippets.tuning.DistributedDeadlock$TestObject.doWork(DistributedDeadlock.java)

at com.kabira.snippets.tuning.DistributedDeadlock$TestObject.$lockObjectsImpl(DistributedDeadlock.java:178)

at com.kabira.snippets.tuning.DistributedDeadlock$TestObject.lockObjects(DistributedDeadlock.java)

at com.kabira.snippets.tuning.DistributedDeadlock$DeadlockTransaction.run(DistributedDeadlock.java:247)

at com.kabira.platform.Transaction.execute(Transaction.java:303) at com.kabira.snippets.tuning.DistributedDeadlock$DeadlockThread.run(DistributedDeadlock.java:219)

This is a stack back trace of the transaction which timed out waiting for a lock.

In the C log file we see the following:

04-21 11:32:08|runtime::Events::TraceWarning|5222|116:1454|Warning|distributed deadlock detected in transaction id 131:3

Transaction 131:3 deadlocked attempting to lock wait com.kabira.snippets.tuning.DistributedDeadlock$TestObject:1 (890258:8376920:2485328244931:1 offset 57043208) locks write { 'com.kabira.snippets.tuning.DistributedDeadlock$DeadlockTransaction'[230:1] } Callstack for transaction 131:3: distributed deadlock on com.kabira.snippets.tuning.DistributedDeadlock$TestObject:1

Objects currently locked in transaction id 131:3

32

Chapter 4. Tuning

Object is write locked in transaction 'com.kabira.snippets.tuning.DistributedDeadlock$DeadlockTransaction'[230:1] Callstack for transaction 230:1: dispatch calling distribution dispatcher on com.kabira.snippets.tuning.DistributedDeadlock$TestObject:2 distribution calling com.kabira.snippets.tuning.DistributedDeadlock$TestObject.$doWorkImpl()V on com.kabira.snippets.tuning.DistributedDeadlock$TestObject:2

Objects currently locked in transaction 'com.kabira.snippets.tuning.DistributedDeadlock$DeadlockTransaction'[230:1] com.kabira.snippets.tuning.DistributedDeadlock$TestObject:1 (890258:8376920:2485328244931:1 offset 57043208) write lock

This tells us that a distributed deadlock has occurred attempting to lock a TestObject:1, which collidedwith a write lock held by transaction 230:1.

33

Analyzing Deadlocks

5Performance monitoring

This chapter describes the tools and approach to monitoring TIBCO BusinessEvents® Extremeapplications.

JVMStandard Java tools may be used to monitor the JVM.

• Visual VM (https://visualvm.dev.java.net).

35

https://visualvm.dev.java.net

Figure 5.1. Visual VM

• JConsole (http://openjdk.java.net/tools/svc/jconsole).

Figure 5.2. JConsole

36

Chapter 5. Performance monitoring

http://openjdk.java.net/tools/svc/jconsole

Graphical monitoring toolsGraphical display of application transaction statistics, application business state machines, CPUutilization, and shared memory utilization is available from the Node panel of TIBCO BusinessEv-ents® Extreme Administrator. Click the Start monitors button:

Figure 5.3. Starting system monitors

This will open window which allows the selection of which graphical monitors to start.

Figure 5.4. Select system monitors to start

TransactionsThe runtime continually collects a node-wide view of the aggregate transaction rate per second, andthe average execution time. The Transaction Monitor in TIBCO BusinessEvents® Extreme Admin-istrator is used to display both the transaction rate and the average transaction execution time.

The transaction rate is a count of all transactions executing over a period of time.

The transaction execution time is an average of the execution time of all transactions over a periodof time. The transaction execution time is the total time from when a transaction is started to whenit commits or rolls-back. This time includes all, or a sub-set, of the following depending on thetransaction:

• application code execution time

• transaction lock acquisition and logging time

37

Graphical monitoring tools

• network latency to remote nodes for distributed transactions

• replication latency to remote nodes for highly-available objects

• transaction commit or rollback time

The Transaction Monitor has these controls:

• Time vs. Rate - Display transaction rate or execution time.

• Sample Duration - Set the sample duration in seconds. This controls the interval between queryingthe server for new data for the display.

• Pause or Start - Pause or re-start transaction monitoring.

Figure 5.5 shows an example of displaying the node transaction rate.

Figure 5.5. Node transaction rate

38


When the Transaction Monitor is started on the domain, the results show the domain-wide aggregateapplication transaction rate. Figure 5.6 shows an example of displaying the domain-wide transactionrate.

Figure 5.6. Domain-wide transaction rate

The Transaction Monitor displays the following information when displaying transaction rates:

• Network address - the address of the node or domain manager being monitored.

• Minimum - the minimum transaction rate since the monitor was started.

• Maximum - the maximum transaction rate since the monitor was started.

• Transactions / Second - a graphical display of the number of transactions per secondplotted against time.

Selecting Time using the Time vs. Rate button on the Transaction Monitor displays the executiontime for transactions on a single node or aggregated for all nodes in the domain. Figure 5.7 shows

39


an example of transaction execution time for a node and Figure 5.8 shows the same thing for a do-main.

Figure 5.7. Node transaction execution time

40


Figure 5.8. Domain-wide transaction execution time

The Transaction Monitor displays the following information when displaying transaction executiontime:

• Network address - the address of the node or domain manager being monitored.

• Execution Time - a graphical display of the minimum, maximum, and average execution timein milliseconds plotted against time. The different values are viewed by selecting the monitor andhovering a mouse pointer over the plot.

Business state machinesDocumentation for the graphical business state machine monitor can be found in the adminstrationsection of the site documentation for the Business State Machine component:

[http://downloads.fluency.kabira.com/sites/businessstatemachine/administration]

41


http://downloads.fluency.kabira.com/sites/businessstatemachine/administration


CPU monitor

Figure 5.9. CPU monitor

42


Shared memory monitor

Figure 5.10. Shared memory monitor

The statistics tabThe TIBCO BusinessEvents® Extreme Administrator offers access a variety of statistics via thestatistics tab of the Node panel. Application, node, and system level reports are available andare selected via the pull down menu:

43

The statistics tab

Figure 5.11. Shared memory monitor

Some of these statistics are collected automatically by the runtime. For these, the Enable andDisable buttons will not be greyed out and not clickable. Clicking the Display button will showthe current values.

Other statistics need to be enabled in order to activate data collection. Generally this is because thecollection of these statistics imposes some performance, memory usage, or disk usage penalty uponthe system. For these, the Enable button must be clicked, and the desired amount of time shouldbe waited before clicking the Display button to show the collected statistics. It is also good practiceto disable the statistic collection, by pressing the Disable button, before displaying the report.This restores the system to its previous performance level, and also keeps the reporting itself fromshowing up in the measurement.

Some statistics support clearing. Those statistics may be cleared at any time by pressing the Clearbutton. For statistics that do not support clearing the Clear button will be greyed out and notclickable.

After a report is displayed, it may be printed by clicking the Print button.

Application monitoringPer Class Transaction StatisticsFiner grained transaction statistics, tied to each class which implements com.kabira.platform.Trans-action are also available. The collection of these statistics imposes a slight performance penalty andis not enabled by default.

To collect these statistics, select Transaction in the pull down menu in the statistics panel:

44


Figure 5.12. Transaction class statistics

Next enable the collection of these statistics by clicking the Enable button:

Figure 5.13. Enable transaction class statistics collection

Allow the data to collect for several seconds or more, and then disable statistics collection byclicking the Disable button. Disabling the statistics collection removes the slight performancepenalty and allows the system to run at full speed.

Disabling the statistics collection does not remove the collected statistics.

Display the collected statistics by clicking the Display button:

45

Application monitoring

Figure 5.14. Disable and display transaction class statistics

For each transaction class which executed during the period when statistics collection was enabled,the report shows the number of executions, the average, minimum and maximum execution times(in milliseconds), and either the name of the transaction, if provided to the Transaction constructor,or the name of the class, if the empty constructor was used.

The following snippet, which demonstrates both a named, and an unnamed transaction, was usedto generate the system load:


import com.kabira.platform.Transaction;

/** * SimpleTransactions example from the ActiveSpaces Transactions Tuning Guide * for generating transaction statistics. * * <p> * <h2> Target Nodes</h2> * <ul> * <li> <b>domainnode</b> = A * </ul> */

public class SimpleTransactions{ public static void main(String[] args) { while (true) { // This is a named transaction new Transaction("Example Transaction") { @Override public void run() { try { Thread.sleep(500); } catch (InterruptedException ex) { } } }.execute();

// This is an unnamed transaction

46


new Transaction() { @Override public void run() { try { Thread.sleep(500); } catch (InterruptedException ex) { } } }.execute(); } }}

The currently collected statistics may be cleared (reset) at any time by clicking the Clear button.

Transaction locking and contentionA report showing transaction locking is available by selecting Transaction Locking in the pull downmenu in the statistics panel:

Figure 5.15. Transaction locking statistics

The collection of transaction locking statistics imposes a slight performance penalty and is not enabledby default. After having selected the Transaction Locking, enable the collection of these statisticsby clicking the Enable button:

47


Figure 5.16. Enable transaction locking statistics collection




Figure 5.17. Disable and display transaction locking statistics

The columns of each row are:

• Type Name - the class for the Managed object.

• Read Locks - the number of transaction read locks that were taken on this object instance duringthe period of statistics collection.

• Write Locks - the number of transaction write locks that were taken on this object instance duringthe period of statistics collection.

48


• Contentions - the number of transaction locks on this object that encountered transaction lockcontention during the period of statistics collection.

• Deadlocks - the number of deadlocks other than promotion deadlocks that occurred attemptingto transaction lock this object during the period of statistics collection.

• Promotion Deadlocks - the number of deadlocks that occurred attempting transaction lock pro-motion (from read to write) on this object during the period of statistics collection.

• Average Microseconds - the average time in microseconds that it took to acquire a transactionlock on this object during the period of statistics collection.

• Minimum Microseconds - the minimum time in microseconds that it took to acquire a transactionlock on this object during the period of statistics collection.

• Maximum Microseconds - the maximum time in microseconds that it took to acquire a transactionlock on this object during the period of statistics collection.

Transaction locking statistics should only be collected by one user at a time. Multipleconcurrent collection will cause invalid data to be reported.

Business State MachineDocumentation for the Business State Machine report can be found in the adminstration section ofthe site documentation for the Business State Machine component:

[http://downloads.fluency.kabira.com/sites/businessstatemachine/administration]

Object ReportA report showing Managed objects statistics.

Figure 5.18. Managed object report

There is a row in the report for each Managed object type that has any non-0 data during since thedata was last cleared. The columns in the report are:

• Class - the class name.

• Cardinality - the number of instances currently in shared memory.

• Creates - the number of instances created since the data was last cleared.

• Destroys - the number of instances destroyed since the data was last cleared.

49




• Updates - the number of times modifications to instances were written to other nodes or the DataGrid since the data was last cleared.

• Refreshes - the number of times instances were pulled from remote nodes or the Data Grid sincethe data was last cleared.

• Flushes - the number of instances flushed from shared memory since the data was last cleared.

• Update Latency - the average, minimum, and maximum latencies, in milliseconds, for updates,since the data was last cleared.

• Refresh Latency - the average, minimum, and maximum latencies, in milliseconds, for refreshes,since the data was last cleared.

The Creates, Destroys, Updates, Refreshes, and Flushes counters are non-transactional. They rep-resent the number of times that these operations occurred without respect to the outcome of theircontaining transactions. For example, if one create was done, but a deadlock occurred, and thetransaction rolled back, and then replayed successfully, the Creates count would be 2.

Query ReportA report showing query counts for Managed objects.

Figure 5.19. Query statistics

There is a row in the report for each Managed object type that has been accessed via ManagedOb-ject.extent() or ManagedObject.cardinality(), and for each key type that has beenaccessed since the data was last cleared. The columns in the report are:

• Select - the number of times the Key was accessed during the period measured.

• Iteration - the number of times ManagedObject.extent() was called for the type during theperiod measured.

• Cardinality - the number of times ManagedObject.cardinality() was call for the typeduring the period measured.

• Class/Key Name - the type name (Extent) or the Key name.

50


DeadlocksWhen TIBCO BusinessEvents® Extreme detects a deadlock, detailed information is sent to the logfiles. See the Analyzing Deadlocks section of the Chapter 4 for more information on interpretingthis log data.

In addition to descriptive logging of each deadlock encountered, there are several statistics maintainedthat can been seen from selecting the Deadlock statistic in the statistics pull down menu, andclicking the Display button:

• Number of Deadlocks - the number deadlocks seen since the last time the statistics were cleared.

• Number of Deadlock Backoffs - the number of times an extra wait was inserted before a deadlockretry. For a given execution of a transaction, the first deadlock will be immediately aborted andretried. The next deadlock will incur a 100 millisecond wait time before it is retried. For eachsubsequent deadlock (in the same execution of a single transaction) the wait time will be doubled.

• Minimum Backoff Time (milliseconds) - the minimum amount of time, in milliseconds, a transactionspent waiting before retrying.

• Maximum Backoff Time (milliseconds) - the maximum amount of time, in milliseconds, a trans-action spent waiting before retrying.

• Average Backoff Time (milliseconds) - the average amount of time, in milliseconds, a singletransaction spent waiting before retrying.

These statistics are always collected, and may be cleared at any time by clicking the Clear button.

All deadlocks are inherently performance hostile and should be avoided by modifying theapplication to avoid the deadlock condition.

Node monitoringA variety of reports are available within the statistics panel which apply to an entire node.

Shared memory usageShow the current shared memory utilization within the node.

51

Node monitoring

Figure 5.20. Shared memory usage

• Shared Memory Size - The total size of the shared memory, in bytes.

• % Utilized - Current percentage of shared memory that is in use.

• Throttle State - Historical list, in reverse order of memory throttling state changes. Seecom.kabira.platform.swbuiltin.EngineServices.throttle() for details aboutmemory throttling.

• Time - The time of the memory throttling state change, or of the last check of the memorythrottling state.

Shared memory mutexA report showing shared memory system mutex locking is available by selecting Shared MemoryMutex in the pull down menu in the statistics panel, and enabling collection by clicking the Enablebutton. The collection of mutex locking statistics imposes a slight performance penalty and is notenabled by default.

Figure 5.21. Shared memory mutex enable

52





Figure 5.22. Shared memory mutex display

• Read Locks - The number of times the lock was locked for shared read access since the last clearof the statistic.

• Write Locks - The number of times the lock was exclusively locked since the last clear of thestatistic.

• Contentions - The number of times that acquiring either a read or a write lock encountered con-tention since the last clear of the statistic.

• avg usecs - The average number of microseconds taken to acquire the lock since the last clear ofthe statistic.

• min usecs - The minimum number of microseconds taken to acquire the lock since the last clearof the statistic.

• max usecs - The maximum number of microseconds taken to acquire the lock since the last clearof the statistic.

• name - A label associating a particular mutex with its owner or function in the system.

Local mutexA report showing process local mutex locking is available by selecting Local Mutex in the pull downmenu in the statistics panel, and enabling collection by clicking the Enable button. The collectionof mutex locking statistics imposes a slight performance penalty and is not enabled by default.

53

Node monitoring

Figure 5.23. Process local mutex enable




Figure 5.24. Process local mutex display

• Read Locks - The number of times the lock was locked for shared read access since the last clearof the statistic.

• Write Locks - The number of times the lock was exclusively locked since the last clear of thestatistic.

• Contentions - The number of times that acquiring either a read or a write lock encountered con-tention since the last clear of the statistic.

• avg usecs - The average number of microseconds taken to acquire the lock since the last clear ofthe statistic.

• min usecs - The minimum number of microseconds taken to acquire the lock since the last clearof the statistic.

• max usecs - The maximum number of microseconds taken to acquire the lock since the last clearof the statistic.

• name - A label associating a particular mutex with its owner or function in the system. The valueis prefixed with the name of the process or JVM which contains the mutex.

54


Shared Memory IPCThe runtime uses a shared memory based interprocess communication mechanism for a within-node RPC mechanism, and also for managing the life cycle of asynchronous method calls.

Figure 5.25. Shared Memory IPC

• Method Type - Synchronous or Asynchronous.

• Sent - The number of method invocations.

• Completed - The number of method invocations that have completed execution.

• Difference - The difference between the number of invocations, and the number of completedinvocations. For asynchronous methods, this can show queuing.

• Dropped - The number of queued method invocations which were dropped because the targetobject had been destroyed.

Partition ReportDisplay information about the High Availability Partitions currently defined in the node.

Figure 5.26. High Availability Partitions Report

• Name - The name of the partition.

• Role - Whether the partition active on this node, or is a replica.

• Cardinality - The number of objects currently in the partition.

55

Node monitoring

• Creates - The number of times since the last clear of the statistic, that an object was created inthis partition. This statistic is not transactional.

• Updates - The number of times since the last clear of the statistics, that an object was modifiedin this partition. This statistic is not transactional.

• Deletes - The number of times since the last clear of the statistics, that an object was deleted inthis partition. This statistic is not transactional.

• Remote Creates Discarded - The number of creates since the last clear of the statistics, that couldnot be sent to a remote node.

• Remote Updates Discarded - The number of updates since the last clear of the statistics, thatcould not be sent to a remote node.

• Remote Deletes Discarded - The number of deletes since the last clear of the statistics, that couldnot be sent to a remote node.

Shared Memory Allocator ReportGeneral details about the shared memory allocator can be found in the Allocator Summary report:

Figure 5.27. Shared Memory Allocator Summary

• Shared Memory Size - the size, in bytes, of the node's shared memory.

• Available Memory In Heap - the amount, in bytes, of shared memory that has not yet been usedby the node.

See the following Shared Memory Allocator Buckets Report section for details about the organiz-ation of the shared memory allocator.

• Available Memory On Freelists - the amount, in bytes, of node shared memory that has alreadybeen allocated and deallocated, and is now available on the freelist.

See the following Shared Memory Allocator Buckets Report section for details about the organiz-ation of the shared memory allocator.

• Total Memory In Use - the amount, in bytes, of shared memory that is currently in use on thenode.

• Percent Of Memory In Use - Total Memory In Use shown as a percentage of Shared MemorySize.

56


• Per Allocation Overhead (PRODUCTION) - The amount, in bytes, of shared memory allocationbookkeeping per allocation, for a node running with with a PRODUCTION build.

• Per Allocation Overhead (DEVELOPMENT) - The amount, in bytes, of shared memory allocationbookkeeping per allocation, for a node running with with a DEVELOPMENT build.

• Shared Memory Page Size - Not currently used.

• Largest Supported Allocation Size - The largest individual shared memory allocation size, in bytes,supported on this node.

Shared Memory Allocator Buckets ReportIn a freshly started node, all shared memory starts in the shared memory heap. When sharedmemory is first allocated, it taken from the heap. When shared memory is freed, it is put on afreelist, organized allocation size. Subsequent allocations will first attempt to find memory on thefreelist.

Detailed information about the current state of shared memory allocations and freelists may befound in the allocator buckets report:

Figure 5.28. Shared Memory Allocator Buckets Report

• Size - the number of bytes for this allocation bucket.

• Total - the number of allocations of this size that have been taken from the shared memory heap.

• In Use - the number of allocations of this size currently in use.

• Cached - the number of allocations of this size that are currently on the freelist.

• Cache Hits - the number of times an allocation request for this size was made and filled from thefreelist.

• Cache Misses - the number of times an allocation request for this size was filled from the sharedmemory heap.

57

Node monitoring

System Threads ReportDisplay information about system threads within the node.

Figure 5.29. System Threads Report

• Current Count - The number of threads that currently exist.

• Configured Minimum - The configured value for the minimum number of this type of systemthread.

• Configured Maximum - The configured value for the maximum number of this type of systemthread.

• Resource - The system resource associated with this thread.

• Consumer - The process or system component associated with this thread resource.

Files ReportDisplay the files in the node directory, and their sizes in bytes.

58


Figure 5.30. Files Report

Node Processes ReportDisplay a list of the application, node, and system processes running on the node.

Figure 5.31. Node Processes Report

• Build Type - Whether the process contains PRODUCTION binaries or DEVELOPMENT (debug)binaries.

• Contains JVM - Whether or not this process contains a Java VM.

• Process Name - The name of the process.

Distribution StatisticsState and configuration information about the TIBCO BusinessEvents® Extreme distribution layeris available with the following TIBCO BusinessEvents® Extreme Administrator console command:

59

Node monitoring

Figure 5.32. Distribution report

Performance statistics for distribution communication between nodes may be gathered and reportedwith the following sequence of TIBCO BusinessEvents® Extreme Administrator console commandswhich enable, report upon, and then disable the collection of distribution statistics:

Figure 5.33. Enable distribution statistics

60


Figure 5.34. Display distribution statistics

Figure 5.35. Disable distribution statistics

Distribution statistics may be cleared with the following TIBCO BusinessEvents® Extreme Admin-istrator console command:

61

Node monitoring

Figure 5.36. Clearing distribution statistics

JNI Cache StatisticsShow information about the per-thread JNI caching done by the runtime for each JVM in the node.

Figure 5.37. JNI Cache statistics

• Thread Type - Shows whether the thread is a Java thread or a runtime thread.

• % Current - Current number of threads that have cached JNI resources.

• % Allocations - Number of JNI cache resource allocations that have been done since the last timethis statistic was cleared.

• % Frees - Number of JNI cache resource deallocations that have been done since the last timethis statistic was cleared.

• % JVM - The JVM that the resources are being cached for.

System monitoringKernel informationShow information about the version of the operating system.

62


Figure 5.38. Kernel information

System information reportShow information about the system, including the number of CPUs and their speed, the amount ofphysical memory, and shared memory mapping.

Figure 5.39. System information

Virtual Memory ReportThe Virtual Memory Report directly captures the output of a platform specific tool. On Unix systemsthis is vmstat.

Because running this tool consumes a small amount of CPU and an amount of disk space that isproportional to how long the tool is run, it is not enabled by default. To enable, select VirtualMemory in the pull down menu in the statistics panel and click the Enable button:

63

System monitoring

Figure 5.40. Virtual Memory Report Enable

Allow the data to collect for the desired amount of time and then disable the collection by clickingthe Disable button.

Display the collected statistics by clicking the Display button.

Figure 5.41. Virtual Memory Report

Network Utilization ReportThe Network Utilization Report directly captures the output of a platform specific tool. On Unixsystems this is nicstat.

Because running this tool consumes a small amount of CPU and an amount of disk space that isproportional to how long the tool is run, it is not enabled by default. To enable, select Network inthe pull down menu in the statistics panel and click the Enable button:

64


Figure 5.42. Network Report Enable



Figure 5.43. Network Utilization Report

Disk Utilization ReportThe disk utilization report directly captures the output of a platform specific tool. On Unix systemsthis is iostat.

Because running this tool consumes a small amount of CPU and an amount of disk space that isproportional to how long the tool is run, it is not enabled by default. To enable, select Disk in thepull down menu in the statistics panel and click the Enable button:

65

System monitoring

Figure 5.44. Disk Report Enable



Figure 5.45. Disk Report

System Activity ReportThe System Activity Report directly captures the output of a platform specific tool. On Unix systemsthis is sar.

Because running this tool consumes a small amount of CPU and an amount of disk space that isproportional to how long the tool is run, it is not enabled by default. To enable, select SystemActivity Reporter in the pull down menu in the statistics panel and click the Enable button:

66


Figure 5.46. System Activity Report Enable



Figure 5.47. System Activity Report

Node Processes ReportDisplay a list of the application, node, and system processes running on the node.

67

System monitoring

Figure 5.48. Node Processes Report

• Build Type - Whether the process contains PRODUCTION binaries or DEVELOPMENT (debug)binaries.

• Contains JVM - Whether or not this process contains a Java VM.

• Process Name - The name of the process.

68


IndexAapproach, 5-6

overview, 1

Ccontention, 5

Ddata grid, 12data grid backed managed objects, 19deadlock, 20deployment, 17distribution, 15

Ffeatures

overview, 2

Gguidelines, 6

Hhigh-availability, 14

Iindexes, 14

Jjvm, 17, 35

Kkeys, 14

Llatency, 5

Mmanaged objects, 11monitoring

application, 44node, 51overview, 3system, 62transactions, 37

multi-node, 19

Ppath length, 5performance monitoring, 35-68

Sscaling

horizontal, 5vertical, 5

shared memory, 18caching, 19

Tthrough-put, 5TIBCO BusinessEvents® Extreme

JVM, 1transaction

execution time, 39rate, 37

transaction monitor, 37transactions, 12tuning, 17-20

overview, 2

Uusing TIBCO BusinessEvents® Extreme features, 11-15

69

tibco businessevents® extreme...about this book this guide describes performance tuning of tibco...

Documents