© JBoss Inc. 2005

Application and System Monitoring Using JBoss

Application Server

Nicholas WhiteheadRobert QuinnPatrick Urban

March 1, 2005

2

Agenda

• Holistic Application and System Performance Monitoring

• Challenges In Collecting and Analyzing Performance Data

• A Proposed Architecture for a Performance Management Data Platform

• An Implementation of the Platform• Questions (and tentative answers)

3

Performance Monitoring & Analysis

4

Performance Monitoring & Analysis

• Heterogeneous Monitoring Targets Network (LAN, WAN, Routers, Switches) Hosts (CPU, Memory, Disk, Threads) Various Operating Systems Application Server (JMX, Instrumentation,

Pooling) Middleware (Message Throughput) JVM (Memory, GC) Database (Exec Plans, Memory, Concurrency,

Locks) Files (ad nauseum) Applicaton Specific Structures

5

Challenges

• Phone Call: “System is slow” Why and what do you do about it ?

• Prevention: Where are your weak spots ?• Planning: How many servers do I need to

get us through the next n months ?• What is the relationship between system

load and: Number of users Day of the week Month of the year My astrological sign ?

6

So Many Monitor Points, So Little Time

7

Techniques For Performance Monitoring

• Commercial Enterprise Frameworks Expensive Not Always Complete

• Manual Data Collection & Merging No Real Time Vizualization Inflxexible High Maintenance

• Adhoc Monitoring No Cross Referencing Multiple Monitors Load Monitored System

8

A Cascading Performance Emergency

• Users Report System is slow.

Where’s Waldo ?

9

Anatomy of a Perfomance Hit

There He Is !

10

A Consolidated Platform For Peformance Data Acquisition

11

Key Attributes

• Central, globally available clearing house for performance and system status data.

• Scale to large numbers of data sources and client dashboards.

• Supply dashboards, reports and status summaries to dashboards.

• Provide platform for post processing acquired performance and status data.

• Monitor Proxy: Multiplex multiple monitor clients to one set of predictable and controllable data acquisition streams.

• Support Correlation of Heterogeneous Metrics

12

JMX – Perfect For The Job

• Extensible and Standardized Container for In-Memory State• Robust, Flexible API• Rich and Dynamic Meta Data• Organized Domain Structure• Built In Monitors• Attribute Change and other Notifications• Wide Support for 3rd Party Tools• Rich Lifecycle Control

13

Services

• Simple State: Holds the last reading for a specific metric. e.g. The last CPU utilization reading on webserver1.

• Data History Buffer: The last n readings of a specific metric. Shows flat, upward or downward trend.

• Persistence: Store performance data for reporting, historical analysis and forecasting.

• Charting: Real time data visualization.• Alerts: Alerting services for specific conditions,

monitor triggers or failed readings.• Boundary Tracking: Track high, low and average

values for performance statistics.• Data Summarization: Aggregation of large

quantities of data into understandble summarized reportlets.

14

Simple State & State History Buffer

• JMX Attributes Contain the Latest Scanned Value of a Performance Metrics

• History Buffer Show the Last n Readings.

15

Persistence & Reporting

• Store performance data for: Reporting Forecasting Capacity Planning Trend Analysis

16

Charting

• Visualization of real time performance data.

• Side by side or large and rich displays containing many graphs for visual correlation.

17

Alerts

• Alerting by email and pager when established threshholds are crossed.

Nagios Monitoring System Service Status Screen

www.nagios.org

18

Boundary Tracking

• Track boundaries of selected metrics High Low Average

19

Summarization & Aggregation

• Large quantities of performance data can be aggregated and summarized to an understandable reportlet.

\\IISWEBSERVER01\Thread(inetinfo/0)\Thread State: 5\\IISWEBSERVER01\Thread(inetinfo/0)\Thread Wait Reason: 0\\IISWEBSERVER01\Thread(inetinfo/1)\Thread State: 5\\IISWEBSERVER01\Thread(inetinfo/1)\Thread Wait Reason:6\\IISWEBSERVER01\Thread(inetinfo/2)\Thread State: 5\\IISWEBSERVER01\Thread(inetinfo/2)\Thread Wait Reason: 0\\IISWEBSERVER01\Thread(inetinfo/3)\Thread State: 5\\IISWEBSERVER01\Thread(inetinfo/3)\Thread Wait Reason:6\\IISWEBSERVER01\Thread(inetinfo/4)\Thread State: 5\\IISWEBSERVER01\Thread(inetinfo/4)\Thread Wait Reason: 0\\IISWEBSERVER01\Thread(inetinfo/5)\Thread State: 5\\IISWEBSERVER01\Thread(inetinfo/5)\Thread Wait Reason:6\\IISWEBSERVER01\Thread(inetinfo/6)\Thread State: 5\\IISWEBSERVER01\Thread(inetinfo/6)\Thread Wait Reason: 0\\IISWEBSERVER01\Thread(inetinfo/7)\Thread State: 5\\IISWEBSERVER01\Thread(inetinfo/7)\Thread Wait Reason:6

20

Helios – An Implementation

• An Open Source Implementation• Powered by JBoss Application Server• Performance Data Service & Clearing

House• Heterogeneous Data Sources• Extensible Monitors and Post

Processing• Customizable and Easy to Configure

Dashboards

21

Evolution of Helios

Original State of Affairs• Multi-tool data collection• Manual Data Merge• No Proxy. All monitor clients put load

on servers.

22

Evolution of Helios

Simple Automation• Automated multi-tool data collection• Daily merge• Manual Reports and Charts

23

Evolution of Helios

Simple Dashboard• Numeric and character data only. • Web page console.• Limited data.

24

Evolution of Helios

Evolving Architecture• Implementation of MBean Proxies.• MBean generated graphics.• Single console still requires rebuild and redeploy to

change.

25

Helios Version 0.1

New Architecture• Dynamic Dashboards

that can be saved and shared.

• Dashboard display widgets can be resized and moved.

• Configurable, extensible and hot deplyable data source monitors.

• Extensible post-processing using plugins.

• External client API for push data.

• More…..

26

Helios Architecture

• Extensible and Robust JMX (Remote Notification Really Work)

• Hibernate Managed Persistence (& HARs !!)

• Scalable and Clusterable Services

• Multi-Protocol Remoting

• Great Platform To Work On

• Free Platform To Work On

27

Monitors - Overview

• Implemented as a JMX MBean Acts as a proxy to a performance data source Performance data is available as attributes Configured using XML in jboss-service.xml Can be configured to pass results to a

collection of plugins Broadcasts notification of new scans and

attribute changes Active Monitors scan a performance data

source (Pull) Passive Monitors receive performance data

from an External Client Agent (Push)

28

Monitors - Types• Currently Available

JMXMonitor - Monitors JMX Attributes WindowsMonitor - Monitors Windows Performance Manager Stats SQLMonitor - Monitors JDBC Data Sources YahooFinancialMonitor - Monitors Yahoo Finance Securities PassiveMonitor – Creates attributes for holding results from a remote agent JRat Java Byte Code Instrumentation – Acquires Java Method Elapsed Times

• In Progress LinuxMonitor: Monitors Linux /proc file performance data JSR77 Autoconfiguring Monitor Text Stream Reader

• File Scraper (Log Analyzer) • Telnet Data Acquisition • SSH Data Acquisition

• Planned OpenArm Helios Mediator: Application Response Measurement Data Acquisition GkRellm Data Acquisition NetPerf Data Acquisition SNMP Data Acquisition PING Monitor

29

Monitors – Sample Configuration

30

Plugins - Overview

• Configured for each Monitor Receives a copy of each scan result Extends a monitor’s functionality Configured using XML in jboss-service.xml Can create a new result and register it an

attribute in the Monitor (back register)

31

Plugins - Types• Currently Available

PersistScalar – persists scalar results PersistSimpleArray – persists simple arrays RollingHistory – Tracks ‘n’ previous versions of a scan VariaGraphPlugin – Creates line graphs (JPG, PNG or SVG) TrackAverageNumber- Computes averages BasicSQLHTMLFormatter – Formats ResultSets as HTML J2EEStatsHTMLFormatter – Formats JSR77 results as HTML TrackHighNumber – Maintains a maximum value TrackLowNumber – Maintains a minimum value TrackAverageNumber – Maintains the average value

• In Progress Graphing Library Nagios External Event Updates File Logging Event

• Planned Customizable Event Broadcaster For Integrating to 3rd Parties Customizable HTML Renderer

32

Plugins – Sample Configuration

33

Subscription Service - Overview

• Allows a client to subscribe to and receive updates for Monitor attributes Implemented as an MBean HTTP Façaded provide for web clients Clients establish a “session” Server Side Proxy manages client’s

subscriptions Subscription listen on the notification bus and

filter attributes notifications based on client’s subscription

Client can subscribe/unsubscribe to attributes Client “polls” and receives all available updates

34

Subscription Service - Components

Http

Session..

.

Clients

Subscription ServiceHTTP Facade

Subscription Service(Mbean)

Client Proxy (Mbean)

Client Proxy (Mbean)

Client Proxy (Mbean)

HTTPSessionListener manages thelifecycle of the Client Proxy.

Non Http clients must manage theclient proxy lifecycle themselvs

SubscribernSubscriberSubscriber

Client Info

Notification Bus

Monitors

JMX Notifications

Client Proxy is responsible formanaging the subscriptions and theDOM

Subscriber is responsible formanaging a single subscription. Allfiltering and/or managing of messagefrequency “differences” are handledhere.

xferDOM

Subscribe/Unsubscribes

getUpdates

Attribute Subscriptions update theDOM directly.

getUpdates marshals the DOM of allavailable updates

35

Subscription Service – Basic Use Cases

Client

Subscribe

Subscription Facade Subscription Service

Subscribe

Client Proxy

Create

MBean Server

Subscribe

Create

Subscriber

Subscribe

NotifyUpdate

Update Notify

Update Notify

Xfer DOM

Create

Subscribe

Subscribe

SubscribeCreate

Subscriber

Subscribe

NotifyUpdate

NotifyUpdate

NotifyUpdate

HttpSessionListener

getUpdates

getUpdates

read

36

Visualizing The Data

Dashboard Manager A WYSIWYG tool for managing and sharing

views of your information real-time via “Dashboards”.

37

The Goals Of Helios Dashboard Manager

• Easy Creation

• Easy Viewing

• Easy Sharing

38

Easy Creation

• All available “Monitors” displayed for selection in a tree style view called the Dictionary

39

Easy Creation Cont.

• Drag and Drop style dashboard creator• Supports resizing and overlapping

40

Easy Viewing

• View dashboards using a browser Currently supports IE 5.5+ and Firefox 1.0

• Supports full-screen viewing and slide show style for multiple dashboards.

• Live Data Updates. Supported output type include… Single Value Array Rich HTML Image .gif, .jpg, .png SVG

• View on any platform via Firefox

• No additional client installs or plugins required

• Clean presentation ready look

41

Easy Sharing

• Dashboards support custom HTML content via “Labels” that allow the creator to annotate a dashboard as necessary.

• Underlying XML descriptor allows for dashboard definitions to be imported and exported between Helios server instances.

42

Future Plans

• Library of Client-Side Rendered Graphs via SVG• More Dashboard Customization Options

Look and Feel Behavior

• Security/Dashboard Ownership Integration

43

Client API

• Remote API support for: Creating New Passive

Monitors. Creating New Monitor

Attributes Updating Monitor

Attributes

• Protocols: RMI JMS Simple XML over

HTTP

• XML over HTTP allows non-Java scripting languages such as Perl and TCL to perform remote data acquisition for Helios.

44

Thanks !

Helios: Greek God Of The Sun. He sees everything that happens and is often called

upon to shed light on events.

Helios. See All. Hear All. Understand All.

https://helios.dev.java.net/index.html